diff --git a/docs/project/vla-backward-corrections-2026-10-13.md b/docs/project/vla-backward-corrections-2026-10-13.md new file mode 100644 index 0000000..cf7a531 --- /dev/null +++ b/docs/project/vla-backward-corrections-2026-10-13.md @@ -0,0 +1,127 @@ +# 后退专项纠偏配对实验(v21) + +v20表明两个训练说法在专家状态上拟合相近,但训练初态也会失稳。 +本轮问:相同更新预算下,集中拟合后退时,加入学生偏离状态的专家纠偏是否改善闭环? +这是后退专项诊断模型;通过旧任务回归检查遗忘,不默认替换v19或部署到robotd。 + +## 数据与对照 + +- 固定v19step4000初始权重,模型与state51、H10/E5、50Hz、黑图契约不变。 +- 重放v20两个已知失败(body700/730)的全部执行动作,要求物理数组逐tick精确相等。 +- 从每条的20/50/80tick接管,共6个候选。同步专家的previous action,再执行200tick。 + 只保存接管后的专家动作为训练标签,学生前缀不作标签;所有候选原始结果保留。 +- 原backward严格判据不变;至少4条接受且覆盖两个来源身体才进入训练。窗口重叠,不能当6个独立身体。 +- 老师对700/730及新评估身体740..742先做200tick正控;必须严格通过。新身体因此已做教师检查, + 不能称完全未触碰,但尚未根据学生结果调参。 +- 新native dataset完整保留v19的129eps/30400frames,再追加接受的纠偏。 + 新归一化由完整训练union计算;没有评估状态参与统计。旧数据、模型不改、不删。 + +两组都从v19相同权重、共享union有效processor开始;control只通过原生episodes选择10条旧后退轨迹, +mixed选择相同10条加接受的纠偏窗口。两组均fresh optimizer、batch8、seed42、1000正式更新, +另各2步smoke丢弃不用作初始化,总预算2004更新。更新预算相同,但原数据曝光改变, +只能比较“集中后退训练的混合配方”,不能归因纯额外纠偏样本的独立效应。 +旧课程其他家族只影响共享归一化,不被这两组采样训练。 + +纠偏沿用两个训练文本交替标注;不加入朝身后退着走或往后退步,不将看过的失败表达偷换成新holdout。 +这里没有文本规则分派、专家接管的学生验收、新增RL损失或模型架构改变。 + +## 验收 + +zero只替换union统计、0更新;control/mixed各自训练后采用相同22case: + +- 后退body700/730(开发)与740..742(新身体),交叉原话/旧留出说法,共10case,每条200tick。 +- 旧HEAD_TASKS六任务在body730各两说法,共12case;forward600tick另报4秒,其余200tick。 +- smoke每组在700做10tick后退;只要求完整、有限、稳定,短长度strictFalse不作失败。 + +相同初态physics/state/prior,噪声seed=body*10000+tick。保留每次实际黑图/state/prior/chunk/latency和trace。 +旧留出说法已使用,不是新语言测试;只有1 optimizer seed,不能把22case当22次独立训练。 +物理暂停等待推理,无实时/视觉/硬件能力声明。指标包括strict、平衡、方向、语言、遗忘和延迟。 + +## 预期与研究判断 + +| 解释/方案 | 预期 | 可区分结果 | +|---|---|---| +| 纠偏状态覆盖不足 | mixed比control在新身体上更稳定且后退 | 仅开发身体改善不足以支持泛化 | +| 原数据拟合精度不足 | control也可能明显改善 | 两组均改善不证明纠偏必要 | +| 纠偏曝光或量不足 | mixed仍失败 | 不凭一轮否定纠偏,报告实际样本曝光 | +| 多任务干扰/遗忘 | 专项后退变好,旧任务变差 | 不能当全课程平替结果 | +| 语言起步映射不足 | 原话改善而旧留出仍不动 | 后续单独设计语言监督,不扩大本轮 | + +采集/准备先审计,再唯一启动训练。任何门禁失败保留证据,不降门槛、不自动增加预算。 +工件在devbox /srv/duckmind-training/v21-owner/run;入口run-backward-curriculum.sh。 +结果交付后暂停自动跟进,不自动追加下一轮或合并PR。 + +## 采集进展 + +prepare进程1852532已完成6候选、5个教师正控和原生数据转换,随后在记录上游Git版本时退出。 +5个纠偏窗口通过,共1000帧;700prefix20全程稳定但净后退和尾部后退未达标, +保留原始轨迹、不纳入训练。5个teacher-only正控均通过,完整学生失败回放与接管点物理精确一致。 +实际union134eps/31400frames,control10eps/2000frames,mixed15eps/3000frames。 +实际以prepare.exit-code=0、data-audit和独立审计为准,当前尚未正式训练。 + +代码d57ecd376e2d712f5d2c62f3db8c666193f2c4a9已推送并核验远端一致,训练组件 +make fix/lint/test通过(17passed、2skipped)。devloop提交推送后仍因旧action-chunks路径的 +postcheck退出1,不能称整个工具全成功;PR5实验描述已更新,未自动合并。 +独立审计重点是非连续episode选择的全局/本地索引映射、实际future-action/padding、 +两组fullunion归一化以及纠偏raw到dataset时序。审计通过才可唯一启动2004更新流程。 + +## 准备收尾恢复 + +首次prepare在数据和zero checkpoint保存后,调用上游`git rev-parse HEAD`失败;devbox上的 +上游是无`.git`的同步源码目录。原prepare.log与prepare-attempt1-source.py保留,未进行训练。 +修复把收尾独立为`finalize-backward-preparation`:读取已保存manifest,核验源/数据/初始权重 +哈希及6个有效processor,重建两组选集,不重新采集、编码视频或更新参数。 +Git revision记录为null并保留原错误,teacher、推理源码、scene的文件哈希作为已验证身份。 +元数据收尾退出0,prepare.exit-code=0;这不抹去首次prepare失败。 +独立数据审核未完成前,不启动训练。恢复代码make fix/lint/test通过(17passed、2skipped)。 + +## 独立审核与启动 + +独立CPU审核11项通过(工件parent independent-preflight-audit.json/.md):旧129eps逐帧保持, +新5窗口与raw精确对应,6候选和5正控严格评分重算一致;2global+268episode数值统计、 +6zero有效张量与manifest哈希一致。原生sampler完整索引检查确认control2000、mixed3000帧, +没有其他任务混入;逐episode首中尾75样本和真实DataLoader48样本的全局索引、H10未来动作 +与padding均准确。CPU审核workers0,真实训练workers2由smoke覆盖。 +全31400黑图解码复用prepare完整结果及哈希,并独立抽检;未使用的近似quantile未重算。 + +唯一训练launcher PID1897903已启动,入口train-backward-pair,冻结源码da1ede7b3f4f74dd7ba5211277773e04252483e3。 +预算仍为每组2smoke+1000正式,共2004;需等zero/control/mixed所有物理验收后比较能力。 +原准备失败日志与attempt1审计继续保留;本次只续元数据收尾,未重复采集或训练。 + +## 最终结果与研究决策 + +runner已完成,run.stage=complete、run.exit-code=0、PID1897903退出。四次原生训练为 +2、2、1000、1000更新,所有checkpoint的155个优化器step计数与native training_step一致; +总2004预算已用完,没有重训或追加。两组正式模型均从同一v19权重开始,smoke权重丢弃。 + +这次等预算纠偏配方没有改善严格后退成功率。混合组部分轨迹更稳定,但偏航或原地不动, +不能算任务完成;两组旧任务回归均下降。这些诊断模型不替换v19、不部署robotd。 + +| 模型 | 后退开发初态原话 | 后退新初态原话 | 后退旧留出措辞 | 后退严格合计 | 后退全程稳定 | 旧任务严格回归 | +|---|---:|---:|---:|---:|---:|---:| +| zero:相同初始权重、新统计、无更新 | 0/2 | 0/3 | 0/5 | 0/10 | 6/10 | 9/12 | +| control:原后退数据 | 0/2 | 1/3 | 0/5 | 1/10 | 6/10 | 7/12 | +| mixed:原数据加纠偏 | 0/2 | 0/3 | 0/5 | 0/10 | 7/10 | 7/12 | + +- mixed在开发body730完成200tick且稳定,但偏航−0.620rad,超过原判据,仍失败。 +- mixed在新body742只后退0.00452m;control同初态后退约0.284m并通过。稳定站着不是后退成功。 +- 旧站立/头部任务zero9/10、control7/10、mixed7/10;前进12秒三者均0/2,4秒前缀分别0/2、0/2、1/2。 +- 三者推理中位数约248/253/251ms;mixed p95约295ms,939次均超过E5的100ms执行窗口。 + 评估仍暂停物理等待推理,不具备实时执行证据。 + +独立最终审核位于devbox /srv/duckmind-training/v21-owner:independent-final-audit.json/.md、 +independent-final-ml-debug-form.json、independent-final-ml-debug.md。核验68条轨迹(66正式比较+2smoke)、 +14235次执行动作、2852份实际输入,重算68项strict、30项gaze、6项前进4秒、212项交叉目标评分、 +176条fit误差,全部与记录一致。同初态、每phase6个有效统计张量、source/data/external/model哈希一致。 +完整run.log共3337行257561字节,原始失败均保留。独立审计曾把日志step1K解析成1,已仅修审计解析器, +不改运行代码,不重训;原prepare的Git元数据失败及零训练收尾恢复也继续保留。 + +限制:一个优化器seed;新初态仅740..742三个,且经过教师筛查;旧留出措辞此前已用过; +纠偏5窗口仅来自两个相关初态,混合也改变原数据曝光。因此不能从1次与0次成功推断control普遍更好, +也不能据此否定纠偏数据方法。没有视觉、真实硬件、连续任务切换或新语言泛化证据。 + +下一步最便宜的诊断是检查**新增纠偏标签本身是否被模型拟合**。当前22个fit探针只取每个family +最早旧episode的t0/t20,完全没有覆盖129..133新纠偏episode;这个证据缺口现在明确记录。 +可先固定state/text/noise,对比zero/control/mixed在新纠偏状态上的解码动作误差;若能拟合却闭环失败, +再单独比较E1/E5执行窗口。以上为后续建议,本轮未执行、未自动增加预算。 +结果交付后暂停duckmind-vla自动跟进,PR5保持open、不合并。 diff --git a/docs/project/vla-backward-diagnosis-2026-10-13.md b/docs/project/vla-backward-diagnosis-2026-10-13.md new file mode 100644 index 0000000..fd38f9e --- /dev/null +++ b/docs/project/vla-backward-diagnosis-2026-10-13.md @@ -0,0 +1,73 @@ +# 后退:语言敏感性与闭环偏离诊断 + +用户授权继续学习后退。先用v19固定checkpoint解释失败,本轮不更新权重、不新增训练数据。 +同一模型在熟悉文字下后退并失稳、在留出文字下原地稳定;这可能由不同机制导致。 + +## 问题与选择 + +| 选择 | 预期区分 | 本轮 | +|---|---|---| +| 同一老师状态换文字 | 语言条件对动作拟合的影响 | 执行 | +| 分阶段拟合与逐关节误差 | 起步拟合好但步态后期误差大 | 执行 | +| 训练初态与旧失败重现 | 已有轨迹也失败,还是初态变化后失败 | 执行 | +| 调整采样或补纠偏 | 改善拟合或扩展闭环覆盖 | 待上述证据,不预先开训练 | +| 改模型/损失/解冻语言塔 | 改变表达能力或语言适配 | 当前无隔离证据,暂不改 | + +诊断先验(主观、非统计后验):稀疏步态拟合/闭环偏移40%、语言条件泛化30%、 +多任务采样与干扰15%、训练/评估实现缺陷5%、未知10%。独立检查token序列长度7/8/15, +均小于48且不同,暂不支持截断或碰撞;VLM冻结不意味着文本嵌入没有区分能力。 + +## 配对设计 + +固定v19 step4000及其原归一化,黑图、state51、H10/E5不变。 +现有入口head_curriculum.py新增diagnose-backward;不另造仿真或模型接口。 + +- 读取老师train700/701/702和calibration720/721/722各200帧原轨迹。 +- 预声明tick0/20/60/100/140/180,state/prior/采样噪声完全相同,只换六种文字。 +- 两个训练文本:向后走、往后退一点;两个未训练文本:往后退步、朝身后退着走。 + 前者虽在采集目录定义中,但未选入训练;后者已用于v19评估,不是全新测试。 +- 错目标控制:保持站立、向前走。共216个配对预测,另1个相同输入重复smoke。 + 保存原始state/prior/target/chunk、第一步/执行前5步/H10误差、逐关节误差、persistence。 +- 使用原evaluate复现body730/731两种措辞,增加训练body700两种措辞,共6条。 + 原evaluate另有22个训练集fit探针;不增加训练。比对旧失败的实际输入/动作/物理状态。 +- 第一组输入保存原生processor张量(token、mask、归一化state、blackimage)。 + 它是同一输入的预处理重建;独立CPU审查确认与实际predict使用同一processor。 + +相同输入smoke应完全相等;旧730/731应重现,若未重现保留差异并先调查。 +216个预测不等于216个独立轨迹或重复训练。当前checkpoint的误差不能证明继续训练带来的因果效果。 +若canonical在老师状态上已优于persistence但身体闭环仍失败,优先调查状态分布偏移; +若第二训练说法或后期步态本身拟合差,先解决拟合/覆盖,而不是一概加同义句。 +错误目标的MSE不是该目标本身正确性的评估;姿态相近也可能使其误差低。 + +## 执行与复核 + +工件:devbox `/srv/duckmind-training/v20-owner/run`,使用v19数据只读软链接。 +本轮0优化更新,模型与输入工件SHA、完整命令、实际processor记录在report。 +旧run及失败证据保留。独立设计审查未发现启动阻断,指出第三文本未训练、 +文本敏感不等于语义正确、完成状态不代表重复完全一致,均按上述边界处理。 + +后续结果与ml-debug表待实际完成;没有预先宣称诊断成立或已开始训练。 + +## v20结果 + +流程完成退出0,共436次模型调用:216离线配对、1次同输入重复、22个原生fit探针、197次闭环请求,0优化更新。 +独立CPU审计逐项重算216组误差和state/prior/target配对;730/731四条旧试验的40项数组精确复现。 +完整证据在v20-owner/independent-backward-audit.md、.json及independent-backward-ml-debug-form.json。 +当前源码后续会继续修改,21份本轮源码及SHA已保存source-snapshot/,不能用新源码冒充旧实验。 + +腿部H10 MSE(rad²,早期0/20,后期60/100/140/180;各split仅3个身体): + +| 文本/对照 | train早期 | train后期 | calibration早期 | calibration后期 | +|---|---:|---:|---:|---:| +| 向后走 | .002822 | .000678 | .003402 | .001053 | +| 往后退一点 | .002717 | .000685 | .003114 | .001091 | +| 朝身后退着走 | .007877 | .002143 | .005934 | .002253 | +| 保持站立(错误目标) | .024135 | .020765 | .019465 | .027040 | +| persistence | .023854 | .194649 | .019468 | .147585 | + +错误目标也能胜过后期persistence,故不能用该比较证明语言语义正确。 +但两个训练文本显著优于错误目标,不能解释为模型完全忽略文本。 +训练身体700原话137tick失稳;730/731分别103/139tick失稳;留出措辞三者均稳定但几乎不动。 +专家状态上后期拟合没有恶化,熟悉初态仍失败,支持优先测试闭环纠偏覆盖;尚不是分布偏移的因果证明。 +留出说法在运动状态上能产生近似后退动作,却不能从站立状态启动,需保留文字×初态交互假设。 +下一轮先处理身体闭环,语言泛化缺口仍单独报告。 diff --git a/docs/project/vla-correction-fit-2026-10-14.md b/docs/project/vla-correction-fit-2026-10-14.md new file mode 100644 index 0000000..c4478e1 --- /dev/null +++ b/docs/project/vla-correction-fit-2026-10-14.md @@ -0,0 +1,93 @@ +# 纠偏标签拟合诊断(v22) + +v21混合纠偏未提高严格后退成功率,但其离线探针没有覆盖新增纠偏episode。 +本轮固定现有三个checkpoint,检查“新增纠偏训练状态上的目标序列是否更接近教师”。 +只诊断拟合,不更新参数、不运行物理、不扩大任务,也不据此宣称闭环成功。 + +## 配对设计 + +复用v21完整union与zero/control/mixed,6个有效state/action统计必须精确相同。 +原10条后退episode及全部5条新增纠偏episode129..133,每条取t0/20/60/100/140/180, +每个样本固定两个torch flow-noise种子;state51、实际previous_target、训练文本、黑图、 +H10动作目标在三个checkpoint间相同。每模型额外重复首个输入一次并要求精确相等。 +总15×6×2×3+3=543次推理,0训练更新、0物理rollout。 + +原始teacher NPZ与native dataset每个选中state/prior/text/H10逐项核验;t180仍有20帧, +H10不得padding。保存90个原始输入、每模型180个预测、原生processor消费的首样本张量、 +三个重复smoke、全部源码/数据/权重哈希及完整日志。旧v21源码22文件先保存独立snapshot。 + +按original/correction、early(t0/20)/late(t60..180)分别报告first/E5/H10物理角度MSE, +包括全部15关节、明确名字映射的10腿关节与逐关节误差;persistence使用相同窗口和分母。 +另外按episode保存均值。两次噪声不是独立训练,两条来源轨迹的重叠纠偏也不是5个独立身体; +所有状态都来自训练数据,没有新语言或heldout效果。 + +## 预期与判别 + +| 可能解释 | 本轮可见证据 | 不能据此声称 | +|---|---|---| +| 纠偏标签拟合不足 | mixed在纠偏状态不优于zero/control | 纠偏方法无效 | +| 样本上拟合但闭环离开分布 | mixed纠偏误差下降而v21任务仍失败 | 已证明唯一闭环失败机制 | +| 动作块后半段掩盖执行段问题 | H10改善而first/E5未改善 | 改E1必然有效 | +| 静止关节拉低平均误差 | 全关节改善而腿部/特定关节无改善 | 学会后退 | + +不将原生flow-velocity训练loss与解码角度MSE直接比较。下一步是否测E1/E5取决于本轮结果, +本轮不附带追加训练或物理实验。独立审计重算原始输入、误差分母、噪声配对和processor契约。 + +## 运行 + +现有head_curriculum.py新增diagnose-correction-fit阶段,沿用SmolPolicy.predict与native dataset。 +工件devbox /srv/duckmind-training/v22-owner/run,launcher run-correction-fit-curriculum.sh。 +零更新诊断完成后记录结论并保持PR5未合并。首次本地lint发现过长日志字符串,修正后重新检查。 + +## 完成结果 + +冻结代码730244f0131954ebadddfe83c6c6447efc9f2fdc;launcher167365完成、exit0。 +实际543次推理(540配对预测+3次首样本重复),0训练更新、0物理rollout。 +因此本轮没有新模型,比较的仍是v21三个checkpoint。 + +纠偏标签确实被更好地拟合了一部分;不能再把v21失败概括为“纠偏样本完全没学到”。 +但这种训练状态上的误差改善并不等于闭环稳定性已经足够,v21 mixed后退仍0/10。 + +下表是**实际执行前5步、10个腿关节的MSE(rad²,越低越好)**。 +每个episode的采样数相同,两次固定噪声;按原始数据/纠偏分组,不把预测条数当独立实验。 + +| 状态来源 | zero | control | mixed | mixed相对control | +|---|---:|---:|---:|---:| +| 原有10条后退轨迹 | 0.001119499 | 0.000446564 | 0.000549084 | 误差高22.96% | +| 新增5条纠偏轨迹 | 0.003567184 | 0.002987197 | 0.002133534 | 误差低28.58% | +| 纠偏接管初期t0/20 | 0.009444411 | 0.008502763 | 0.005864667 | 误差低31.03% | +| 纠偏后续t60/100/140/180 | 0.000628570 | 0.000229413 | 0.000267968 | 误差高16.81% | + +按episode聚合,5/5纠偏窗口mixed误差更低,10/10原有轨迹mixed误差更高。 +这里“更高”是相对control;两组在原有轨迹上的误差仍都低于zero,不能说相对初始模型全面退步。 +纠偏改善主要出现在接管初期,但该时段残余误差仍大;没有经物理验证的MSE及格门槛, +不能只凭下降比例判定拟合“足够好”。early与late样本数不同,合计按实际6采样点权重计算。 + +persistence同一E5腿部误差:原有0.070530605、纠偏0.080287203,三模型均低于它。 +该基线对行走很弱,v20错任务文本也可能优于它,不能据此证明任务语义或平衡控制。 +本轮仅两个训练说法,没有新语言测试,也没有重新测v21身体闭环。 + +## 审核与后续判断 + +独立CPU审核:90份输入逐项对应原始teacher与native数据;540预测和3重复、55080个指标数值、 +18个有效processor张量通过;三个native输入张量精确一致,模型/源码/数据/raw哈希未变。 +源码/data/model均保留;v21旧运行源码snapshot已在修改前保存。 +本地make fix/lint/test通过(17passed、2skipped),shell语法检查通过。 +完整工件与独立审核在devbox /srv/duckmind-training/v22-owner。 + +目前证据更支持“纠偏样本拟合改善与原有数据拟合之间有取舍,且离线改善没有转化为闭环成功”。 +它不证明唯一失败原因。一个优化器seed、两个纠偏来源身体、重叠窗口限制因果与泛化推断。 +下一步可对当前v21权重做有界E1/E5物理对照,区分预测一次后连续执行5步对失稳的影响; +与更早版本的E1失败不能直接互相替代。先保持数据/权重/归一化/初态固定,不盲目追加训练。 +该后续对照尚未执行,当前自动跟进保持暂停,PR5未合并。 + +### 误差集中位置 + +两个固定噪声分别统计,都呈现纠偏early改善、late略差;early配对预测18/20误差降低, +late13/40降低。这是相关样本描述,不是独立试验显著性。 +接管t0的E5腿部MSE mixed0.010801623/control0.015796513;t20为0.000927710/0.001209014。 +残差主要集中在接管瞬间。mixed t0的right_knee、right_hip_pitch、left_knee、left_hip_pitch +分别约0.030177435、0.019991029、0.019531991、0.012933928rad²,共占10腿关节平方误差约76.5%。 +因此不能把结果说成“恢复动作已充分拟合、只剩执行器问题”;髋/膝接管动作的精度仍需验证。 +可先对比当前模型E1/E5是否改变同初态失稳,同时保留恢复入口误差作为竞争解释; +若再训练,应围绕该短时恢复片段做可区分实验,而非仅增加普通稳态行走轨迹。 diff --git a/docs/project/vla-feedback-2026-10-14.md b/docs/project/vla-feedback-2026-10-14.md new file mode 100644 index 0000000..92c7c0d --- /dev/null +++ b/docs/project/vla-feedback-2026-10-14.md @@ -0,0 +1,128 @@ +# 执行跨度与后退闭环(v23) + +v22确认部分纠偏标签拟合改善,但v21混合模型后退仍0/10。本轮固定v21 control/mixed +权重与union归一化,比较每执行1步(E1)和5步(E5)重新推理;不训练、不改数据。 +模型仍预测H10,输入黑图、语言、身体36维及实际previous_target15维。 + +## 配对与预算 + +复用5个已用身体初态700/730/740/741/742,各交叉原话与旧留出说法,两模型两种E, +共40条200tick上限的物理轨迹。两个mixed冒烟各10tick,随后完整E5两组,再E1两组。 +所有evaluate保留22条原有训练集fit探针,推理总上限4944次,优化更新为0。 +每个公共控制tick的flow种子固定为body*10000+tick;相同reset physics/state/prior必须相等。 +E5预测、动作、物理轨迹应精确复现v21;记录摄影图微差和延迟,但二者不参与该数值门禁。 + +使用原200tick严格后退判据,分别记录平衡、方向、位移、初态、实际输入与逐tick执行。 +两个冒烟需完整、稳定、有限输出;10tick的strict=False不代表冒烟失败。 +所有旧数据/模型/source先保留hash,旧v22源文件已保存snapshot,再扩展既有入口。 + +## 解释边界与研究决策 + +E1同时改变观测频率、动作块执行位置(总取第0步)及flow采样频率。 +改善只能归因这套重规划方式,不能孤立证明“观测陈旧”是根因。 +同一body公共tick噪声相同,但闭环分岔后state不同;不是相同输入的离线因果对照。 +这5个初态、两种说法都已用于开发,不证明新语言或新初态泛化;一训练seed。 + +| 选项 | 预期可区分结果 | 本轮取舍 | +|---|---|---| +| 固定模型E1/E5 | E1改善严格后退或延后失稳,支持重规划方式影响闭环 | 执行 | +| 直接追加纠偏训练 | 同时改变预测与分布覆盖,无法隔离执行跨度 | 暂不执行 | +| 更换模型结构 | 同时改变容量与归纳偏置,成本更大 | 暂不执行 | + +若两种E均失败,不能否定纠偏或语言条件策略;恢复入口髋膝预测残差仍是竞争解释。 +若E1仅更稳定但不后退,不算任务成功。暂停物理等待模型不代表实时控制,E1的20ms预算 +比E5的100ms更紧,实测推理延迟单独报告。没有专家接管,没有实机部署。 + +## 执行与审核 + +复用head_curriculum.py的evaluate,仅参数化execute_steps,旧调用默认E5不变。 +新增diagnose-feedback编排及run-feedback-curriculum.sh。工件位于devbox +/srv/duckmind-training/v23-owner/run。完成后独立CPU重算质量、动作索引、配对输入、 +实际processor、模型/源码/数据hash和全日志,填写ml-debug form并更新PR5,不自动合并。 + +## 完成结果 + +冻结代码069d16e0d1a9167eb1c24f606d08b3a2eed2e292;launcher578291/Python578293。 +run.stage=complete、exit0,42条轨迹(40main+2smoke),实际4795次推理、0优化更新, +总耗时约1460秒。两组E5全部数值轨迹与v21精确复现,独立审核还复算了metric JSON与判据。 + +| 模型与执行跨度 | 严格后退成功↑ | 全程稳定↑ | 原话成功↑ | 旧同义说法成功↑ | +|---|---:|---:|---:|---:| +| control E1 | 2/10 | 9/10 | 2/5 | 0/5 | +| control E5 对照 | 1/10 | 6/10 | 1/5 | 0/5 | +| mixed E1 | 1/10 | 10/10 | 1/5 | 0/5 | +| mixed E5 对照 | 0/10 | 7/10 | 0/5 | 0/5 | + +control成功对应E1的741/742原话,E5仅742原话;mixed仅E1的742原话通过。 +两组E1都少了3次失稳,但各只多1次严格任务成功。每组旧同义说法都稳定5/5却成功0/5。 +因此“执行跨度影响稳定性”和“语言条件下产生正确后退动作”必须分别解决。 +这只是固定五个已用身体初态、旧措辞、一个训练seed下的描述结果,不代表总体成功率显著提高。 + +control body700原话E5在114tick失稳,E1在151tick失稳且终点yaw更差;不能把更长存活的 +终点位移当作更好步态。独立审核保存每对共同前缀长度的位移与最大heading,避免时长混淆。 +推理中位数E5约235/258ms、E1约255/256ms;均超过各自100/20ms控制窗口。 +E1主验收每模型约553/569秒,E5约126/139秒;CUDA峰值allocated约1.26GB,非全GPU占用。 +本轮不能部署为50Hz实时推理,也没有测异步推理。 + +## 独立审核与竞争解释 + +完整187行日志、42轨迹、4795调用、20对首chunk、12有效processor统计张量、 +24源码/20checkpoint/6dataset/3外部文件hash通过审核。全部实际模型输入为黑图; +每tick执行动作对应chunk索引;previous_target按14实际关节核验,缺失mouth单独处理。 +本地fix/lint/test通过:17passed、2skipped;shell语法检查通过。 + +实际安装LeRobot0.6.2的predict_action_chunk与select_action复用同一生成函数;reset仅清动作队列。 +动作标签从当前帧开始、teacher接管时重建last_action、有效归一化均未发现具体错配。 +原生loss先切实际15动作维再按有效时刻归一,不能用旧版padding稀释解释当前结果。 +这些检查排除了所查范围的具体错误,不是对整个训练栈的无缺陷证明。 + +另有CPU状态尺度分析,仅覆盖当时已完整的control-E5/mixed-E5/control-E1共30条轨迹, +对照原后退教师2000帧与纠偏1000帧。gravity_z的std约0.000791;明显失稳前的标准化值可 +超过教师范围,但最大的值已经伴随实际倾斜。该现象不证明归一化导致摔倒,不据此改epsilon。 +15条旧同义说法均稳定、gravity_z标准化绝对值<0.83,却全失败;小std不能统一解释语言失败。 + +## 后续研究决策 + +保留E1作为诊断线索,不替换现有运行配置或宣称平替教师。当前有两条相互独立的排查线: +原话下的身体闭环仍有方向/恢复误差;同义说法则常退化为稳定但没有完成任务。 +下一步优先用已保存轨迹比较相同初态下原话/同义说法的动作演化,以及失稳前的动作差分, +再决定做语言覆盖对照还是恢复入口单任务拟合。单纯继续堆普通纠偏数据缺少可区分的依据。 +以上是建议,未自动启动下一轮训练、推理或部署。 + +完整证据在/srv/duckmind-training/v23-owner:independent-feedback-audit.json/md、 +independent-feedback-ml-debug-form.md、independent-state-scale.json/md;run保存每条原始trace与输入。 +旧v22源码snapshot与此前所有模型、失败轨迹继续保留。PR5不自动合并。 + + +## 保存轨迹的动作演化复核 + +后续只读取已有工件:40条主轨迹、20组原话/旧同义说法配对、8次跌倒、134个输入文件hash; +没有新增模型调用、训练或物理仿真。独立CPU复核保存完整ml-debug表和可重放的NumPy脚本。 +腿部指标只取10个腿关节,不把头部运动混入步态;动作差分使用实际目标减上一实际执行目标, +单位rad;测量关节速度单位rad/s。语言配对使用共同执行时长,避免把提前跌倒与完整4秒直接比较。 + +- 20/20条旧说法“朝身后退着走”均稳定4秒,但后退仅1.9–3.1mm。目标仍有小幅抖动, + 实际腿部速度衰减至近零,表现为未形成持续步态,而非积极执行错误方向的步态。 +- 同一body730、control E5的共同140tick中,原话后退0.333826m,旧说法仅0.002110m。 + 原话也会近乎静止:control E1的730/740、mixed E1的730/740/741、mixed E5的742。 + 因此不能将所有静止现象归于语言,也不能把E1存活改善全部解释成行走能力改善。 +- body730在相同state/prior/黑图、模型与归一化下,旧说法首个H10腿部动作块不等于站立: + control的原话/旧说法RMS差0.05170rad、旧说法/站立差0.05483rad;mixed分别0.05280/0.05217rad。 + 首块受到文本影响,但不能把连续动作距离解释为“模型将旧说法分类为站立”;后续差异又包含状态反馈。 +- 7/7次E5跌倒的最大腿部目标跳变都发生在新块首步,但执行前身体已倾斜0.326–0.806rad。 + 这些跳变可能是失稳反应或放大因素,不能证明它们启动了跌倒。E1唯一跌倒样本的最大跳变 + 反而伴随倾角减小;仅按跳变幅度不能判断动作是否破坏平衡。 + +本轮后退训练实际只有两种表达“向后走”“往后退一点”,每条episode固定一种表达; +旧同义说法已反复用于开发,不是新语言验收。这些证据支持把语言覆盖与闭环步态分别验证, +尚不支持立即改归一化、增加平滑器或直接追加普通纠偏训练。 + +下一项建议是固定现有模型、归一化、观测和actual previous_target,做小规模“多措辞×多flow噪声” +离线配对:同时保留两种训练表达、旧开发表达、预先声明的新同义表达和站立对照; +取起步与专家行走中的固定状态,比较前5步及完整H10的腿部预测、教师误差和组内采样波动。 +若措辞间差异稳定超过同措辞采样波动,再做等预算文本覆盖对照;若主要由采样波动决定, +优先定位生成与执行的交互。离线拟合仍不能代替闭环成功,新表达一旦用于诊断就不再作最终留出。 +该建议尚未执行,也未分配新训练预算。 + +证据补充:/srv/duckmind-training/v23-owner/independent-action-evolution.py/json/md; +保留每条完整倾角时间线、动作差分、实际速度、共同前缀统计和站立首块对照。 diff --git a/docs/project/vla-head-actions-2026-10-13.md b/docs/project/vla-head-actions-2026-10-13.md new file mode 100644 index 0000000..c0ee83d --- /dev/null +++ b/docs/project/vla-head-actions-2026-10-13.md @@ -0,0 +1,253 @@ +# 头部动作课程:从向前走扩展到多任务语言策略 + +## 目标与基线 + +在固定 Microduck 构型上,让已有学生学习保持站立、向左看、向右看、抬头、低头, +并保留向前走能力。仍使用黑色固定图像,研究的是语言条件运动策略;不证明视觉能力。 +起点是 [v17](vla-long-walking-2026-10-13.md) mixed step1000, +该模型长时平衡有所改善,但严格直线行走尚未与专家对齐。 + +PR #4 已合并;本轮工作树 `vla-head-actions` 从 `origin/main` 的 `0d332fd` 开始。 +实验入口为 `training/experiments/head_curriculum.py`,启动命令保存在同目录 +`run-head-curriculum.sh`;LeRobot 原生训练循环及 SmolVLA 架构保持不变。 + +## 数据与预算 + +- 保留 v17 全部向前走数据:30 episodes、10,600 frames,包含已有重复前缀,不能视为全新轨迹。 +- 新增五类动作各 10 条候选:seeds 600..609,每条 200 ticks / 4 秒; + 相同 seed 的物理状态、36 维观测和上一执行目标在不同任务间必须完全一致。 +- 每类交替使用两种训练表达;完整保存成功与失败轨迹,仅接受合格示范。 + 每类至少 8 条通过才继续,全部通过时训练集为 80 episodes / 20,600 frames。 +- 独立采集 seeds 620..622 的六类教师验收轨迹,不参与训练或归一化统计。 +- state51 = body36 + actual previous target15;action 为 15 维绝对关节目标 rad。 + H=10、E=5、50 Hz,训练与实际推理图像均为 uint8 零值 256×256×3。 +- 2 步 smoke 丢弃,正式从 v17 mixed 权重重新开始 2000 步;batch8、optimizer seed42, + 优化器重启。总新增优化预算 2002,不自动追加。 + +样本仍由原生 LeRobot 按帧采样。向前走约占一半帧,五个新任务分配另一半。 +这同时改变任务覆盖、数据比例与归一化,不能声称隔离了单一原因。 +2000×8/20600≈0.78 次平均帧暴露;如果失败,需要区分学习不足、遗忘和实现问题。 + +## 验收 + +baseline 使用 v17 原统计;zero 使用相同权重与新训练统计;final 使用训练后权重。 +三者分别在相同 3 个身体初态 × 6 任务 × 2 说法上测试,保持相同推理噪声。 +36 条 rollout 只有 3 个身体初态,一次优化器训练,不是 36 个独立训练重复。 + +预先声明的新表达:站在原地保持不动、脑袋朝左边转一点、脑袋朝右边转一点、 +抬起脑袋看看上面、垂下脑袋看看下面、朝着正前方迈步前行。 +这些文本必须不出现在本轮训练集;历史模型是否见过其他近义表达不作无依据保证。 + +头部动作运行 4 秒。要求全程稳定、最大水平漂移 < 0.08 m、yaw < 0.5 rad, +末一秒 head_pitch/head_yaw 对目标的 MAE < 0.15 rad,有指令的关节还需沿正确方向超过 0.1 rad。 +颈部和roll误差完整报告;未命令时不作为零目标的主要验收轴。除此之外,还从保存的完整物理 +状态重算校正后相机光轴相对躯干的方向:相对初始中性姿态,受命轴需沿正确方向 >0.1 rad, +另一轴平均绝对变化 <0.15 rad。站立要求两轴变化均小于0.15 rad。教师和学生共用此规则。 +`*_delta` 是相对默认姿态,而不是相对本次初态。该复合判据是本实验的操作性定义; +原有误差尺度与漂移门限被保留,新增持续性与视线要求在教师、站立和错误目标上标定;不把关节零值等同于自然语言目标。 +同一轨迹按五种目标评分,报告混淆矩阵,不能只看是否保持平衡。 + +向前走运行 12 秒,沿用 `recovery_outcome` strict,并单列 4 秒前缀和平衡稳定率。 +保存每次实际模型输入、chunk、完整物理 trace 与延迟;同步等待推理会暂停物理时间, +不构成实时 robotd 部署证据。短 smoke 只检验真实数据/训练/推理/执行通路; +10 ticks 下 strict=False 是预期,不能当作完整任务评分。 + +## 实验前判断 + +| 选择 | 预期影响 | 本轮决策 | +|---|---|---| +| 五类静态任务 + 行走 replay | 增加可区分语言任务,检查行走遗忘 | 执行 | +| 一次加入所有技能 | 难以定位多种失败,还涉及其他教师模型 | 暂不执行 | +| 改结构、loss 或移除 previous target | 可能改变表达能力,但尚无本轮证据 | 固定现有实现 | + +主观先验而非结论:多任务学习并保持行走 45%;任务干扰/采样失衡 25%;忽略文本 15%; +数据或评估 bug 5%;未知 10%。若同初态输出近似相同、离线拟合却很好,重点查语言启动 +与 previous-target 维持捷径;若 zero 已全面退步,应先区分统计变化;若只在训练表达有效, +不宣称语言泛化。目标是新增能力验证,非证明某训练配方优于所有替代方案。 + +独立审查指出:跨关节或先时间平均会掩盖错动作,现按每关节 tail MAE; +原 collector 仅存成功轨迹的限制由本入口保留全部 Trace 解决;smoke 中无效目标或不完整 +执行必须阻止正式训练。本次使用同模型新上下文独立审查,未进行不同模型家族交叉审查。 + +## 运行与证据 + +远端代码保留 home 相对路径 `~/rustprojects/compforge/duckmind/.worktrees/vla-head-actions`。 +依赖复用原工作树虚拟环境,但 `PYTHONPATH` 明确指向新工作树 training/src 与 policy/src, +已核对两个实际 import 路径;不修改旧环境安装。工件写入根盘 +`/srv/duckmind-training/v18-owner/run`,启动前根盘约 53 GiB,data 盘约 5.2 GiB。 +不删除已有数据或模型。 + +首次 prepare 在第一条轨迹写报告时出现 `TypeError: Object of type bool is not JSON serializable`, +是 NumPy bool 标量,零训练。原始目录保留为 `prepare-attempt1`,完整日志为 `prepare.log`; +修复为内置 bool 并增加 JSON 序列化和错误动作测试,重新采集日志为 `prepare2.log`。 + +第二次 prepare 完成50条训练和18条教师验收轨迹后停在抬头门禁:10/10抬头都稳定, +但neck约−0.229rad越过原先对零目标的0.15rad误差。原命令确实包含neck=0, +不能说四轴判据本身没有依据;这里要学的是文本任务,不是精确复制每个命令寄存器。 + +独立审查提醒两pitch轴方向相反,可能互相抵消。使用`calibrate-gaze`重算校正后相机光轴: +三个初态抬头的视线仰角分别由−0.1774/−0.1808/−0.1748变为+0.0375/+0.0520/+0.0394rad, +变化+0.2149/+0.2329/+0.2142rad。站立尾段约−0.19rad,低头约−0.40rad。 +因此在学生验收前锁定上文的文本任务判据,并保留原四关节误差。 +第二次完整数据和校准结果保留于`prepare-attempt2`,不作为一次成功prepare报告。 + +第三次prepare已完成,完整日志`prepare3.log`共1767行,无新异常。 +独立审计`independent-preflight-audit.json`核验68份raw、80episodes/20600frames, +旧10600帧及文本不变、新10000帧逐帧对应raw、eval和新措辞排除, +episode/global float64统计与6个zero processor张量相等,权重/数据SHA一致, +80个episode末帧的原生H10padding及黑图均通过。 + +教师五类任务训练50/50、独立初态15/15通过;错误目标误通过0/60。 +前进教师4秒strict3/3,12秒strict0/3,仅heading失败,三条均稳定600ticks。 +这再次说明不能把长时平衡与严格直线行走混为一谈。 + +已启动launcher PID4163605(先smoke,随后baseline/zero/正式2000/final), +实时状态见`run.pid`、`run.stage`、`run.log`、`run.exit-code`。源代码提交`eed6fd9`, +新PR为 https://github.com/compforge/duckmind/pull/5 。最终结果见下文。 +代码规范通过,CPU测试13 passed / 2 skipped;这不代替GPU smoke和闭环任务评分。 +依赖记录见`runtime-provenance.json`,输入源哈希见`run-source-sha256.json`。 + +独立校准也使用620..622,因此这些是未进入训练的校准/评估初态,不能称完全未触碰的盲测。 +自动跟进已切换到本轮,预算耗尽并交付结果后暂停,不自动开启下轮实验。 + +## ml-debug 运行表(准备阶段历史快照) + +| 检查 | 当前证据 / 待验 | +|---|---| +| 完整日志与配置 | 首次 prepare 完整 traceback 已读;最终prepare3完整1767行无新异常,config.json固定任务/种子/预算 | +| SHOULD 与实测 | 同初态、黑图、stats与native边界独立通过;训练预算实际值待完成 | +| null 与尺度 | 教师静态任务15/15、错目标0/60;行走12s strict0/3但全部稳定 | +| 初始 demo | baseline / zero 待执行 | +| persistence | 入口已包含每任务起点/20tick的12chunks对照;运行结果待填,不代替闭环 | +| baseline / held-out | 待执行;新初态620..622不进入训练 | +| warmup / lr | 原生 warmup100、decay2000;实际日志待填 | +| 完整 sample | 第一条教师 trace 已落盘;模型实际输入/chunk 待 smoke | +| 最差 step / grad | 未训练,unknown;需要原生训练日志 | +| 意外日志 | NumPy bool 序列化失败;explained: 已修复并测试 | +| 尚缺证据 | 真实GPU smoke、正式训练与最终物理效果 | +| 竞争解释 | 见上方先验,均尚未作为结果归因 | +| 新上下文审查 | 发现平均误差与smoke gate风险,已修正;剩余需实际产物验证 | +| 最便宜区分实验 | 同初态不同文本/同噪声的首个chunk + 短闭环,区分文本启动与姿态维持 | +| 耗时与GPU | prepare 使用 ONNX CPU 与 MuJoCo 渲染;正式峰值显存待原生日志 | + +## 2026-10-13 04:28 UTC 阶段记录 + +smoke实际完成2次更新(原生training_step=2),六任务各完整执行10ticks且稳定; +严格判据False符合短轨迹定义。baseline和zero各36条已完成,正式训练已开始并保存step500。 +当前没有final结果,不能声称已学会新动作或发生训练后遗忘;源文件哈希检查仍一致。 + +| 任务 | baseline任务通过 / 稳定 | zero任务通过 / 稳定 | +|---|---|---| +| stand | 1/6 / 2/6 | 3/6 / 6/6 | +| look_left | 0/6 / 0/6 | 0/6 / 0/6 | +| look_right | 0/6 / 1/6 | 0/6 / 0/6 | +| look_up | 0/6 / 1/6 | 0/6 / 3/6 | +| look_down | 0/6 / 6/6 | 3/6 / 6/6 | +| forward(12秒) | 0/6 / 4/6 | 0/6 / 0/6 | + +两组权重SHA均为`4f5e7dca40aa806de5c1aa5bd3b5fe489f1263679ce1c3afad49e10bd6e71936`。 +每行6条仍只是3body×2措辞。零更新就出现能力变化,证明不能把final相对baseline的全部差值 +归因于参数学习。保持既定流程,不因为这个预先设计的对照结果更改预算或判据。 + +当前完整日志快照1206行/78392字符无Traceback,step10 loss0.997、step530 loss0.076, +记录的mem_gb约2.33;只是优化过程,不代替闭环指标。后续保存的checkpoint以实际training_step为准。 +训练集12个chunk中,静态任务在20tick后的persistence误差很小,也说明离线拟合可被维持姿态 +捷径满足,仍需同初态文本启动和完整任务验收。 + +## 最终结果:新增任务有效,语言幅度与长时行走仍不足 + +本轮 `run.stage=complete`、`run.exit-code=0`,原 launcher 已退出。 +原生 checkpoint 的实际步数为 smoke 2、正式 2000,总预算 2002 已用完。 +最终模型位于工件根目录 `model/checkpoints/002000/pretrained_model`, +权重 SHA256 为 `d7880c045b6bc8366fbccceef1a25883c76be5821c6f8fc47c9b9f8898932749`。 +没有额外训练、专家接管或文本规则分派。 + +| 任务 / 验收时长 | 训练原话通过 ↑ | 新说法通过 ↑ | 全时段稳定 ↑ | +|---|---:|---:|---:| +| 保持站立 / 4秒 | 3/3 | 3/3 | 6/6 | +| 向左看 / 4秒 | 3/3 | 3/3 | 6/6 | +| 向右看 / 4秒 | 3/3 | 3/3 | 6/6 | +| 抬头 / 4秒 | 3/3 | 0/3 | 6/6 | +| 低头 / 4秒 | 3/3 | 3/3 | 6/6 | +| 向前走 / 12秒 | 0/3 | 0/3 | 2/6 | +| 向前走 / 前4秒 | 3/3 | 2/3 | 5/6 | + +五类新增任务为27/30;其中原话15/15、新说法12/15。不同任务从相同身体状态和上一目标开始, +模型收到的任务信息仅是文本。最终站立、左看、右看、抬头、低头的视线方向可区分, +最终30条静态任务没有同时通过两个目标的记录;这支持本轮任务条件生效,不能推广为任意语言能力。 +初态只有620..622三个,且曾用于教师判据校准;只有一个优化器seed,不能以case数替代独立重复。 + +“抬起脑袋看看上面”三条都稳定、视线向上变化0.110–0.138rad,方向和视线门限通过; +但head_pitch目标是−0.4rad,实际tail均值仅−0.144至−0.162rad,MAE为0.238–0.256rad, +超过0.15rad门限。原话“抬头”的视线变化为0.220–0.229rad。失败主要表现为幅度不足, +不能写成完全不理解抬头;也不因结果接近而放宽已锁定判据。 + +向前走原话前4秒3/3通过,但12秒仅2/3完整稳定,另一条在593tick失稳;新说法在 +329/124/291tick失稳。原话两条完整轨迹仍越过heading门限。教师12秒strict也只有0/3, +但三条均完整稳定,因此长时严格直线目标超出了这批教师示范的实际质量;学生平衡也尚未对齐。 +相对baseline,行走12秒稳定由4/6降至2/6;相对同统计zero则由0/6升至2/6。 +这是一次小样本、同时改变训练数据和统计的结果,不能直接归因于纯粹的灾难性遗忘。 + +### 控制组与耗时 + +| 阶段 | 五类静态任务通过 ↑ | 行走4秒strict ↑ | 行走12秒strict ↑ | 行走12秒稳定 ↑ | +|---|---:|---:|---:|---:| +| baseline:原权重、原统计 | 1/30 | 4/6 | 0/6 | 4/6 | +| zero:原权重、新统计 | 6/30 | 0/6 | 0/6 | 0/6 | +| final:训练后权重、新统计 | **27/30** | **5/6** | 0/6 | 2/6 | +| 教师:结构化命令,单次/初态 | 15/15 | 3/3 | 0/3 | **3/3** | + +教师没有语言输入,分母不能与学生两种表达合并。zero中3条站立同时符合低头门限, +提示复合目标存在边界重叠;该评分不是互斥分类。最终结果未出现这一重叠,但不能据此宣称 +判据能识别所有错误动作。baseline/zero差异说明归一化是模型行为契约的一部分,不能仅保存权重。 + +final实际1709次推理的中位数260ms、p95 354ms;baseline1349次中位253ms, +zero1031次中位249ms。不同阶段轨迹长度不同,延迟仅作各自运行描述。 +每块执行5tick仅覆盖100ms,预测10tick覆盖200ms;当前同步推理速度不能满足原执行节拍。 +评估暂停物理等待模型,不是实时部署结果。正式2000步训练约8分53秒;原生日志记录 +`mem_gb:2.33`,不是整台GPU或所有阶段的独立峰值测量。 + +12个训练集chunk的平均MSE:baseline0.013995、zero0.007955、final0.000868rad², +persistence0.015215rad²。final仅6/12逐样本优于persistence:两条行走及四种头部动作起点; +静态动作20tick后的保持阶段仍更接近persistence。平均误差改善不替代任务闭环,也没有做 +persistence身体闭环对照。 + +### 结论与下一步判断 + +已完成的工作是一次多任务语言条件运动训练;新增任务在这些初态和表达上有效。 +当前更值得分开处理的两个问题是同义表达导致动作幅度变化,以及行走的长时稳定性与教师方向偏差。 +下一轮优先做不更新权重的预声明多措辞、未使用初态验收,区分抬头问题是单句敏感还是普遍幅度衰减; +若稳定复现再补语言覆盖。行走则先核验并改善教师长轨迹质量,再决定学生数据配比。 +本轮没有启动这些后续实验,自动跟进在结果交付后暂停。 + +### 独立复核与 ml-debug 决策 + +`independent-final-recomputed.json`重新检查132条轨迹:114条学生(含smoke/baseline/zero/final) +和18条教师。由saved physics经校正camera模型与`mj_forward`重算gaze,质量数值差≤1e-12、 +所有布尔判定与goal_scores一致。学生实际输入全黑,state/prior、E5实际执行、action到下一prior、 +同seed初始physics/state/prior均相等;source/dataset/replay/model hashes、各阶段6个有效统计 +张量、实际2000+2更新均通过。仅做CPU重算,没有新增学习模型推理或仿真rollout。 + +| ml-debug 项目 | 最终证据与限制 | +|---|---| +| 完整日志与配置 | `run.log` 2922逻辑行、216603字符,无Traceback;H10/E5、state51、batch8、seed42、fresh optimizer、train_expert_only=True | +| SHOULD 对照 | 同初态、黑图、数值统计、实际执行与预算全部独立核验;10tick smoke的strictFalse是预期 | +| null / 上限 | 教师静态15/15、错误目标0/60;教师12s strict0/3但稳定3/3。无random policy闭环,不能编造chance rate | +| 零更新 demo | baseline1/36→zero6/36,但行走稳定4/6→0/6;权重相同、统计变化,见控制组表 | +| dummy | final训练诊断MSE0.000868 vs persistence0.015215rad²,仅6/12逐样本胜出;不是验证集或dummy闭环 | +| baseline / held-out | final静态27/30 vs baseline1/30、zero6/30;新说法12/15。三个初态参与过教师校准,不是未触碰盲测 | +| lr / 优化 | step10 loss0.997/lr6.4e-6→step110 loss0.077/lr9.9e-5→step2000 loss0.022/lr2.5e-6;不能由这些点断言确切开始/停止学习时刻 | +| 一个完整样本 | `final/620-look_left-unseen-inputs.npz`:文本“脑袋朝左边转一点”、36state+15prior、黑图→40个10×15chunk→200tick;配对trace完整稳定,gaze yaw变化+0.274749rad | +| 最差梯度 | 首个10步聚合loss0.997、global grad norm6.594;逐模块梯度未记录,无法指向特定模块 | +| 意外现象 | 新说法抬头gaze通过而pitch MAE不通过;explained: 幅度偏小,不是方向错误。zero站立/低头重叠;explained: 两个容差区有交集 | +| 尚缺证据 | 更多预声明表达与全新初态、优化器重复、任务切换、实时部署。未直接保存token IDs/归一化张量/flow noise,仅保存raw输入、配置及确定性噪声种子 | +| 竞争解释 | 对剩余差距的主观研究权重:有限语言覆盖/训练曝光45%,多任务与归一化共同作用25%,训练代码缺陷5%,评估与语义错配15%,未知10%;不是统计后验 | +| 支持与反证 | 语言:同任务改写幅度不同,但四类新说法全过;统计:zero行走骤降,但final恢复部分;实现:raw/有效stats/时序通过,仍缺模块级诊断;评估:教师全过短静态且错目标0/60,但zero目标重叠、教师12秒也strict失败 | +| 新上下文复核 | 132轨迹重算无评分或执行不一致;需要保留评分重叠、校准复用、单训练种子与同步物理限制。完整表见`independent-final-audit.json` | +| 最便宜区分检查 | 下一轮不更新权重、预声明多种抬头改写×新初态;仅个别说法幅度差支持语言覆盖问题,普遍衰减则需查目标尺度/闭环。未启动 | +| 时间与显存 | 正式2000步8分53秒、日志mem_gb2.33;final推理median260ms/p95 354ms。各阶段峰值未独立测量,不拿GPU总量或暂停物理冒充实时性能 | + +三种可能使“学会了”夸大的因素分别有对应检查:目标维持捷径由同初态不同任务及起点chunk诊断约束, +但持续保持阶段仍可走捷径;归一化变化由zero隔离,不能把baseline全差值归于学习;有限初态/语言 +和判据重叠由分层分母、原始错误目标与独立gaze核验暴露,尚未消除外推风险。 +因此这是这套实现与这组测试的正向证据;行走或某句话失败不否定语言条件动作策略本身。 diff --git a/docs/project/vla-language-coverage-2026-10-15.md b/docs/project/vla-language-coverage-2026-10-15.md new file mode 100644 index 0000000..04a9b16 --- /dev/null +++ b/docs/project/vla-language-coverage-2026-10-15.md @@ -0,0 +1,61 @@ +# v25:后退文本覆盖配对实验 + +v24 在固定教师状态和采样噪声下发现,旧说法“朝身后退着走”产生的动作差异大于模型自身的采样波动,而两句新说法没有同样明显的偏差。这支持检验文本覆盖,但不能解释原话闭环也会失败的问题。本轮只比较文本覆盖,不改变身体观测、动作空间或模型结构。 + +## 配对与预算 + +使用 v21 的 134 条源轨迹、31400 帧,原生 LeRobot merge 生成六份相同轨迹副本。对照组保留每条轨迹原文;增广组通过原生 modify_tasks 给每条后退轨迹分别配六种表达:向后走、往后退一点、朝身后退着走、向后迈步、往身后走、往后退步。修改后重新加载 parquet,并修正 task/episode/index 的统计。 + +两组所有非 task 数值、视频字节、episode 顺序和物理统计相同。归一化使用完整 v21 训练数据的 canonical float64 数值,重复六次只改变 count。804 条副本不等于 804 条独立示范。实际采样选取全部 15 条后退源轨迹及其它十类动作各两条,共 35 条源轨迹的六份副本、42000 个 anchor。两组都用原生 EpisodeAwareSampler,seed=42;保存前 8000 anchor 的确定性重建序列与文本曝光。这是预期采样证据,不冒充训练 hook 的实录。 + +初始权重为 v19 step4000,使用 v21 zero-checkpoint 的归一化。state51=身体36+上一实际执行目标15;固定黑图;H10/E5、50Hz、batch8、optimizer seed42。每组 2 步冒烟丢弃、1000 步正式训练,fresh optimizer,总计最多 2004 更新。训练通过已有 duckmind_training.train 调用 LeRobot,不另写 trainer。两组都保留其它动作回放,但不预先假定不会遗忘。 + +## 验收 + +初始模型和两组最终模型各运行 22 个预定 case: + +- 后退:旧开发身体730及预声明新初态760/761,交叉原话、已转入增广训练的旧说法、两句新说法“向身后迈几步”“往后方走一段”,共12条,每条200tick。 +- 旧动作:其它十类各在身体730运行原话,共10条;forward600tick并另报4秒前缀,其它200tick。 + +新初态需先检查旧工件库存,再执行教师可达性校准;它们经过教师检查后不称完全未触碰环境。真正语言与身体同时留出的只有两句新话×两个新初态=4格/模型。单一优化器种子不足以证明一般性显著优势。无效 moving-start 单列,保留分母,不更换 seed、不修改成功阈值。smoke 通过新增 labeled-text 参数执行10tick,仅检验运行链路,不要求200tick任务判据。 + +保存完整实际输入、chunk、执行动作和物理轨迹,独立重算任务成功、平衡、语言差异、旧动作回归及延迟。固定黑图不证明视觉能力,暂停物理等待推理不证明实时部署。不得将关键词分派或专家接管当学生能力。 + +## 运行记录 + +入口:`training/experiments/head_curriculum.py prepare-language-pair / train-language-pair`,启动脚本 `run-language-curriculum.sh`。工件目录 `/srv/duckmind-training/v25-owner/run`。先准备并独立审计,再启动唯一训练进程。代码提交 `c9989ce95b47fc565c5c3c4f0b651000ca95cdc0`;冻结 driver SHA256 `6b9364a7898f9a97103c00709850b3ea0aa74d71e5084b6de90e6ea90bcbebcd`。准备进程1076120已退出0;2026-10-15 20:37(Asia/Shanghai)启动唯一训练编排进程1100371,当前实际进入smoke-control。正式两组1000步会在smoke和baseline之后自动开始,尚不能称训练完成。 + + +准备阶段的证据: + +- 两组各804副本/188400帧,所有非 task 数值相等,视频SHA多重集合相等;源数据未修改。源episode计数不是统计独立样本数,尤其纠偏窗口来自重叠状态。 +- 教师730/760/761各完整200tick且严格通过(3/3),后退进展分别约0.570/0.560/0.537m;不使用这些校准轨迹训练。 +- 采样顺序重建的8000anchors两组相同,其中后退3407(42.59%);增广六说法曝光分别550、556、561、618、573、549。其余4593为十类回放。 +- 库存初查542份旧报告/config/audit的结构化seed/body字段未命中760/761;独立复核会记录完整扫描范围。 +- 本机training lint/test通过(17 passed, 2 skipped),managed提交同时验证policy lint/test。devloop提交推送成功后仍因旧action-chunks路径postcheck退出1,已核实remote SHA一致。 +- native merge尝试加载torchcodec时报告缺少libavutil,随后成功回退pyav;显式数据读入也用pyav。准备无学生推理、无优化更新。 + +独立preflight已通过:parent目录 `independent-preflight-audit.json/.md/.py` 保存96071项检查(72.48秒),完整数值、episode/global统计、有效processor和源哈希通过;真实CPU DataLoader每组前16个anchor通过,全部210 selected episode末尾H10 padding通过。独立旧库存扫描788个JSON,760/761身份字段及两句fresh文本均无命中。扫描清单和哈希已保存;源episode数量不是统计独立性。 + +唯一launcher PID1100371,进度以 `run/run.stage`、`run/run.log`、`run/run.exit-code` 为准。正式结果待填,不把准备或冒烟通过当作学会后退。没有重启自动跟进。 + +20:41 中途快照:两组 native step2 和155个optimizer参数step均为2;真实num_workers2链路通过。两条smoke均完整稳定10tick,实际观测/previous_target/两个预测块与10个执行目标逐项对齐;六有效归一化张量一致。`independent-interim-audit.json/.md` 的134项检查通过(passed-snapshot)。smoke严格任务不通过是因为只跑10tick,不满足200tick门槛,不能当学会或训练失败。 + +初始模型快照仅前两条:身体730原话99tick失稳;旧增广说法200tick稳定,但未达到后退任务判据。两条已独立重算。此时正式训练尚未开始;编排将在baseline完整后分别从同一初始checkpoint训练两组,结果仍待完成,不能从局部baseline推断文本增广收益。 + + +## 最终结果(取代以上运行中快照) + +2026-10-15 21:07:21(Asia/Shanghai)全流程 complete/exit0,实际2+2冒烟、1000+1000正式,共2004更新。独立最终审计 `/srv/duckmind-training/v25-owner/independent-final-audit.json/.md/.py` 3749检查、零不匹配,覆盖全部66主验收和2冒烟、实际输入/执行/初态/六统计/完整日志与哈希。 + +| 模型 | 后退严格通过 | 后退完整稳定 | 其它十任务通过 | 前进4秒 | 前进12秒 | +|---|---:|---:|---:|---|---| +| baseline | 0/12 | 9/12 | 7/10 | 失败 | 318tick失稳 | +| control | 0/12 | 9/12 | 8/10 | 通过 | 600tick,航向失败 | +| augmented | 0/12 | 9/12 | 8/10 | 通过 | 561tick失稳 | + +三组原话在三个初态都发生后退后失稳;其它三说法完整稳定,但只移动约0.9–3毫米,不能当后退成功。增加同义表达的这个1000更新配方未改善严格成功;不能推断所有语言增强无效。每组只有一个optimizer seed。 + +后退训练状态两个原话anchor MSE:baseline .001509/.001914,control .001346/.002380,augmented .001297/.001697 rad²;persistence .012062/.023630。该诊断没有覆盖全部六说法,也不证明闭环。推理中位约 .25秒,超过E5的 .1秒执行窗口,暂停物理验收不能称实时。 + +后续转入 `vla-small-curriculum-2026-10-16.md`:先诊断六说法拟合,再做匹配初态的小课程。旧模型和全部失败证据保留。 diff --git a/docs/project/vla-motion-curriculum-2026-10-13.md b/docs/project/vla-motion-curriculum-2026-10-13.md new file mode 100644 index 0000000..e67e998 --- /dev/null +++ b/docs/project/vla-motion-curriculum-2026-10-13.md @@ -0,0 +1,248 @@ +# 运动课程:后退、转身与运动中停止 + +## 问题与范围 + +[v18](vla-head-actions-2026-10-13.md) 已增加若干静态头部动作,但尚未替代完整动作目录。 +本轮学习后退、左转、右转、前进中停止、后退中停止,同时回放现有六类能力。 +用户已明确继续训练。只在 MuJoCo 中实验,不访问真实硬件,不购买算力。 +复用现有课程入口 `training/experiments/head_curriculum.py --course motion`, +启动脚本为 `run-motion-curriculum.sh`;不复制训练循环或改变 SmolVLA 模型结构。 + +## 设计与预算 + +- 初始权重:v18 step2000,SHA256 `d7880c045b6bc8366fbccceef1a25883c76be5821c6f8fc47c9b9f8898932749`。 +- 保留 v18 80 episodes / 20,600 frames;从其 data-audit 读取 family,逐条核对帧数与文本, + 不能把头部数据误标为 forward。已有重复前缀保持原样,不当独立新示范。 +- 五个新场景各10条、200tick/4秒,train seeds700..709;合格示范才入训练,但保存所有原始失败。 + 每类至少8条合格,校准教师每类3/3通过才允许进入模型阶段。 +- 校准种子720..722与学生评估730..732分开。新场景每类两条训练措辞、一条预声明留出措辞; + 两种停止场景共享文字,靠身体观测区分状态。旧任务沿用已测试过的措辞,只是回归检查。 +- 若全部合格则130 episodes / 30,600 frames。仍使用原生按帧采样,forward约34.6%, + 每个2000帧任务约6.54%;4000×8/30600≈1.05次平均帧暴露,并不代表收敛。 +- state51 = 身体36 + 上一实际执行target15;绝对关节目标、50Hz、H10/E5、训练/推理均固定黑图。 + 不宣称视觉能力。归一化按新训练集合计算,float64 global及episode统计均核验。 +- 2步smoke丢弃;从同一v18权重开始4000正式更新,batch8、optimizer seed42、fresh optimizer。 + 总预算4002,不能自动追加。原生checkpoint、数据、日志、失败轨迹全部保留。 + +## 停止语义与验收 + +教师准备阶段先实际行走100ticks,并在清空原点和速度窗口前保存101份身体指标。 +除速度大小外,末秒沿预期方向的有符号位移需>0.03m且稳定;否则不能当作停止初态。 +见证按(seed, initial_twist)分组,保证同起步模式的文本/模型比较起点完全一致。 +不要求停止场景与静止起步的物理状态相同。 + +仅给停止任务移动初态,会使模型通过“遇到运动就停”获得高分。因此每个学生阶段额外包含 +continue_forward、continue_backward:和对应stop相同的物理/state/prior及噪声,输入则是 +“向前走”或“向后走”。它们是验收场景,不新增Command,也不向学习模型输入场景标签。 +每阶段3body×13场景×2措辞=78条;不是78个独立身体或训练重复。 + +训练前在校准初态比较教师停止、教师继续原命令、保持最后target。保持目标是dummy, +不自动算为有效的双足平衡控制。站立轨迹不能交叉评分成停止;不同initial_twist的目标 +不参与同一goal_scores比较。两种停止的语言意图相同,不能互当错误类别。 +这仍是“专家准备身体,再交给学生”,没有验证学生自行走动后的任务切换、取消与排队语义。 + +既有六任务判据保持不变;新增判据先用教师、错误方向和dummy校准,学生评估前锁定: + +- 后退:将沿初始朝向的forward位移反号,复用原严格前进的持续进展、侧移、heading、平衡判据。 +- 转身:累计yaw沿指令方向>0.2rad,末秒同方向增量>0.15rad,全程水平漂移<0.20m,完整稳定且尾段upright。 + 沿用Simulation累加的yaw,不把跨π后的累计转角再次wrap。 +- 停止:完整稳定、末秒平均速度<0.06m/s、全程位移<0.3m、heading<0.5rad、尾段upright; + 初态运动资格由上述preparation证据单独约束。 + +forward运行12秒且另报4秒前缀,其余200ticks。baseline使用v18原统计,zero同权重换新统计, +final训练后使用新统计。三者分别报告任务成功、平衡、方向、文本对照与推理延迟。 +同期保存所有实际模型输入和执行轨迹;物理暂停等待推理,不是实时robotd部署结果。 + +## 实验前判断与门禁 + +| 选择 | 能回答的问题 | 本轮 | +|---|---|---| +| 既有模型+新动作示范+完整replay | 扩展任务与保留旧能力能否同时发生 | 执行 | +| same-moving stop/continue配对 | 停止是否依赖语言,而不是初态捷径 | 执行 | +| 仅增大步数或修改模型结构 | 可能改善拟合,难分辨新数据语义 | 暂不执行 | +| 实时异步执行、视觉输入、任务连续切换 | 另有工程/感知问题 | 不纳入本轮 | + +主观先验:任务覆盖和训练曝光不足40%;多任务干扰及归一化变化25%;停止状态捷径15%; +训练或评分实现问题10%;未知10%。不是统计后验。若教师不合格,先定位示范或判据问题, +不依赖学生结果调门限;若stop和continue输出均停,不能称语言控制停止成功。 + +独立设计审查发现并纳入:moving证据被mark_origin清空、replay family硬编码、停止状态捷径、 +累计yaw与同初态分组。校准采用现有速度/方向尺度,转身末秒和漂移尺度仍需教师对照证据。 +CPU测试初次失败来自横移fixture设置了恒定坐标偏移、并非横向运动;修正成随时间增长的位移。 +没有因此放宽判据。校准完成、数据审计、真实smoke通过后才进入正式4000步。 + +工件计划:`/srv/duckmind-training/v19-owner/run`,开始前根盘54GiB可用,data盘227GiB可用, +L4空闲。复用既有venv/HF cache,源代码保持同home相对路径;不删除旧数据或模型。 +当前PR5仍open,继续在其工作树内扩展课程,不自动合并。 + +## ml-debug 运行表(预运行) + +| 项目 | 证据 / 待验 | +|---|---| +| 完整日志/配置 | 待生成prepare/run日志,课程参数在resolved config落盘 | +| SHOULD | 同start mode初态精确一致;真实moving方向;train-only统计;黑图;4002预算 | +| null与尺度 | v18为baseline,新增stop/continue/hold-target教师对照;不编造随机成功率 | +| 零更新 | baseline与zero对照待执行;新stats可能改变行为 | +| persistence | 按manifest每任务首帧/20tick训练诊断;不是验证集或身体闭环 | +| 留出集 | train700..709、calibration720..722、eval730..732;旧措辞不是新语言验收 | +| lr/梯度 | 原生warmup100与decay4000,实际日志待填,逐模块梯度未知 | +| 完整sample | 待保存真实state/prior/黑图/chunk及每tick物理轨迹 | +| 意外情况 | CPU fixture错误已修,真实教师失败尚待观察 | +| 缺口 | 单optimizerseed、有限措辞,未验证真实时间、视觉和连续任务切换 | +| 竞争解释 | 见上方先验与配对检查 | +| 新上下文审查 | 上述五项设计风险已纳入,代码及数据独立审计待完成 | +| 最便宜区分 | 校准同moving初态停止/继续/保持target,学生同初态改文本 | +| 耗时/显存 | 待实际日志,不将显存容量当作使用峰值 | + +## 起步失败与零训练修复 + +首次 prepare 在 seed701 / stop_backward 的起步检查退出1,9条已完成示范均通过; +原日志、原始轨迹与初次独立审计保留在 `v19-owner/prepare-attempt1/`,没有优化器更新。 +异常原先没有保存 preparation 指标,现通过 PreparationError 携带完整起步证据, +采集入口保存失败 JSON;不合格训练候选记入分母并剔除,校准失败仍阻断训练。 + +使用未修改物理算法的 Simulation.prepare 重复 seed701 两次,101份指标完全一致: +末秒有符号后退进度 -0.000100819m、平均速度0.0213567m/s,全程稳定;seed700对照 +后退进度0.168852m、速度0.207903m/s。证据 `moving-start-reproduction.json`。 +这表明是老师未进入运动,而非跌倒;为何同一命令在该初态不启动仍未知,不能归因学生。 + +课程入口新增 `check-starts`,只调用老师检查32个预声明起步条件,无学生推理、无训练: + +| split | 前进初态 | 后退初态 | +|---|---:|---:| +| train700..709 | 10/10 | 9/10,701无效 | +| calibration720..722 | 3/3 | 3/3 | +| evaluation730..732 | 3/3 | 2/3,732无效 | + +732同样稳定但未起步,signed进度-0.000673m。没有更换种子或降低门限。 +每个学生阶段仍保留78个预定case,其中732的stop_backward和continue_backward各两条措辞 +共4条记入 `invalid_starts`,不调用学生、不计作策略失败;其余74条为合格物理验收分母。 +必须同时报告78预定/74可评估,后退moving任务只有2个身体初态。初态预检已触碰评估身体, +但未据学生结果调参;不能描述为完全未触碰状态。任意后续异常仍需保存,不能隐藏剔除。 + +这次修复改变的是证据与候选处理,不改变模型、物理、准备时长、Command或成功门限。 +独立审查也发现evaluate原先会在相同无效起步中断,现以单列记录保持完整分母。 +`make fix/lint/test`通过(17通过、2跳过)。第二次prepare已启动,源码冻结; +完整教师校准和数据审计尚待完成,4002次总预算仍未使用。 + +ml-debug修订:当前错误与训练代码/优化器无关(0更新);是起步候选不合格与异常证据缺失。 +竞争解释仍包括teacher局部吸引域、命令幅度敏感、初态扰动影响,尚未区分; +对照701原地稳定与700实际后退足以拒绝无效停止数据,不足以判定老师网络内部原因。 +最小修复是保留拒绝证据并继续检查剩余候选,不用放宽门限制造成功。 + +## 准备完成与启动 + +第二次prepare退出0。最终129 episodes / 30,400 frames:旧80/20,600完全保留, +新候选50条中49条/9,800帧被接受。独立CPU审计8项全通过,远端报告 +`/srv/duckmind-training/v19-owner/independent-preflight-audit.json`。 +审计逐帧比对state/action/text/previous_target;独立重算258组episode与2组global数值统计, +校验zero的6个有效processor张量与相同初始权重,解码30,400黑图帧,核对6个数据文件、 +21个源文件、teacher/upstream SHA。25组moving初态精确对齐各停止/继续/对照首帧。 + +教师新任务校准每类3/3通过;继续前后走正例各3/3,继续和保持target误判停止均0/6。 +旧forward教师12秒严格0/3、4秒3/3:12秒均稳定且前进,但heading失败,末yaw约-0.68rad。 +这界定了旧教师长时直线能力,不能包装为新任务校准失败,也不降低原门限。 + +2026-10-13 06:39 UTC(北京时间14:39)启动唯一train-evaluate,PID621889。 +实际run.stage=smoke-train,入口源码commit13a1afdf69f6aa62fb569db6d9a869a750b13ad4。 +先2步smoke及13场景各10tick,再baseline/zero;通过后自动4000正式更新、final和hashcheck。 +目前是流程已启动,不是正式训练已完成或新动作已学会。native日志和run.stage/pid/exit-code +位于工件run下,模型/data/trace全保留,自动跟进已恢复。 + +ml-debug阶段证据:prepare日志266,200字节、退出0;三类SHOULD(真moving、train-only统计、 +黑图/时序)已经上述独立检查,预算/身体模型结果仍待后续。完整教师样本可检查 +`run/teacher/train-700-backward.npz`与report对应row;无效样本保留在 +`run/teacher/train-701-stop_backward-preparation-failed.json`。prepare没有优化、梯度和lr, +这些项目不适用;smoke/正式训练需读取各自原生日志。准备/审计为CPU与渲染工作,未测峰值显存, +不得补造使用量。独立审查未发现剩余启动阻断,仍不能由教师成功推出学生会学会。 + +## 最终结果(2026-10-13) + +本轮已完成,runner `run.stage=complete`、`run.exit-code=0`,进程退出。 +精确预算为2次丢弃的smoke更新+4000正式更新;未追加训练、推理或物理rollout。 +最终模型: +`/srv/duckmind-training/v19-owner/run/model/checkpoints/004000/pretrained_model`, +SHA256 `04f21127aa546d1a8e5adbb7ec9ea25584fa720471c5c10b03757e7f74da0837`。 + +**结论:停止有局部改善,后退与转身仍未可靠掌握。工程检查通过不等于动作平替成功。** + +| 阶段 | 严格任务通过↑ | 全程稳定↑ | 预定 / 可评估 / 无效初态 | +|---|---:|---:|---:| +| final | **41/74** | **59/74** | 78 / 74 / 4 | +| baseline(旧权重旧统计) | 34/74 | 56/74 | 78 / 74 / 4 | +| zero(旧权重新统计) | 25/74 | 52/74 | 78 / 74 / 4 | + +这是一个optimizer seed、三个身体初态的描述性比较,不是独立重复训练的统计显著性结论。 +改变统计即可使34降到25,所以不能把baseline→final所有差异归为训练本身, +也不能把final→zero差异归因为新示范单独的因果效应:本轮还进行了4000次参数更新。 + +### 新动作与语言 + +| 任务 | 训练原话通过↑ | 本轮留出措辞通过↑ | 合计↑ | +|---|---:|---:|---:| +| 前进中停止 | 2/3 | 3/3 | **5/6** | +| 后退中停止 | 1/2 | 2/2 | **3/4** | +| 左转身 | 1/3 | 0/3 | 1/6 | +| 后退 | 0/3 | 0/3 | 0/6 | +| 右转身 | 0/3 | 0/3 | 0/6 | + +后退moving只有两个有效身体初态。seed732未真正后退,其stop_backward与continue_backward +两种措辞共4个slot被保留为invalid_starts,0次学生预测,不是策略失败,也没有替换种子。 + +原话与留出措辞的失败方式不同:后退原话3条均失稳,其中2条达到持续方向进展; +左右转身原话均有指令方向的累计转动,但左转只有1/3稳定、右转0/3稳定。 +相应留出措辞往往保持稳定,却没有达到目标方向。这里只描述轨迹, +尚不能证明是语言表征、有限数据、闭环分布偏移或多任务干扰中的哪一种原因。 + +停止不能单看单项分数。相同moving物理/state/prior与噪声,只换“停止/继续走”: +final仅3/10组同时通过各自目标并互不通过对方目标(baseline2/10、zero1/10), +且三个成功配对均来自向前初态。继续向前4/6,继续后退0/4。 +所以不能宣称已经可靠掌握开放文字的行走/停止切换; +这也没有验证学生自己走动后更换任务、取消或动作队列行为。 + +### 旧动作回归 + +五个静态任务合计仍27/30,与同初态baseline相同: +站立、左看、右看、低头各6/6;抬头原话3/3、旧评估措辞0/3。 +旧措辞已经使用过,是回归检查,不能包装为全新语言泛化。 + +向前走4秒严格成功baseline2/6→final4/6; +12秒严格成功0/6→1/6,全程稳定2/6→5/6。仍存在偏航和一次失稳, +不能由稳定性改善推断长时间直线行走达标。 +老师校准的12秒严格结果也是0/3、4秒3/3,保留原判据与老师边界。 + +### 工程与独立复核 + +独立CPU审计通过,完整报告在工件父目录: +- `independent-final-audit.json` +- `independent-completed-training-audit.json` +- `independent-final-ml-debug.md`与`independent-final-ml-debug-form.json` + +重查三个phase全部222条轨迹、42,827个执行动作、8,583份实际黑图输入; +独立重算222个quality、36个4秒子评分、1,416个交叉目标评分,以及90条头部任务的gaze。 +所有phase首帧物理/state/prior匹配,执行目标等于对应预测块实际前5步, +previous_target无未来泄漏;六个有效processor张量分别匹配各自统计; +21个源码文件、6个数据文件和模型SHA一致。数据内容审计复用了之前结果, +但重新核验其SHA;轨迹与评分本次全部重新计算。 + +完整日志5,472行、正式400条metrics;native训练配置seed42、batch8、freshoptimizer, +500..4000共8个正式checkpoint,唯一两次trainer分别2/4000。 +正式训练约18分21秒(含加载),日志allocated memory峰值2.33GiB, +不是整张显卡或reserved峰值。step10 loss0.531,step4000 loss0.058; +没有中间物理能力评估,不能从loss确定哪一步学会动作。逐模块梯度未知。 + +最终3156次推理中位数249ms、p95 298ms;所有调用都超过5×20ms=100ms执行窗口。 +本实验暂停物理等待推理,不能当作实时robotd闭环控制证据。输入始终黑图,无视觉能力结论。 +训练集22个chunk诊断中模型MSE优于persistence为16/22(baseline12/22、zero13/22), +不是留出集或任务成功率;raw、fit.npz、实际输入和所有失败证据均保留。 + +### 研究判断与后续边界 + +本轮证据支持“多任务继续训练带来部分能力变化”,不支持“已平替老师其它Command”。 +后续优先聚焦一个失败动作(建议后退),分别定位**训练原话闭环失稳**和 +**留出表达未触发目标方向**,先从已保存的同初态预测块与逐关节误差分析开始。 +两者可能需要不同的数据/训练处理,不能直接认为“多训几步”或“多加改写”就够。 + +完整ml-debug表、null/teacher/zero对照、竞争解释与局限见上述独立报告。 +未新增实验验证这些原因;不由这一份数据与单次训练否定语言条件策略路线。 +本轮预算已用尽,暂停自动跟进;不自行启动v20,不自动合并PR5。 diff --git a/docs/project/vla-small-curriculum-2026-10-16.md b/docs/project/vla-small-curriculum-2026-10-16.md new file mode 100644 index 0000000..17d2d7b --- /dev/null +++ b/docs/project/vla-small-curriculum-2026-10-16.md @@ -0,0 +1,63 @@ +# 小课程语言拟合与闭环诊断 + +用户授权继续“站立、前进、后退”小数据充分拟合。先用 v26 零更新诊断区分文本条件拟合与闭环执行,之后再冻结小课程预算。 + +## 已有证据 + +v25 已结束,独立最终审计 3749 项零不匹配。baseline/control/augmented 后退严格成功均 0/12;各有 9/12 全程稳定,但同义表达通常仅移动毫米。原话会后退后倒地。旧任务成功分别 7/10、8/10、8/10。1000 更新只抽取 8000 个样本,约占复制后的 42000 训练帧 0.19 遍;这不证明欠拟合是唯一原因,也不足以否定语言增强。 + +原始证据:devbox `/srv/duckmind-training/v25-owner/independent-final-audit.json` 与 `.md`,完整训练和物理记录在同目录 `run/`。仅同步仿真,无实时或视觉能力结论。 + +## v26:六种训练说法是否拟合 + +复用 head_curriculum.py 的诊断入口,保持 v24 参数和路径兼容。v21 原始教师 700/702 与第一条已接受纠偏轨迹各取 0/20/60/140 tick,12 个已见训练状态。六种 v25 后退文本、站立和前进两个异目标对照,2 个相同 flow noise,比较 v25 初始模型与 augmented 最终模型,共 386 次推理(含每模型一次精确重复),零训练、零物理推进。 + +保存物理 state、实际 prior、教师目标、所有 native inputs、RNG、预测、时延。分原始/纠偏、起步/持续报告腿关节 E5/H10 误差、persistence、同义文本与随机噪声差异。两个对照文本没有对应教师目标,仅评估动作分离,不能把其后退误差当任务分数。模型统计仍为 v21 full train 统计。 + +预测:若同义表达的教师状态误差仍明显分化,小课程优先加强语言绑定;若教师状态相近但闭环失败,则后续应检查状态偏离。两者可并存,不能将离线误差等同走路成功。 + +## 小课程约束 + +下一阶段仅三任务,匹配身体初态,完整轨迹按训练/评估拆分。图像保持黑图,state51/H10/E5、原生 LeRobot 训练器不变。所有文本的首段、持续段与教师接管状态分别检查,防止 previous_target 的惯性续写掩盖语言无效。训练预算、统计选择与 checkpoint 存储在启动前固定。当前尚未启动新训练。 + + +## v27 冻结方案 + +采用v25 augmented最终权重。训练初态700/730,各采stand/forward/backward200tick;三任务共享相同reset physics/state/prior。6条真实轨迹各配6文本,36episode/7200帧;每task同曝光权重。老师须严格通过且不被误判为另外两任务;失败保留raw并中止,不调阈值。评估770/771不进训练、不提前教师筛选,旧报告body/seed字段检索未命中。 + +LeRobot0.6.2 fresh warmstart原生会重建processor统计,没有独立normalization-reference选项。使用新dataset真实float64统计,zero只换统计、保持v25权重;zero和正式模型共享新统计。不能把v25到zero变化当学习效果。 + +预算2步smoke丢弃+4000正式,batch8/H10/E5/seed42,optimizer重启,不追加;原生save_freq1000,保留1000/2000/3000/4000。4个正式+smoke+zero约9.2GiB,启动前root18GiB可用,不删除旧工件。4000步约4.44次采样遍历,不预称充分拟合。 + +zero与1000/2000/4000分别对每条轨迹六个时刻0/5/10/20/60/140、全部六文本、两个flow noise进行432次训练状态推理;每次保存state/prior/target/prediction与文本/噪声索引。这是分期anchor诊断,不是全帧验收。zero/1000/2000各训练初态三任务原话6条,final四初态三任务原话/第六种训练说法/留出说法36条;全部200tick。新初态只有final,不能报告pre/post改进。不同阶段冻结物理噪声;一optimizer seed不构成训练重复。smoke为首训练初态三任务各10tick。 + +教师采集和学生推理均复用现有Simulation/Trace/outcome,学生仅使用文本条件模型,无规则分派。独立设计审查未发现启动阻断;源码检查training lint/test为17 passed/2 skipped,Linux真实训练链路待smoke。完整新数据审计之后才启动训练。入口run-small-curriculum.sh,工件 `/srv/duckmind-training/v27-owner/run`。 + + +## v26 已完成 + +run.stage=complete、exit0,386推理,零更新/零物理推进。独立CPU审计5874项通过、零差异,报告 `/srv/duckmind-training/v26-owner/independent-final-audit.json/.md`。六种训练文本在原轨迹起步/持续、纠偏接管早/晚四组的腿部E5与H10教师误差全部低于baseline。汇总E5 MSE(12状态×2噪声,rad²): + +| 文本 | baseline | augmented | +|---|---:|---:| +| 向后走 | .001458 | .000908 | +| 往后退一点 | .001291 | .000767 | +| 朝身后退着走 | .002637 | .001065 | +| 向后迈步 | .001626 | .000873 | +| 往身后走 | .001252 | .000826 | +| 往后退步 | .001650 | .000935 | + +残差集中在原轨迹起步:augmented六说法E5 MSE约.001365–.002182;持续段约.000372–.000423。朝身后退着走起步.003671→.002182,持续.001624→.000423,纠偏早段.003520→.000680。不能据此断言同义表达完全没有学到;下一步检查共同初态中的语言选择与闭环恢复。 + +这里没有v25等预算control,只能归因整套继续训练配方,不能独立证明文本增强收益。12个状态均来自训练轨迹,纠偏early是接管后早段,不是静止起步。未测新语言/新初态,未推进身体,也没有实时结论。Native输入快照是同路径独立preprocess调用,非predict内部hook。 + +v27准备已完成:6/6教师严格成功,错误目标无误通过;36ep/7200帧已生成,zero权重不变。正在独立preflight,尚未启动正式训练。代码提交78e9b5644c663740732be57ccf0ca53aa062b920已推到PR5,remote一致。devloop完成提交推送后因过期action-chunks路径postcheck退出1,不能称整个脚本成功;training与policy验证已通过。 + + +## v27 运行记录 + +独立preflight最终8440检查通过;CPU native loader首8与全部36episode尾部、6教师raw、36副本、全部7200黑图、episode/global统计、6zero有效张量与哈希已核实。审计自身前两次sampler调用/epoch使用错误保留attempt1/2,修正的是审计脚本,未修改数据或训练实现。准备后root可用16.11GiB。 + +2026-10-16 16:08(Asia/Shanghai)启动唯一launcher PID74446,run.pid已匹配,smoke完成2次实际更新,155个optimizer参数step均2;真实num_workers2通过。三任务各完整稳定10tick,实际输入/previous_target/chunk前5步与关节执行逐项吻合,独立smoke88checks通过。报告在owner `independent-smoke-audit.json/.md`。随后进入zero-fit;正式4000更新在zero-fit/zero-eval后自动开始,尚不能报正式完成。 + +完整日志:`/srv/duckmind-training/v27-owner/run/run.log`。状态看同目录run.stage/run.exit-code,禁止重复启动。源码快照在owner/source-snapshot;当前仅smoke2更新已证实,总预算4002。代码与teacher/data冻结,不自动扩新实验或重新启用自动跟进。正式结果完成后仍需独立审核,不能从loss或smoke推断成功。 diff --git a/docs/project/vla-text-noise-2026-10-15.md b/docs/project/vla-text-noise-2026-10-15.md new file mode 100644 index 0000000..5be94d7 --- /dev/null +++ b/docs/project/vla-text-noise-2026-10-15.md @@ -0,0 +1,111 @@ +# 固定观测的语言与采样噪声对照(v24) + +## 问题与选择 + +用既有v21 control/mixed模型区分措辞影响与Flow Matching采样波动。v23的闭环分岔同时 +包含身体状态变化,本轮直接固定教师观测,复用SmolPolicy.predict及原生processor; +不更新权重、不执行仿真,不把离线动作差异称为新动作能力。 + +| 选项 | 可区分问题 | 本轮 | +|---|---|---| +| 固定观测,多措辞×噪声 | 措辞效应和同措辞采样波动的相对尺度 | 执行 | +| 同义文本增广训练 | 数据配方是否改善语言条件步态 | 留待本轮证据 | +| 改归一化或动作平滑 | 同时改变输入或执行动力学 | 不执行 | + +## 预声明设计 + +- 两模型、同v21 union统计;原始教师train700和calibration720各取tick0/60/140。 +- 文本:训练过的“向后走”“往后退一点”;旧开发说法“朝身后退着走”; + 新诊断说法“向后迈步”“往身后走”;对照“保持站立”“向前走”。 +- 噪声seed24001..24004在模型/状态/文本间配对。6状态×7文本×4seed×2模型=336调用; + 每模型首样本精确重复一次,共338调用、0优化更新、0物理步。不恢复定时跟进。 +- H10、51维state含实际previous_target、黑图256×256。保存完整预测、教师未来目标、 + 原始与native预处理输入、token/mask、CPU/CUDA调用前后RNG状态、延迟及文件hash。 +- 原始轨迹经pair-config定位,核验prior[t]=action[t-1];全部样本H10完整无padding。 + 重复烟测、形状/有限性、同输入非语言张量、有效processor和工件身份是解释前提。 + +对10个腿关节分别报告前5步/全部10步:同噪声文本RMS差、同文本跨噪声RMS差、 +教师MSE及保持上一目标的persistence MSE。前5种表达是同目标组;站立/前进仅报告分离度, +不以它们对后退标签的误差评分。逐状态保留结果,不能把调用数或噪声seed当独立训练重复。 +两个新增说法用于本轮后不再作为最终未触碰语言测试。 + +## 竞争预测与解释边界 + +若同噪声下文本差异重复出现且大于同文本噪声波动,支持进一步做语言覆盖配对实验; +若主要是同文本跨seed波动,优先研究生成/执行交互。五种后退表达内部差异较大并非好事; +目标是同义表达一致、不同意图区分。仅有动作距离或较小教师MSE均不证明后退闭环成功。 +全为后退教师状态,状态或prior捷径仍是可能解释;需要后续物理和跨任务观测验证。 + +独立设计复核要求区分上述两种距离、保留六个状态明细、记录不消耗随机数的RNG证据, +不把四个噪声seed或相邻tick当泛化样本。本实现遵循该要求。 + +入口training/experiments/head_curriculum.py diagnose-text-noise,启动器run-text-noise-curriculum.sh; +工件/srv/duckmind-training/v24-owner/run。完成后追加原始指标、完整ml-debug表和独立复核。 + + +## 完成结果 + +run.stage=complete、exit0;338次调用、0更新、0物理步,耗时184.96秒。 +两个模型各168格点及一次精确重复;首样本同输入输出完全一致。 +下表为六个固定观测、四个采样seed在前5步/10腿关节上的汇总。 +动作差异是同seed相对“向后走”的RMS;噪声尺度是同文本六对seed差异的RMS, +先对平方量求平均再开根号。原话相对自身差异恒为0,不能解释为更好。 + +| 文本 | control差异rad | control噪声rad | mixed差异rad | mixed噪声rad | +|---|---:|---:|---:|---:| +| 向后走(参照) | 0 | 0.02482 | 0 | 0.02468 | +| 往后退一点(训练) | 0.00690 | 0.02262 | 0.00759 | 0.02395 | +| 朝身后退着走(旧开发) | 0.03660 | 0.01786 | 0.03613 | 0.01803 | +| 向后迈步(本轮新诊断) | 0.01475 | 0.02253 | 0.01630 | 0.02275 | +| 往身后走(本轮新诊断) | 0.01253 | 0.02311 | 0.01389 | 0.02283 | +| 保持站立(异目标对照) | 0.10296 | 0.01185 | 0.10364 | 0.01206 | +| 向前走(异目标对照) | 0.17113 | 0.02712 | 0.17153 | 0.02940 | + +旧开发表达在每个模型的6/6观测上,配对文本差异都大于该表达的跨seed噪声尺度。 +两个新表达的整体差异则低于自身噪声尺度;不能概括为“所有未见措辞都不会”。 +站立/前进分离明显,排除了在这些输入上完全不使用语言的解释;分离不等于正确执行。 + +| 后退表达 | control教师MSE↓ rad² | mixed教师MSE↓ rad² | +|---|---:|---:| +| 向后走 | 0.000690 | 0.000644 | +| 往后退一点 | 0.000601 | 0.000644 | +| 朝身后退着走 | 0.001764 | 0.001836 | +| 向后迈步 | 0.000795 | 0.000879 | +| 往身后走 | 0.000877 | 0.000912 | +| 保持上一实际目标(数值基线) | 0.071619 | 0.071619 | + +这是后退教师状态的离线拟合,不是身体闭环验证;较低MSE可以与v23中的失稳并存。 +四个噪声seed不是四次训练重复,三个tick也不是三个独立episode。两种checkpoint的方向一致 +增强了该观察的可信度,但它们有共同训练来源,不能宣称总体统计显著。 + +中位推理延迟control248.8ms、mixed231.8ms;分模型含证据写盘耗时68.69/64.57秒, +CUDA峰值allocated约1.263/1.264GB,不含全部设备占用。该延迟仍不支持20ms实时逐步推理。 + +## 下一步决策 + +优先做等预算文本覆盖对照:两组使用相同轨迹、初始权重、归一化、优化步数与任务采样, +一组维持现有表达,另一组让同一轨迹对应多种语义一致的措辞,保留旧能力回放。 +旧开发说法可以成为训练材料,但不能再作为语言泛化验收;预先留出其它说法和初态, +同时检查原话步态、同义表达、方向/平衡及其它动作遗忘。学习仍直接输出Action Chunk, +不增加关键词到Command的分派。 + +如果增强表达改善离线一致性却不改善闭环,继续定位恢复与执行;不把语言覆盖当作唯一根因。 +本轮尚未启动该训练或自动跟进。完整H10和逐状态结果保存在report.json,原始输出可独立重算。 + + +## 独立复核与交付 + +独立CPU审计1060项通过:两组各168格点和84项指标全部重算,原始teacher时序、 +首样本精确重复、完整日志、模型/源码/原始数据hash一致。相同seed的CPU/CUDA调用前后 +RNG状态在所有状态、文本、模型之间相同,四个CUDA种子状态互异;这是随机状态证据, +没有声称直接保存原生sample_noise张量。原生噪声空间为H10×32,解码输出H10×15。 + +每状态的非语言预处理张量固定、两模型全部native张量一致。七段token序列互异, +有效长度7/8/15/8/10/9/7均小于48,解码保留完整文本,未发现截断或碰撞。 +预处理快照通过与predict相同的processor路径另行生成,未截取predict内部batch; +该取证边界保留,不能将独立快照描述为内部hook记录。 + +本地training的fix/lint通过,test为17passed、2skipped;真实GPU调用与独立数组复算 +验证本轮新增诊断。完整表单、原始输入样本、逐状态结果和复核脚本在 +/srv/duckmind-training/v24-owner/independent-text-noise-audit.json/md及相邻审计工件。 +PR5继续开放,不合并。 diff --git a/training/experiments/head_curriculum.py b/training/experiments/head_curriculum.py new file mode 100644 index 0000000..1bf4cdb --- /dev/null +++ b/training/experiments/head_curriculum.py @@ -0,0 +1,2659 @@ +"""Bounded language-action curricula with replay and matched text trials. + +Run prepare first; inspect its teacher/data audit before train-evaluate. Native LeRobot +owns the optimizer and checkpoint format. This driver owns the experiment and evidence. +""" + +import argparse +import json +import os +import shutil +import subprocess +import sys +import time +from dataclasses import asdict, dataclass, replace +from pathlib import Path + +import numpy as np +import torch +from duckmind_policy.schema import JOINT_NAMES +from lerobot.datasets.lerobot_dataset import LeRobotDataset + +from duckmind_training.collect import sha256 +from duckmind_training.contract import ( + IMAGE_KEY, + PREVIOUS_TARGET_STATE_NAMES, + TASKS, + Task, + policy_state, +) +from duckmind_training.correction_quality import ( + locomotion_outcome, + recovery_outcome, + stationary_head_outcome, +) +from duckmind_training.corrections import Trace, expert_rollout +from duckmind_training.dataset import precise_numeric_stats +from duckmind_training.learned import SmolPolicy +from duckmind_training.simulation import PreparationError, Simulation + + +@dataclass(frozen=True) +class Curriculum: + name: str + new_tasks: tuple[str, ...] + tasks: tuple[str, ...] + train_seeds: tuple[int, ...] + calibration_seeds: tuple[int, ...] + eval_seeds: tuple[int, ...] + unseen: dict[str, str] + formal_steps: int + replay_episodes: int + replay_frames: int + + +HEAD_TASKS = ("stand", "look_left", "look_right", "look_up", "look_down", "forward") +HEAD_UNSEEN = dict( + zip( + HEAD_TASKS, + ( + "站在原地保持不动", + "脑袋朝左边转一点", + "脑袋朝右边转一点", + "抬起脑袋看看上面", + "垂下脑袋看看下面", + "朝着正前方迈步前行", + ), + strict=True, + ) +) +MOTION_TASKS = ("backward", "turn_left", "turn_right", "stop_forward", "stop_backward") +COURSES = { + "heads": Curriculum( + "v18-heads", + HEAD_TASKS[:-1], + HEAD_TASKS, + tuple(range(600, 610)), + (620, 621, 622), + (620, 621, 622), + HEAD_UNSEEN, + 2000, + 30, + 10600, + ), + "motion": Curriculum( + "v19-motion", + MOTION_TASKS, + HEAD_TASKS + MOTION_TASKS + ("continue_forward", "continue_backward"), + tuple(range(700, 710)), + (720, 721, 722), + (730, 731, 732), + { + **HEAD_UNSEEN, + "backward": "朝身后退着走", + "turn_left": "让身体朝左边转过去", + "turn_right": "让身体朝右边转过去", + "stop_forward": "结束走动,停在这里", + "stop_backward": "结束走动,停在这里", + "continue_forward": "朝着正前方迈步前行", + "continue_backward": "朝身后退着走", + }, + 4000, + 80, + 20600, + ), +} +SELECTED = {task.name: task for task in TASKS} +for direction in ("forward", "backward"): + moving = SELECTED[direction] + SELECTED["continue_" + direction] = replace( + moving, name="continue_" + direction, initial_twist=moving.twist + ) +HEAD_KEYS = ("neck_pitch_delta", "head_pitch_delta", "head_yaw_delta", "head_roll_delta") +H, E = 10, 5 +BLACK = np.zeros((256, 256, 3), dtype=np.uint8) + + +def write(path: Path, value: object) -> None: + path.write_text(json.dumps(value, ensure_ascii=False, indent=2)) + + +def note(root: Path, message: str) -> None: + print(message, flush=True) + with (root / "run.md").open("a") as stream: + stream.write(message + "\n") + + +def dataset(root: Path, *, chunks: bool = False) -> LeRobotDataset: + return LeRobotDataset( + "duckmind/" + root.name, + root=root, + video_backend="pyav", + delta_timestamps={"action": [i / 50 for i in range(H)]} if chunks else None, + ) + + +def outcome(task: Task, metrics: list[dict], steps: int, gaze: np.ndarray) -> dict: + if task.name in ("forward", "continue_forward"): + result = recovery_outcome(metrics, steps) + result["quality_4s"] = recovery_outcome(metrics[:201], 200) + return result + if task.name == "continue_backward": + return locomotion_outcome(replace(task, name="backward"), metrics, steps) + if task.name in MOTION_TASKS: + return locomotion_outcome(task, metrics, steps) + result = stationary_head_outcome(task, metrics, steps) + # Optical-axis motion prevents compensating joints from faking a look direction. + shift = gaze[-50:] - gaze[0] + shift = np.arctan2(np.sin(shift), np.cos(shift)) + expected = (task.head[2], -task.head[1]) # azimuth left+, elevation up+ + gaze_checks = [ + bool(np.mean(shift[:, i]) * np.sign(v) > 0.1) + if v + else bool(np.mean(np.abs(shift[:, i])) < 0.15) + for i, v in enumerate(expected) + ] + result["gaze"] = { + "tail_shift_azimuth_elevation": shift.mean(0).tolist(), + "axis_pass": gaze_checks, + } + result["accepted"] = result["accepted"] and all(gaze_checks) + return result + + +def initial_witness(root: Path, sim: Simulation, seed: int, task: Task, preparation: dict) -> None: + folder = root / "initial" + folder.mkdir(exist_ok=True) + state, _ = sim.observe() + values = { + "state": state, + "previous_target": sim.previous_target(), + "physics": sim.integration_state(), + } + # Stop tasks must share the same moving reset across texts/models, not a standing reset. + group = "-moving-" + str(task.initial_twist[0]) if any(task.initial_twist) else "" + path = folder / f"{seed}{group}.npz" + prep_path = path.with_suffix(".json") + if prep_path.exists(): + assert json.loads(prep_path.read_text()) == preparation + else: + write(prep_path, preparation) + if path.exists(): + with np.load(path) as saved: + for key, value in values.items(): + np.testing.assert_array_equal(value, saved[key]) + else: + np.savez_compressed(path, **values) + + +def processor_audit(checkpoint: Path, data: Path) -> dict: + from safetensors.torch import load_file + + stats = dataset(data).meta.stats + config = json.loads((checkpoint / "train_config.json").read_text()) + assert config["dataset"]["image_transforms"]["enable"] is False + observed = {} + for filename, registry, keys in ( + ("policy_preprocessor.json", "normalizer_processor", ("observation.state", "action")), + ("policy_postprocessor.json", "unnormalizer_processor", ("action",)), + ): + steps = json.loads((checkpoint / filename).read_text())["steps"] + matches = [s for s in steps if s["registry_name"] == registry] + assert len(matches) == 1 + step = matches[0] + assert step["config"]["norm_map"] == { + "VISUAL": "IDENTITY", + "STATE": "MEAN_STD", + "ACTION": "MEAN_STD", + } + values = load_file(checkpoint / step["state_file"]) + for key in keys: + for stat in ("mean", "std"): + name = f"{key}.{stat}" + actual = values[name].cpu() + torch.testing.assert_close( + actual, torch.tensor(stats[key][stat], dtype=actual.dtype), rtol=0, atol=0 + ) + observed[f"{registry}:{name}"] = actual.tolist() + return observed + + +def source_witness() -> dict[str, str]: + training = Path(__file__).resolve().parents[1] + files = [ + Path(__file__), + *training.joinpath("experiments").glob("run-*-curriculum.sh"), + *training.joinpath("src/duckmind_training").glob("*.py"), + *training.parent.joinpath("policy/src/duckmind_policy").glob("*.py"), + training / "requirements.txt", + ] + return {str(p.resolve()): sha256(p) for p in files if p.is_file()} + + +def prepare(args: argparse.Namespace) -> None: + course = args.course + root = args.root + root.mkdir(exist_ok=False, parents=True) + assert shutil.disk_usage(root).free > 20 * 2**30 + write( + root / "config.json", + { + **{k: str(v) for k, v in vars(args).items()}, + "curriculum": asdict(course), + "tasks": [asdict(SELECTED[n]) for n in course.tasks], + "unseen": course.unseen, + "train_seeds": course.train_seeds, + "calibration_seeds": course.calibration_seeds, + "eval_seeds": course.eval_seeds, + "budget_updates": course.formal_steps + 2, + "formal_updates": course.formal_steps, + "optimizer_seed": 42, + "batch_size": 8, + "horizon": H, + "execute_steps": E, + "fixed_image": True, + "physics_paused": True, + "source": sha256(Path(__file__)), + "source_files": source_witness(), + }, + ) + note( + root, + f"# {course.name} curriculum\n" + "SHOULD: same initial physics/state/prior within each start mode; text is the task cue.\n" + "SHOULD: teacher and null/wrong-goal scale before training; preserve every failure.\n" + "SHOULD: training-only float64 stats, black images, state51, H10/E5; " + f"2 smoke +{course.formal_steps} formal updates maximum.\n", + ) + raw = root / "teacher" + raw.mkdir() + report = { + "stage": "started", + "episodes": [], + "evaluation": [], + "teacher_sha256": sha256(args.teacher), + "upstream_sha256": sha256(args.upstream / "scripts/infer_policy.py"), + } + numeric = [] + source = dataset(args.replay) + # Replay retains episode labels as well as every numerical frame and language string. + replay_audit = args.replay.parent / "data-audit.json" + families = ( + [e["family"] for e in json.loads(replay_audit.read_text())["episodes"]] + if course.name == "v19-motion" + else ["forward"] * source.num_episodes + ) + assert len(families) == source.num_episodes + for ep in range(source.num_episodes): + rows = source.hf_dataset.filter(lambda row, ep=ep: row["episode_index"] == ep) + if course.name == "v19-motion": + declared = json.loads(replay_audit.read_text())["episodes"][ep] + assert declared["frames"] == len(rows) + assert declared["text"] == str(source.meta.tasks.iloc[int(rows[0]["task_index"])].name) + numeric.append( + { + "observation.state": np.asarray(rows["observation.state"]), + "action": np.asarray(rows["action"]), + "texts": [str(source.meta.tasks.iloc[int(i)].name) for i in rows["task_index"]], + "family": families[ep], + } + ) + assert sum(len(e["action"]) for e in numeric) == course.replay_frames + assert source.num_episodes == course.replay_episodes and source.meta.fps == 50 + assert tuple(source.features["observation.state"]["names"]) == PREVIOUS_TARGET_STATE_NAMES + assert tuple(source.features["action"]["names"]) == JOINT_NAMES + sim = Simulation(args.upstream, args.teacher) + try: + for split, seeds in [ + ("train", course.train_seeds), + ("calibration", course.calibration_seeds), + ]: + for seed in seeds: + for name in course.tasks if split == "calibration" else course.new_tasks: + task = SELECTED[name] if split == "calibration" else SELECTED[name].sample(seed) + steps = 600 if name == "forward" else 200 + identity = f"{split}-{seed}-{name}" + try: + preparation = sim.prepare(task, seed) + except PreparationError as error: + write(raw / f"{identity}-preparation-failed.json", error.evidence) + row = { + "id": identity, + "seed": seed, + "name": name, + "split": split, + "steps": 0, + "command": asdict(task), + "quality": {"accepted": False, "reason": "invalid_initial_state"}, + "preparation_error": str(error), + } + report["episodes" if split == "train" else "evaluation"].append(row) + write(raw / "report.json", report) + note(root, f"Teacher {identity}: {error}") + if split != "train": + raise + continue + initial_witness(root, sim, seed, task, preparation) + sim.command(task) + trace = expert_rollout(sim, steps) + identity = f"{split}-{seed}-{name}" + trace.save(raw / f"{identity}.npz") + gaze = trace_gaze(sim, trace.physics) + quality = outcome(task, trace.metrics, steps, gaze) + row = { + "id": identity, + "seed": seed, + "name": name, + "split": split, + "steps": len(trace.actions), + "command": asdict(task), + "quality": quality, + } + # Judge each outcome against all goals to expose wrong-goal shortcuts. + row["goal_scores"] = { + n: outcome(SELECTED[n], trace.metrics, steps, gaze)["accepted"] + for n in course.tasks + if n != "forward" and SELECTED[n].initial_twist == task.initial_twist + } + report["episodes" if split == "train" else "evaluation"].append(row) + write(raw / "report.json", report) + note( + root, + f"Teacher {identity}: ticks={len(trace.actions)} " + f"accepted={quality['accepted']}", + ) + if split == "train" and quality["accepted"]: + states = policy_state( + np.asarray(trace.states), + np.asarray(trace.previous_targets), + include_previous_target=True, + ) + actions = np.asarray(trace.actions) + np.testing.assert_array_equal(states[1:, 36:], actions[:-1]) + numeric.append( + { + "observation.state": states, + "action": actions, + "texts": [task.texts[(seed - course.train_seeds[0]) % 2]] + * len(actions), + "family": name, + } + ) + if course.name == "v19-motion": + controls = [] + for seed in course.calibration_seeds: + for name in ("stop_forward", "stop_backward"): + task = SELECTED[name] + for mode in ("continue", "hold_target"): + preparation = sim.prepare(task, seed) + initial_witness(root, sim, seed, task, preparation) + if mode == "continue": + sim.command(replace(task, twist=task.initial_twist)) + trace = expert_rollout(sim, 200) + else: + trace = Trace() + trace.mark(sim) + target = sim.previous_target().copy() + for _ in range(200): + if not sim.metrics()["stable"]: + break + trace.step(sim, target) + identity = f"control-{seed}-{name}-{mode}" + trace.save(raw / f"{identity}.npz") + quality = outcome(task, trace.metrics, 200, trace_gaze(sim, trace.physics)) + controls.append({"id": identity, "mode": mode, "quality": quality}) + write(raw / "stop-controls.json", controls) + note(root, f"Stop control {identity}: {quality!r}") + assert all( + not row["quality"]["accepted"] for row in controls if row["mode"] == "continue" + ) + finally: + sim.close() + required_calibration = course.new_tasks + ( + ("continue_forward", "continue_backward") if course.name == "v19-motion" else () + ) + for name in course.new_tasks: + assert ( + sum(r["quality"]["accepted"] and r["name"] == name for r in report["episodes"]) >= 8 + ), name + for name in required_calibration: + rows = [r for r in report["evaluation"] if r["name"] == name] + assert len(rows) == len(course.calibration_seeds) and all( + r["quality"]["accepted"] for r in rows + ), name + # Standing should never score as any of the four head commands. + assert all( + not r["goal_scores"][n] + for r in report["evaluation"] + if r["name"] == "stand" + for n in HEAD_TASKS[1:5] + ) + report["stage"] = "complete" + write(raw / "report.json", report) + prepare_data(args, numeric) + + +def prepare_data(args: argparse.Namespace, numeric: list[dict]) -> None: + import av + from lerobot.configs.video import RGBEncoderConfig + from lerobot.policies.factory import make_pre_post_processors + from lerobot.policies.smolvla.configuration_smolvla import SmolVLAConfig + + root, course = args.root, args.course + data_path = root / "dataset" + features = { + "observation.state": { + "dtype": "float32", + "shape": (51,), + "names": list(PREVIOUS_TARGET_STATE_NAMES), + }, + "action": {"dtype": "float32", "shape": (15,), "names": list(JOINT_NAMES)}, + IMAGE_KEY: { + "dtype": "video", + "shape": (256, 256, 3), + "names": ["height", "width", "channels"], + }, + } + ds = LeRobotDataset.create( + "duckmind/" + course.name, + fps=50, + features=features, + root=data_path, + robot_type="microduck", + use_videos=True, + image_writer_threads=4, + video_backend="pyav", + rgb_encoder=RGBEncoderConfig(vcodec="h264", preset="fast", crf=23), + ) + try: + for episode in numeric: + for state, action, text in zip( + episode["observation.state"], episode["action"], episode["texts"], strict=True + ): + assert np.isfinite(state).all() and np.isfinite(action).all() + ds.add_frame( + { + "observation.state": state, + "action": action, + IMAGE_KEY: BLACK.copy(), + "task": text, + } + ) + ds.save_episode() + finally: + ds.finalize() + precise_numeric_stats( + data_path, ds, [{k: e[k] for k in ("observation.state", "action")} for e in numeric] + ) + loaded = dataset(data_path, chunks=True) + for key in ("observation.state", "action"): + values = np.concatenate([e[key] for e in numeric]) + np.testing.assert_array_equal(np.asarray(loaded.hf_dataset[key]), values) + for stat, expected in [ + ("mean", values.astype(float).mean(0)), + ("std", values.astype(float).std(0)), + ]: + np.testing.assert_allclose(loaded.meta.stats[key][stat], expected, rtol=1e-6, atol=1e-8) + actual_texts = [ + str(loaded.meta.tasks.iloc[int(i)].name) for i in loaded.hf_dataset["task_index"] + ] + assert actual_texts == [text for e in numeric for text in e["texts"]] + assert not set(course.unseen.values()).intersection(actual_texts) + offset = 0 + for e in numeric: + length = len(e["action"]) + item = loaded[offset + length - 1] + np.testing.assert_array_equal(item["action_is_pad"].numpy(), [False] + [True] * 9) + np.testing.assert_array_equal( + item["action"].numpy(), np.repeat(e["action"][-1:], 10, axis=0) + ) + rows = loaded.hf_dataset.select(range(offset, offset + length)) + np.testing.assert_allclose(np.asarray(rows["timestamp"]), np.arange(length) / 50, atol=1e-6) + offset += length + decoded = 0 + for path in data_path.rglob("*.mp4"): + with av.open(str(path)) as stream: + for frame in stream.decode(video=0): + assert not frame.to_ndarray(format="rgb24").any() + decoded += 1 + assert decoded == offset + zero = root / "zero-checkpoint" + shutil.copytree(args.initial, zero) + cfg = SmolVLAConfig.from_pretrained(str(args.initial)) + pre, post = make_pre_post_processors( + cfg, + pretrained_path=str(args.initial), + dataset_stats=loaded.meta.stats, + preprocessor_overrides={ + "device_processor": {"device": "cuda"}, + "normalizer_processor": { + "features": {**cfg.input_features, **cfg.output_features}, + "norm_map": cfg.normalization_mapping, + "stats": loaded.meta.stats, + }, + "rename_observations_processor": {"rename_map": {}}, + }, + postprocessor_overrides={ + "unnormalizer_processor": { + "features": cfg.output_features, + "norm_map": cfg.normalization_mapping, + "stats": loaded.meta.stats, + } + }, + ) + pre.save_pretrained(zero) + post.save_pretrained(zero) + assert sha256(zero / "model.safetensors") == sha256(args.initial / "model.safetensors") + write( + root / "data-audit.json", + { + "episodes": [ + {"family": e["family"], "frames": len(e["action"]), "text": e["texts"][0]} + for e in numeric + ], + "frames": offset, + "black_frames": decoded, + "zero_processors": processor_audit(zero, data_path), + "initial_sha256": sha256(args.initial / "model.safetensors"), + "source_sha256": {str(p): sha256(p) for p in args.replay.rglob("*") if p.is_file()}, + "dataset_sha256": {str(p): sha256(p) for p in data_path.rglob("*") if p.is_file()}, + }, + ) + write( + root / "image-contract.json", + {"shape": [256, 256, 3], "dtype": "uint8", "value": 0, "train_and_infer": True}, + ) + note( + root, + f"Prepared {len(numeric)} episodes/{offset} frames. " + "Split provenance remains in the experiment manifest. No optimizer updates yet.", + ) + + +def evaluate( + args: argparse.Namespace, + checkpoint: Path, + phase: str, + *, + smoke: bool = False, + execute_steps: int = E, + texts: dict[str, list[tuple[str, str]]] | None = None, + forward_steps: int = 600, +) -> None: + assert 1 <= execute_steps <= H + course = args.course + root = args.root + out = root / phase + out.mkdir(exist_ok=False) + reference = args.replay if phase == "baseline" else root / "dataset" + report = { + "phase": phase, + "execute_steps": execute_steps, + "stage": "started", + "trials": [], + "invalid_starts": [], + "physics_paused": True, + "model_sha256": sha256(checkpoint / "model.safetensors"), + "processors": processor_audit(checkpoint, reference), + } + model = SmolPolicy(checkpoint) + # Train-set diagnostics only: compare identical states/text/noise with persistence. + ds = dataset(root / "dataset", chunks=True) + episodes = json.loads((root / "data-audit.json").read_text())["episodes"] + samples, seen, offset = [], set(), 0 + for episode in episodes: + if episode["family"] not in seen: + samples.extend([(episode["family"], offset), (episode["family"], offset + 20)]) + seen.add(episode["family"]) + offset += episode["frames"] + predictions, targets, persistence, inputs = [], [], [], [] + for _family, index in samples: + item = ds[index] + state = item["observation.state"].numpy() + assert not item[IMAGE_KEY].any() + torch.manual_seed(18000000 + index) + prediction = model.predict(item["task"], state[:36], BLACK, state[36:]) + predictions.append(prediction) + targets.append(item["action"].numpy()) + persistence.append(np.repeat(state[None, 36:], H, axis=0)) + inputs.append(state) + np.savez_compressed( + out / "fit.npz", + prediction=predictions, + target=targets, + persistence=persistence, + state=inputs, + indices=[i for _, i in samples], + ) + report["fit_train_only"] = [ + { + "task": family, + "index": index, + "model_mse_rad2": float(np.mean((prediction - target) ** 2)), + "persistence_mse_rad2": float(np.mean((prior - target) ** 2)), + } + for (family, index), prediction, target, prior in zip( + samples, predictions, targets, persistence, strict=True + ) + ] + language_cases = ( + texts + if texts is not None + else { + name: ( + [("trained", SELECTED[name].texts[0])] + if smoke + else [("trained", SELECTED[name].texts[0]), ("unseen", course.unseen[name])] + ) + for name in course.tasks + } + ) + assert set(language_cases) == set(course.tasks) + for cases in language_cases.values(): + assert cases and len({label for label, _ in cases}) == len(cases) + assert all(label.replace("-", "").isalnum() for label, _ in cases) + seeds = course.eval_seeds[:1] if smoke else course.eval_seeds + sim = Simulation(args.upstream, args.teacher) + try: + for seed in seeds: + for name in course.tasks: + task = SELECTED[name] + for language, text in language_cases[name]: + identity = f"{seed}-{name}-{language}" + try: + preparation = sim.prepare(task, seed) + except PreparationError as error: + write(out / f"{identity}-preparation-failed.json", error.evidence) + report["invalid_starts"].append( + { + "id": identity, + "body": seed, + "task": name, + "language": language, + "text": text, + "reason": "invalid_initial_state", + "error": str(error), + "student_predictions": 0, + } + ) + write(out / "report.json", report) + note(root, f"{phase} {identity}: invalid_initial_state: {error}") + if smoke: + raise + continue + initial_witness(root, sim, seed, task, preparation) + # Do not set teacher command: learned text is the sole task input. + trace = Trace() + trace.mark(sim) + saved = { + "tick": [], + "state": [], + "previous_target": [], + "image": [], + "chunk": [], + "latency_s": [], + } + identity = f"{seed}-{name}-{language}" + steps = 10 if smoke else forward_steps if name == "forward" else 200 + reason = "completed" + try: + for tick in range(steps): + if not sim.metrics()["stable"]: + reason = "unstable" + break + if tick % execute_steps == 0: + state, _ = sim.observe() + prior = sim.previous_target() + torch.manual_seed(seed * 10000 + tick) + start = time.perf_counter() + chunk = model.predict(text, state, BLACK, prior) + elapsed = time.perf_counter() - start + for k, value in [ + ("tick", tick), + ("state", state), + ("previous_target", prior), + ("image", BLACK.copy()), + ("chunk", chunk), + ("latency_s", elapsed), + ]: + saved[k].append(value) + assert chunk.shape == (H, 15) and np.isfinite(chunk).all() + action = chunk[tick % execute_steps] + if np.any(np.abs(action) > np.pi): + reason = "invalid_target" + break + trace.step(sim, action) + finally: + trace.save(out / f"{identity}.npz") + np.savez_compressed( + out / f"{identity}-inputs.npz", + **{k: np.asarray(v) for k, v in saved.items()}, + ) + gaze = trace_gaze(sim, trace.physics) + quality = outcome(task, trace.metrics, steps, gaze) + if not trace.metrics[-1]["stable"]: + reason = "unstable" + row = { + "id": identity, + "body": seed, + "task": name, + "language": language, + "text": text, + "ticks": len(trace.actions), + "reason": reason, + "quality": quality, + "stable": len(trace.actions) == steps + and all(m["stable"] for m in trace.metrics), + "goal_scores": { + n: outcome(SELECTED[n], trace.metrics, steps, gaze)["accepted"] + for n in course.tasks + if n != "forward" and SELECTED[n].initial_twist == task.initial_twist + }, + "latency_median_s": float(np.median(saved["latency_s"])), + } + report["trials"].append(row) + write(out / "report.json", report) + note( + root, + f"{phase} {identity}: ticks={row['ticks']} " + f"accepted={quality['accepted']} stable={row['stable']}", + ) + report["scheduled_cases"] = len(seeds) * sum( + len(cases) for cases in language_cases.values() + ) + report["eligible_cases"] = len(report["trials"]) + assert report["scheduled_cases"] == len(report["trials"]) + len(report["invalid_starts"]) + if smoke: + assert len(report["trials"]) == report["scheduled_cases"] + assert all(r["ticks"] == 10 and r["reason"] == "completed" for r in report["trials"]), ( + report["trials"] + ) + report["stage"] = "complete" + write(out / "report.json", report) + finally: + sim.close() + del model + torch.cuda.empty_cache() + + +def run(args: argparse.Namespace) -> None: + root = args.root + prepared = json.loads((root / "config.json").read_text()) + assert prepared["curriculum"] == json.loads(json.dumps(asdict(args.course))), ( + "prepared curriculum differs" + ) + assert prepared["source"] == sha256(Path(__file__)), "source changed since prepare" + assert prepared["source_files"] == source_witness(), "runtime source changed since prepare" + for key in ("replay", "initial", "upstream", "teacher"): + assert prepared[key] == str(getattr(args, key)), f"prepared {key} differs" + assert not (root / "run.pid").exists(), "run already started; inspect before any recovery" + (root / "run.pid").write_text(str(os.getpid())) + audit = json.loads((root / "data-audit.json").read_text()) + witness = source_witness() + write(root / "run-source-sha256.json", witness) + assert all(sha256(Path(p)) == value for p, value in audit["dataset_sha256"].items()) + assert sha256(args.initial / "model.safetensors") == audit["initial_sha256"] + + def stage(name: str) -> None: + (root / "run.stage").write_text(name) + note(root, f"## {name}") + + def train(steps: int, label: str) -> Path: + command = [ + sys.executable, + "-c", + "from duckmind_training.train import main; main()", + "--dataset", + str(root / "dataset"), + "--model", + str(args.initial), + "--output", + str(root / label), + "--steps", + str(steps), + "--batch-size", + "8", + "--chunk-size", + str(H), + ] + note(root, f"Native trainer argv: {command!r}") + subprocess.run(command, check=True) + checkpoint = root / label / "checkpoints" / f"{steps:06d}" / "pretrained_model" + processor_audit(checkpoint, root / "dataset") + return checkpoint + + try: + stage("smoke-train") + smoke = train(2, "model-smoke") + stage("smoke-evaluate") + evaluate(args, smoke, "smoke", smoke=True) + stage("evaluate-baseline") + evaluate(args, args.initial, "baseline") + stage("evaluate-zero") + evaluate(args, root / "zero-checkpoint", "zero-eval") + stage("train") + final = train(args.course.formal_steps, "model") + stage("evaluate-final") + evaluate(args, final, "final") + stage("verify") + assert all(sha256(Path(p)) == value for p, value in witness.items()) + assert all(sha256(Path(p)) == value for p, value in audit["dataset_sha256"].items()) + assert sha256(args.initial / "model.safetensors") == audit["initial_sha256"] + stage("complete") + except Exception as error: + note(root, f"Failed: {error!r}") + (root / "run.exit-code").write_text("1\n") + raise + (root / "run.exit-code").write_text("0\n") + + +def trace_gaze(sim: Simulation, physics: list | np.ndarray) -> np.ndarray: + """Recompute corrected camera optical axis in the trunk frame, without mutating the body.""" + import mujoco + + data = mujoco.MjData(sim.model) + body = mujoco.mj_name2id(sim.model, mujoco.mjtObj.mjOBJ_BODY, "trunk_base") + assert body >= 0 + angles = [] + for state in physics: + mujoco.mj_setState(sim.model, data, state, mujoco.mjtState.mjSTATE_INTEGRATION) + mujoco.mj_forward(sim.model, data) + ray = -data.cam_xmat[sim.camera.camera].reshape(3, 3)[:, 2] + relative = data.xmat[body].reshape(3, 3).T @ ray + angles.append( + [ + float(np.arctan2(relative[1], relative[0])), + float(np.arctan2(relative[2], np.hypot(*relative[:2]))), + ] + ) + return np.asarray(angles) + + +def calibrate_gaze(args: argparse.Namespace) -> None: + """Use saved physics to calibrate task meaning before looking at student scores.""" + sim = Simulation(args.upstream, args.teacher) + rows = [] + try: + paths = sorted((args.root / "teacher").glob("calibration-*.npz")) + if not paths: + paths = sorted((args.root / "teacher").glob("eval-*.npz")) + if not paths: + raise ValueError("no saved calibration trajectories") + for path in paths: + with np.load(path) as saved: + angles = trace_gaze(sim, saved["integration_state"]) + rows.append( + { + "id": path.stem, + "initial_azimuth_elevation": angles[0].tolist(), + "tail_mean_azimuth_elevation": angles[-50:].mean(0).tolist(), + } + ) + write(args.root / "gaze-calibration.json", rows) + finally: + sim.close() + + +def check_starts(args: argparse.Namespace) -> None: + """Teacher-only preflight; no student predictions or optimizer updates.""" + path = args.root.parent / "start-check.json" + assert not path.exists() + course = args.course + report = {"stage": "started", "source_files": source_witness(), "cases": []} + sim = Simulation(args.upstream, args.teacher) + try: + for split, seeds in ( + ("train", course.train_seeds), + ("calibration", course.calibration_seeds), + ("evaluation", course.eval_seeds), + ): + for seed in seeds: + for name in ("stop_forward", "stop_backward"): + try: + evidence = sim.prepare(SELECTED[name], seed) + error = None + except PreparationError as failure: + evidence, error = failure.evidence, str(failure) + report["cases"].append( + { + "split": split, + "seed": seed, + "task": name, + "error": error, + "preparation": evidence, + } + ) + write(path, report) + print(f"Start {split}/{seed}/{name}: {error or 'valid'}", flush=True) + report["stage"] = "complete" + write(path, report) + finally: + sim.close() + + +def diagnose_backward(args: argparse.Namespace) -> None: + """Separate on-expert imitation, text sensitivity and closed-loop execution. + + spec: weights, normalization, body state and flow noise stay fixed across texts. + Task-name lookup supplies diagnostic labels only; the policy receives plain text. + link: docs/project/vla-backward-diagnosis-2026-10-13.md + """ + root, source = args.root, args.replay.parent + root.mkdir(parents=True, exist_ok=False) + (root / "run.pid").write_text(str(os.getpid())) + (root / "dataset").symlink_to(args.replay) + (root / "data-audit.json").symlink_to(source / "data-audit.json") + witness = source_witness() + checkpoint_sha = sha256(args.initial / "model.safetensors") + texts = (*SELECTED["backward"].texts, args.course.unseen["backward"], "保持站立", "向前走") + episodes = [f"train-{seed}-backward" for seed in (700, 701, 702)] + [ + f"calibration-{seed}-backward" for seed in (720, 721, 722) + ] + ticks = (0, 20, 60, 100, 140, 180) + report = { + "stage": "started", + "optimizer_updates": 0, + "argv": sys.argv, + "paths": { + k: str(getattr(args, k).resolve()) + for k in ("root", "replay", "initial", "upstream", "teacher") + }, + "text_roles": [ + "trained", + "trained", + "never-trained-catalog", + "v19-held-out", + "wrong-goal-stand", + "wrong-goal-forward", + ], + "model_sha256": checkpoint_sha, + "source_files": witness, + "processors": processor_audit(args.initial, args.replay), + "texts": texts, + "episodes": episodes, + "ticks": ticks, + "rows": [], + "artifact_sha256": {}, + } + note( + root, + "# Backward diagnosis\nSHOULD: 0 updates; identical state/prior/noise across texts.\n" + "SHOULD: 6 episodes × 6 ticks × 6 texts = 216 paired predictions.\n" + "Training/calibration/development states only; no new holdout claim.", + ) + model = SmolPolicy(args.initial) + predictions, targets, states, priors = [], [], [], [] + try: + (root / "run.stage").write_text("offline") + for episode_index, identity in enumerate(episodes): + path = source / "teacher" / f"{identity}.npz" + report["artifact_sha256"][str(path)] = sha256(path) + with np.load(path) as raw: + np.testing.assert_array_equal(raw["previous_target"][1:], raw["action"][:-1]) + for tick in ticks: + state, prior = raw["state"][tick], raw["previous_target"][tick] + target = raw["action"][tick : tick + H] + assert target.shape == (H, 15) + noise = 20000000 + episode_index * 1000 + tick + # Capture native processor outputs for one matched input per text. + if episode_index == 0 and tick == 0: + batches = {} + for text_index, text in enumerate(texts): + batch = model.preprocess( + { + "observation.state": torch.from_numpy( + policy_state(state, prior, include_previous_target=True) + ), + IMAGE_KEY: torch.from_numpy(BLACK.copy()) + .permute(2, 0, 1) + .float(), + "task": text, + } + ) + for key, value in batch.items(): + if isinstance(value, torch.Tensor): + batches[f"{text_index}:{key}"] = value.cpu().numpy() + np.savez_compressed(root / "native-inputs.npz", **batches) + for text_index, text in enumerate(texts): + torch.manual_seed(noise) + chunk = model.predict(text, state, BLACK, prior) + if not predictions: + torch.manual_seed(noise) + repeated = model.predict(text, state, BLACK, prior) + np.testing.assert_array_equal(chunk, repeated) + report["same_input_smoke"] = "exact" + predictions.append(chunk) + targets.append(target) + states.append(state) + priors.append(prior) + error = (chunk - target) ** 2 + persistence = np.repeat(prior[None], H, axis=0) + row = { + "episode": identity, + "tick": tick, + "text_index": text_index, + "noise_seed": noise, + "first_mse": float(error[0].mean()), + "executed5_mse": float(error[:E].mean()), + "chunk_mse": float(error.mean()), + "per_joint_mse": error.mean(0).tolist(), + "persistence_mse": float(((persistence - target) ** 2).mean()), + } + report["rows"].append(row) + note(root, f"Offline {identity}: {len(report['rows'])}/216 predictions saved.") + write(root / "report.json", report) + np.savez_compressed( + root / "offline.npz", + prediction=predictions, + target=targets, + state=states, + previous_target=priors, + ) + assert len(predictions) == 216 + del model + torch.cuda.empty_cache() + (root / "run.stage").write_text("closed-loop") + # Reuse the native evaluation/Trace path; no second simulator/controller implementation. + args.course = replace(args.course, tasks=("backward",), eval_seeds=(700, 730, 731)) + evaluate(args, args.initial, "reproduce") + comparisons = [] + for seed in (730, 731): + for language in ("trained", "unseen"): + identity = f"{seed}-backward-{language}" + for suffix in (".npz", "-inputs.npz"): + old, new = ( + source / "final" / (identity + suffix), + root / "reproduce" / (identity + suffix), + ) + report["artifact_sha256"][str(old)] = sha256(old) + with np.load(old) as a, np.load(new) as b: + # Wall time/rendered evidence frames may vary; policy input is always black. + keys = ( + ("tick", "state", "previous_target", "image", "chunk") + if suffix.startswith("-") + else ( + "state", + "action", + "previous_target", + "integration_state", + "actuator_target", + ) + ) + comparisons.append( + { + "id": identity, + "file": suffix, + "exact": { + key: bool(np.array_equal(a[key], b[key])) for key in keys + }, + } + ) + report["reproduction"] = comparisons + assert sha256(args.initial / "model.safetensors") == checkpoint_sha + assert source_witness() == witness + assert all(sha256(Path(p)) == digest for p, digest in report["artifact_sha256"].items()) + report["stage"] = "complete" + exact = all(all(c["exact"].values()) for c in comparisons) + note( + root, + "Complete: 0 updates; 216 paired predictions +1 same-input smoke; " + f"6 closed-loop cases. Reproduction exact: {exact}", + ) + (root / "run.stage").write_text("complete") + (root / "run.exit-code").write_text("0") + except BaseException: + report["stage"] = "failed" + (root / "run.exit-code").write_text("1") + raise + finally: + write(root / "report.json", report) + + +def prepare_backward_corrections(args: argparse.Namespace) -> None: + """Collect teacher recovery labels from exactly replayed student prefixes.""" + root = args.root + if args.diagnosis_root is None: + raise ValueError("--diagnosis-root is required for backward corrections") + root.mkdir(parents=True, exist_ok=False) + assert shutil.disk_usage(root).free > 20 * 2**30 + (root / "prepare.pid").write_text(str(os.getpid())) + source = dataset(args.replay) + original = json.loads((args.replay.parent / "data-audit.json").read_text()) + numeric = [] + for index, ep in enumerate(original["episodes"]): + rows = source.hf_dataset.filter(lambda row, index=index: row["episode_index"] == index) + assert len(rows) == ep["frames"] + numeric.append( + { + "observation.state": np.asarray(rows["observation.state"]), + "action": np.asarray(rows["action"]), + "texts": [str(source.meta.tasks.iloc[int(i)].name) for i in rows["task_index"]], + "family": ep["family"], + } + ) + assert len(numeric) == 129 and sum(len(e["action"]) for e in numeric) == 30400 + control_episodes = [i for i, e in enumerate(numeric) if e["family"] == "backward"] + assert len(control_episodes) == 10 + raw = root / "teacher" + raw.mkdir() + report = {"stage": "started", "candidates": [], "controls": [], "source_sha256": {}} + task = SELECTED["backward"] + sim = Simulation(args.upstream, args.teacher) + note( + root, + "# Backward focused correction pair\n" + "SHOULD: exact student-prefix replay; teacher-only labels after handover.\n" + "SHOULD: retain all rejected windows; unchanged strict backward gate.\n" + "SHOULD: 2+1000 updates per arm, maximum2004, shared union normalization.\n" + "This is a backward-only diagnostic; old-task regression is required.", + ) + try: + for seed in (700, 730, 740, 741, 742): + sim.prepare(task, seed) + sim.command(task) + trace = expert_rollout(sim, 200) + trace.save(raw / f"control-{seed}.npz") + quality = locomotion_outcome(task, trace.metrics, 200) + report["controls"].append({"seed": seed, "quality": quality}) + write(raw / "report.json", report) + assert quality["accepted"], (seed, quality) + for seed in (700, 730): + path = args.diagnosis_root / "reproduce" / f"{seed}-backward-trained.npz" + report["source_sha256"][str(path)] = sha256(path) + with np.load(path) as saved: + # First prove the entire recorded failure is reproducible before selecting prefixes. + sim.prepare(task, seed) + np.testing.assert_array_equal( + sim.integration_state(), saved["integration_state"][0] + ) + for tick, action in enumerate(saved["action"]): + sim.apply(action) + np.testing.assert_array_equal( + sim.integration_state(), saved["integration_state"][tick + 1] + ) + assert not sim.metrics()["stable"] + for prefix in (20, 50, 80): + sim.prepare(task, seed) + for action in saved["action"][:prefix]: + sim.apply(action) + before = sim.integration_state().copy() + np.testing.assert_array_equal(before, saved["integration_state"][prefix]) + np.testing.assert_array_equal( + sim.previous_target(), saved["action"][prefix - 1] + ) + assert sim.metrics()["stable"] + sim.resume_teacher(task) + np.testing.assert_array_equal(before, sim.integration_state()) + trace = expert_rollout(sim, 200) + identity = f"recovery-{seed}-{prefix}" + trace.save(raw / f"{identity}.npz") + quality = locomotion_outcome(task, trace.metrics, 200) + row = {"id": identity, "seed": seed, "prefix": prefix, "quality": quality} + report["candidates"].append(row) + write(raw / "report.json", report) + note(root, f"{identity}: ticks={len(trace.actions)} quality={quality!r}") + if quality["accepted"]: + states = policy_state( + np.asarray(trace.states), + np.asarray(trace.previous_targets), + include_previous_target=True, + ) + actions = np.asarray(trace.actions) + np.testing.assert_array_equal(states[0, 36:], saved["action"][prefix - 1]) + np.testing.assert_array_equal(states[1:, 36:], actions[:-1]) + numeric.append( + { + "observation.state": states, + "action": actions, + "texts": [task.texts[(len(numeric) - 129) % 2]] * len(actions), + "family": "backward", + } + ) + accepted = [r for r in report["candidates"] if r["quality"]["accepted"]] + assert len(accepted) >= 4 and {r["seed"] for r in accepted} == {700, 730}, accepted + report["stage"] = "complete" + write(raw / "report.json", report) + finally: + sim.close() + prepare_data(args, numeric) + finalize_backward_preparation(args) + + +def finalize_backward_preparation(args: argparse.Namespace) -> None: + """Finish provenance from saved artifacts without repeating collection or training.""" + root = args.root + assert not (root / "pair-config.json").exists() and not (root / "run.pid").exists() + report = json.loads((root / "teacher/report.json").read_text()) + audit = json.loads((root / "data-audit.json").read_text()) + accepted = [r for r in report["candidates"] if r["quality"]["accepted"]] + assert report["stage"] == "complete" and len(report["candidates"]) == 6 + assert len(accepted) >= 4 and {r["seed"] for r in accepted} == {700, 730} + assert len(audit["episodes"]) == 129 + len(accepted) + assert audit["frames"] == 30400 + 200 * len(accepted) + for key in ("source_sha256", "dataset_sha256"): + assert all(sha256(Path(p)) == value for p, value in audit[key].items()) + assert audit["initial_sha256"] == sha256(args.initial / "model.safetensors") + assert sha256(root / "zero-checkpoint/model.safetensors") == audit["initial_sha256"] + processor_audit(root / "zero-checkpoint", root / "dataset") + control_episodes = [ + i for i, e in enumerate(audit["episodes"][:129]) if e["family"] == "backward" + ] + assert len(control_episodes) == 10 + revision = subprocess.run( + ["git", "-C", str(args.upstream), "rev-parse", "HEAD"], + text=True, + capture_output=True, + check=False, + ) + if revision.returncode: + note( + root, + "Upstream Git revision unavailable; verified file hashes retain source identity: " + + revision.stderr.strip(), + ) + config = { + "argv": sys.argv, + "source_files": source_witness(), + "initial_sha256": sha256(args.initial / "model.safetensors"), + "paths": { + k: str(getattr(args, k).resolve()) + for k in ("root", "replay", "initial", "upstream", "teacher") + }, + "control_episodes": control_episodes, + "mixed_episodes": control_episodes + list(range(129, len(audit["episodes"]))), + "updates_per_arm": {"discarded_smoke": 2, "formal": 1000}, + "max_updates": 2004, + "eval_backward_bodies": [700, 730, 740, 741, 742], + "regression_body": 730, + "physics_paused": True, + "external_sha256": { + str(p): sha256(p) + for p in ( + args.teacher, + args.upstream / "scripts/infer_policy.py", + args.upstream / "src/mjlab_microduck/robot/microduck/scene.xml", + ) + }, + "upstream_commit": revision.stdout.strip() if revision.returncode == 0 else None, + "upstream_git_error": revision.stderr.strip() if revision.returncode else None, + } + write(root / "pair-config.json", config) + (root / "prepare.exit-code").write_text("0") + note( + root, + f"Preparation complete; {len(accepted)}/6 accepted across2 development bodies. " + "No student inference or optimizer update during preparation.", + ) + + +def run_backward_pair(args: argparse.Namespace) -> None: + """Native episode selection isolates focused fitting from added recovery coverage.""" + root = args.root + cfg = json.loads((root / "pair-config.json").read_text()) + assert cfg["source_files"] == source_witness() + assert cfg["initial_sha256"] == sha256(args.initial / "model.safetensors") + assert all(sha256(Path(p)) == value for p, value in cfg["external_sha256"].items()) + audit = json.loads((root / "data-audit.json").read_text()) + assert all(sha256(Path(p)) == value for p, value in audit["dataset_sha256"].items()) + assert not (root / "run.pid").exists() + (root / "run.pid").write_text(str(os.getpid())) + course = args.course + + def stage(name: str) -> None: + (root / "run.stage").write_text(name) + note(root, f"## {name}") + + def train(label: str, steps: int, episodes: list[int]) -> Path: + command = [ + sys.executable, + "-c", + "from duckmind_training.train import main; main()", + "--dataset", + str(root / "dataset"), + "--model", + str(args.initial), + "--output", + str(root / label), + "--steps", + str(steps), + "--batch-size", + "8", + "--chunk-size", + str(H), + "--episodes", + *map(str, episodes), + ] + note(root, f"Native trainer argv: {command!r}") + subprocess.run(command, check=True) + checkpoint = root / label / "checkpoints" / f"{steps:06d}" / "pretrained_model" + processor_audit(checkpoint, root / "dataset") + return checkpoint + + def evaluate_pair(checkpoint: Path, label: str) -> None: + args.course = replace( + course, tasks=("backward",), eval_seeds=tuple(cfg["eval_backward_bodies"]) + ) + evaluate(args, checkpoint, label + "-backward") + args.course = replace(course, tasks=HEAD_TASKS, eval_seeds=(cfg["regression_body"],)) + evaluate(args, checkpoint, label + "-regression") + + try: + for arm in ("control", "mixed"): + stage("smoke-" + arm) + smoke = train("smoke-" + arm, 2, cfg[arm + "_episodes"]) + args.course = replace(course, tasks=("backward",), eval_seeds=(700,)) + evaluate(args, smoke, "smoke-eval-" + arm, smoke=True) + stage("zero") + evaluate_pair(root / "zero-checkpoint", "zero") + for arm in ("control", "mixed"): + stage("train-" + arm) + checkpoint = train("model-" + arm, 1000, cfg[arm + "_episodes"]) + stage("evaluate-" + arm) + evaluate_pair(checkpoint, arm) + stage("verify") + assert cfg["source_files"] == source_witness() + assert cfg["initial_sha256"] == sha256(args.initial / "model.safetensors") + assert all(sha256(Path(p)) == value for p, value in audit["dataset_sha256"].items()) + stage("complete") + (root / "run.exit-code").write_text("0") + except BaseException: + (root / "run.exit-code").write_text("1") + raise + + +def diagnose_correction_fit(args: argparse.Namespace) -> None: + """Compare decoded predictions on the exact v21 training states, with no updates. + + spec: pair state, text, target, normalization and flow noise across checkpoints. + link: docs/project/vla-correction-fit-2026-10-14.md + """ + root, source = args.root, args.replay.parent + root.mkdir(parents=True, exist_ok=False) + (root / "run.pid").write_text(str(os.getpid())) + witness = source_witness() + audit = json.loads((source / "data-audit.json").read_text()) + pair = json.loads((source / "pair-config.json").read_text()) + assert (source / "run.exit-code").read_text().strip() == "0" + assert len(audit["episodes"]) == 134 and audit["frames"] == 31400 + assert all(sha256(Path(p)) == value for p, value in audit["dataset_sha256"].items()) + old_ids = pair["control_episodes"] + correction_ids = sorted(set(pair["mixed_episodes"]) - set(old_ids)) + assert len(old_ids) == 10 and correction_ids == list(range(129, 134)) + checkpoints = { + "zero": source / "zero-checkpoint", + "control": source / "model-control/checkpoints/001000/pretrained_model", + "mixed": source / "model-mixed/checkpoints/001000/pretrained_model", + } + model_hashes = {name: sha256(path / "model.safetensors") for name, path in checkpoints.items()} + ds = dataset(args.replay, chunks=True) + ticks = (0, 20, 60, 100, 140, 180) + offsets = np.cumsum([0] + [e["frames"] for e in audit["episodes"]]) + original_root = Path(pair["paths"]["replay"]).parent + original_report = json.loads((original_root / "teacher/report.json").read_text()) + original_rows = [ + r + for r in original_report["episodes"] + if r["name"] == "backward" and r["quality"]["accepted"] + ] + correction_report = json.loads((source / "teacher/report.json").read_text()) + correction_rows = [r for r in correction_report["candidates"] if r["quality"]["accepted"]] + raw_paths = { + **{ + episode: original_root / "teacher" / (row["id"] + ".npz") + for episode, row in zip(old_ids, original_rows, strict=True) + }, + **{ + episode: source / "teacher" / (row["id"] + ".npz") + for episode, row in zip(correction_ids, correction_rows, strict=True) + }, + } + raw_hashes = {str(path): sha256(path) for path in raw_paths.values()} + samples, states, targets, images = [], [], [], [] + for episode in old_ids + correction_ids: + assert audit["episodes"][episode]["family"] == "backward" + assert audit["episodes"][episode]["frames"] == 200 + with np.load(raw_paths[episode]) as raw: + raw_state = raw["state"].copy() + raw_prior = raw["previous_target"].copy() + raw_action = raw["action"].copy() + np.testing.assert_array_equal(raw_prior[1:], raw_action[:-1]) + expected_text = ( + SELECTED["backward"].texts[old_ids.index(episode) % 2] + if episode in old_ids + else SELECTED["backward"].texts[correction_ids.index(episode) % 2] + ) + for tick in ticks: + item = ds[int(offsets[episode]) + tick] + assert int(item["episode_index"]) == episode + assert int(item["frame_index"]) == tick and not item["action_is_pad"].any() + assert not item[IMAGE_KEY].any() + state, target = item["observation.state"].numpy(), item["action"].numpy() + assert state.shape == (51,) and target.shape == (H, 15) + np.testing.assert_array_equal(state[:36], raw_state[tick]) + np.testing.assert_array_equal(state[36:], raw_prior[tick]) + np.testing.assert_array_equal(target, raw_action[tick : tick + H]) + assert item["task"] == expected_text + samples.append( + { + "episode": episode, + "tick": tick, + "text": item["task"], + "group": "correction" if episode in correction_ids else "original", + "period": "early" if tick < 60 else "late", + "noise_seeds": [22000000 + episode * 1000 + tick * 2 + j for j in range(2)], + } + ) + states.append(state) + targets.append(target) + images.append(BLACK.copy()) + states, targets, images = np.asarray(states), np.asarray(targets), np.asarray(images) + np.savez_compressed(root / "inputs.npz", state=states, target=targets, image=images) + leg_indices = [i for i, name in enumerate(JOINT_NAMES) if name.startswith(("left_", "right_"))] + assert len(leg_indices) == 10 + report = { + "stage": "started", + "optimizer_updates": 0, + "physics_rollouts": 0, + "argv": sys.argv, + "source_files": witness, + "samples": samples, + "model_sha256": model_hashes, + "raw_sha256": raw_hashes, + "dataset_sha256": audit["dataset_sha256"], + "input_sha256": sha256(root / "inputs.npz"), + "leg_indices": leg_indices, + "inference_budget": 543, + "inference_calls": 0, + "arms": {}, + } + note( + root, + "# Correction-state fitting diagnosis\n" + "SHOULD: 0 updates/rollouts; " + "15 episodes × 6 ticks × 2 noises × 3 models + 3 repeats = 543 calls.\n" + "SHOULD: identical state, stored text, black image, noise and targets across models.\n" + "Training-state fitting only; overlapping corrections from two bodies are not holdout.", + ) + try: + for arm, path in checkpoints.items(): + (root / "run.stage").write_text(arm) + stats = processor_audit(path, args.replay) + model = SmolPolicy(path) + native = model.preprocess( + { + "observation.state": torch.from_numpy(states[0].copy()), + IMAGE_KEY: torch.from_numpy(images[0].copy()).permute(2, 0, 1).float() / 255, + "task": samples[0]["text"], + } + ) + np.savez_compressed( + root / f"{arm}-native-input.npz", + **{k: v.cpu().numpy() for k, v in native.items() if isinstance(v, torch.Tensor)}, + ) + predictions, latencies, rows = [], [], [] + arm_report = {"checkpoint": str(path), "processors": stats, "rows": rows} + report["arms"][arm] = arm_report + try: + for sample_id, sample in enumerate(samples): + state, target, image = states[sample_id], targets[sample_id], images[sample_id] + for noise in sample["noise_seeds"]: + torch.manual_seed(noise) + start = time.perf_counter() + chunk = model.predict(sample["text"], state[:36], image, state[36:]) + elapsed = time.perf_counter() - start + report["inference_calls"] += 1 + assert chunk.shape == (H, 15) and np.isfinite(chunk).all() + predictions.append(chunk) + latencies.append(elapsed) + if len(predictions) == 1: + torch.manual_seed(noise) + repeat = model.predict(sample["text"], state[:36], image, state[36:]) + report["inference_calls"] += 1 + np.savez_compressed( + root / f"{arm}-smoke.npz", first=chunk, repeated=repeat + ) + np.testing.assert_array_equal(chunk, repeat) + arm_report["same_input_smoke"] = "exact" + error = (chunk - target) ** 2 + persistence_error = (state[None, 36:] - target) ** 2 + row = { + "sample_id": sample_id, + "noise_seed": noise, + "latency_s": elapsed, + } + for name, count in (("first", 1), ("executed5", E), ("chunk", H)): + row[name + "_per_joint"] = error[:count].mean(0).tolist() + row[name + "_persistence_per_joint"] = ( + persistence_error[:count].mean(0).tolist() + ) + row[name + "_mse"] = float(error[:count].mean()) + row[name + "_legs_mse"] = float(error[:count, leg_indices].mean()) + row[name + "_persistence_mse"] = float(persistence_error[:count].mean()) + row[name + "_persistence_legs_mse"] = float( + persistence_error[:count, leg_indices].mean() + ) + rows.append(row) + if sample["tick"] == ticks[-1]: + note( + root, + f"{arm}: episode{sample['episode']}; {len(rows)}/180 predictions", + ) + write(root / "report.json", report) + finally: + np.savez_compressed( + root / f"{arm}-predictions.npz", prediction=predictions, latency_s=latencies + ) + del model + torch.cuda.empty_cache() + arm_report["prediction_sha256"] = sha256(root / f"{arm}-predictions.npz") + arm_report["summary"] = {} + for group in ("original", "correction"): + for period in ("early", "late"): + selected = [ + row + for row in rows + if samples[row["sample_id"]]["group"] == group + and samples[row["sample_id"]]["period"] == period + ] + arm_report["summary"][group + "_" + period] = { + "predictions": len(selected), + **{ + key: float(np.mean([row[key] for row in selected])) + for key in selected[0] + if key.endswith("mse") + }, + } + arm_report["per_episode"] = { + str(episode): { + key: float( + np.mean( + [ + row[key] + for row in rows + if samples[row["sample_id"]]["episode"] == episode + ] + ) + ) + for key in rows[0] + if key.endswith("mse") + } + for episode in old_ids + correction_ids + } + write(root / "report.json", report) + assert report["inference_calls"] == 543 + assert source_witness() == witness + assert all(sha256(Path(p)) == value for p, value in raw_hashes.items()) + assert all(sha256(Path(p)) == value for p, value in audit["dataset_sha256"].items()) + assert all( + sha256(checkpoints[name] / "model.safetensors") == value + for name, value in model_hashes.items() + ) + assert sha256(root / "inputs.npz") == report["input_sha256"] + report["stage"] = "complete" + (root / "run.stage").write_text("complete") + (root / "run.exit-code").write_text("0") + note( + root, + "## Complete\n543 calls, zero updates/rollouts. Decoded MSE is not physical success.", + ) + except BaseException: + report["stage"] = "failed" + (root / "run.exit-code").write_text("1") + raise + finally: + write(root / "report.json", report) + + +def diagnose_feedback(args: argparse.Namespace) -> None: + """Compare execution horizons on fixed checkpoints without optimizer updates. + + spec: hold model, normalization, initial state and common-tick flow seeds fixed. + why: E1 also changes chunk offset and sampling frequency; it is not pure feedback isolation. + link: docs/project/vla-feedback-2026-10-14.md + """ + root, source = args.root, args.replay.parent + root.mkdir(parents=True, exist_ok=False) + (root / "run.pid").write_text(str(os.getpid())) + started = time.perf_counter() + witness = source_witness() + audit = json.loads((source / "data-audit.json").read_text()) + assert (source / "run.exit-code").read_text().strip() == "0" + assert all(sha256(Path(p)) == h for p, h in audit["dataset_sha256"].items()) + (root / "dataset").symlink_to(args.replay, target_is_directory=True) + shutil.copy2(source / "data-audit.json", root / "data-audit.json") + checkpoints = { + arm: source / f"model-{arm}/checkpoints/001000/pretrained_model" + for arm in ("control", "mixed") + } + checkpoint_hashes = { + str(p): sha256(p) + for checkpoint in checkpoints.values() + for p in checkpoint.rglob("*") + if p.is_file() + } + cfg = { + "optimizer_updates": 0, + "maximum_inference_calls": 4944, + "main_rollouts": 40, + "smoke_rollouts": 2, + "max_ticks": 200, + "bodies": [700, 730, 740, 741, 742], + "execute_steps": [5, 1], + "horizon": H, + "physics_paused": True, + "fresh_holdout": False, + "argv": sys.argv, + "source_files": witness, + "checkpoint_files": checkpoint_hashes, + "dataset_sha256": audit["dataset_sha256"], + } + write(root / "config.json", cfg) + note( + root, + "# v23 execution-horizon comparison\n" + "SHOULD: identical state/prior/physics at reset and common-tick noise seeds.\n" + "SHOULD: E5 numeric traces reproduce v21; images in Trace are not model inputs.\n" + "SHOULD: 0 updates, <=4944 inference calls, 40 main +2 smoke rollouts.\n" + "E1 changes state feedback, chunk offset and flow sampling frequency together.\n", + ) + course = replace(args.course, tasks=("backward",), eval_seeds=tuple(cfg["bodies"])) + summary = {"stage": "started", "phases": [], "optimizer_updates": 0} + try: + for execute in (5, 1): + phase = f"smoke-e{execute}" + (root / "run.stage").write_text(phase) + args.course = replace(course, eval_seeds=(700,)) + evaluate(args, checkpoints["mixed"], phase, smoke=True, execute_steps=execute) + smoke = json.loads((root / phase / "report.json").read_text()) + assert all(row["stable"] for row in smoke["trials"]) + args.course = course + for execute in (5, 1): + for arm, checkpoint in checkpoints.items(): + phase = f"{arm}-e{execute}" + (root / "run.stage").write_text(phase) + torch.cuda.reset_peak_memory_stats() + phase_started = time.perf_counter() + evaluate(args, checkpoint, phase, execute_steps=execute) + report = json.loads((root / phase / "report.json").read_text()) + assert report["eligible_cases"] == 10 and not report["invalid_starts"] + if execute == 5: + # Keep E5 an exact numerical reproduction before interpreting E1. + for row in report["trials"]: + for suffix in (".npz", "-inputs.npz"): + with np.load(root / phase / (row["id"] + suffix)) as new: + with np.load( + source / (arm + "-backward") / (row["id"] + suffix) + ) as old: + for key in new.files: + if key not in ("image", "latency_s"): + np.testing.assert_array_equal(new[key], old[key]) + summary["phases"].append( + { + "phase": phase, + "accepted": sum(r["quality"]["accepted"] for r in report["trials"]), + "stable": sum(r["stable"] for r in report["trials"]), + "denominator": 10, + "elapsed_s": time.perf_counter() - phase_started, + "gpu_peak_bytes": torch.cuda.max_memory_allocated(), + "e5_reproduced": execute == 5, + } + ) + write(root / "report.json", summary) + note(root, f"Completed {phase}: {summary['phases'][-1]}") + phases = ["smoke-e5", "smoke-e1", "control-e5", "mixed-e5", "control-e1", "mixed-e1"] + calls = 0 + for phase in phases: + report = json.loads((root / phase / "report.json").read_text()) + calls += len(report["fit_train_only"]) + for path in (root / phase).glob("*-inputs.npz"): + with np.load(path) as inputs: + calls += len(inputs["tick"]) + assert calls <= cfg["maximum_inference_calls"] + assert witness == source_witness() + assert all(sha256(Path(p)) == h for p, h in checkpoint_hashes.items()) + assert all(sha256(Path(p)) == h for p, h in audit["dataset_sha256"].items()) + summary.update( + stage="complete", inference_calls=calls, elapsed_s=time.perf_counter() - started + ) + write(root / "report.json", summary) + (root / "run.stage").write_text("complete") + (root / "run.exit-code").write_text("0") + note(root, f"## Complete\n{summary}\nPhysics paused; no real-time claim.") + except BaseException: + (root / "run.exit-code").write_text("1") + raise + + +def diagnose_text_noise(args: argparse.Namespace) -> None: + """Measure text and flow-noise sensitivity on frozen teacher observations. + + spec: every text/noise pair uses identical state, actual prior and black image. + why: separate paraphrase dispersion from wrong-goal separation; neither proves motion. + link: docs/project/vla-text-noise-2026-10-15.md + """ + root, source = args.root, args.replay.parent + root.mkdir(parents=True, exist_ok=False) + (root / "run.pid").write_text(str(os.getpid())) + started = time.perf_counter() + language_fit = args.stage == "diagnose-language-fit" + pair = json.loads((source / "pair-config.json").read_text()) + teacher_root = Path(pair["paths"]["replay"]).parent / "teacher" + if language_fit: + # spec: all six v25 training paraphrases share the same teacher target. + # link: docs/project/vla-small-curriculum-2026-10-16.md + assert args.diagnosis_root is not None + assert (args.diagnosis_root / "run.exit-code").read_text().strip() == "0" + texts = (*LANGUAGE_TEXTS, "保持站立", "向前走") + roles = (*(["v25-trained"] * 6), "stand-control", "forward-control") + seeds, ticks = (26001, 26002), (0, 20, 60, 140) + accepted = [ + r + for r in json.loads((source / "teacher/report.json").read_text())["candidates"] + if r["quality"]["accepted"] + ] + raw_paths = [ + teacher_root / "train-700-backward.npz", + teacher_root / "train-702-backward.npz", + source / "teacher" / (accepted[0]["id"] + ".npz"), + ] + checkpoints = { + "baseline": args.initial, + "augmented": args.diagnosis_root + / "model-augmented/checkpoints/001000/pretrained_model", + } + else: + texts = ( + "向后走", + "往后退一点", + "朝身后退着走", + "向后迈步", + "往身后走", + "保持站立", + "向前走", + ) + roles = ( + "trained", + "trained", + "old-development", + "new-diagnostic", + "new-diagnostic", + "stand-control", + "forward-control", + ) + seeds, ticks = (24001, 24002, 24003, 24004), (0, 60, 140) + raw_paths = [ + teacher_root / (name + ".npz") + for name in ("train-700-backward", "calibration-720-backward") + ] + checkpoints = { + arm: source / f"model-{arm}/checkpoints/001000/pretrained_model" + for arm in ("control", "mixed") + } + sample_count = len(raw_paths) * len(ticks) + text_count, noise_count = len(texts), len(seeds) + backward_count = text_count - 2 + call_budget = len(checkpoints) * (sample_count * text_count * noise_count + 1) + hashes = {str(p): sha256(p) for p in raw_paths} + hashes.update( + {str(p): sha256(p) for c in checkpoints.values() for p in c.rglob("*") if p.is_file()} + ) + witness = source_witness() + samples, states, priors, targets = [], [], [], [] + for path in raw_paths: + with np.load(path) as raw: + np.testing.assert_array_equal(raw["previous_target"][1:], raw["action"][:-1]) + for tick in ticks: + samples.append( + { + "episode": path.stem, + "tick": tick, + "group": "original" if path.parent == teacher_root else "correction", + "period": "startup" if tick < 60 else "sustained", + } + ) + states.append(raw["state"][tick]) + priors.append(raw["previous_target"][tick]) + targets.append(raw["action"][tick : tick + H]) + states, priors, targets = np.asarray(states), np.asarray(priors), np.asarray(targets) + assert ( + states.shape == (sample_count, 36) + and priors.shape == (sample_count, 15) + and targets.shape == (sample_count, H, 15) + ) + np.savez_compressed( + root / "observations.npz", state=states, previous_target=priors, target=targets, image=BLACK + ) + cfg = dict( + texts=texts, + roles=roles, + noise_seeds=seeds, + samples=samples, + horizon=H, + maximum_inference_calls=call_budget, + optimizer_updates=0, + physics_steps=0, + source_files=witness, + input_files=hashes, + argv=sys.argv, + processors={a: processor_audit(c, args.replay) for a, c in checkpoints.items()}, + ) + write(root / "config.json", cfg) + note( + root, + f"# Frozen observation language fitting: {args.stage}\n" + f"SHOULD: {call_budget} predictions including repeats; 0 updates and 0 physics steps.\n" + "SHOULD: same state/prior/black image and pre-call RNG across texts and arms.\n" + "SHOULD: finite H10x15 physical targets, unchanged source/checkpoints/raw.\n" + f"Teacher backward error is meaningful only for first {backward_count} texts; " + "controls measure separation.", + ) + report = dict(stage="started", inference_calls=0, optimizer_updates=0, arms={}) + legs = [i for i, name in enumerate(JOINT_NAMES) if name.startswith(("left_", "right_"))] + assert len(legs) == 10 + try: + for arm, checkpoint in checkpoints.items(): + (root / "run.stage").write_text(arm) + model = SmolPolicy(checkpoint) + torch.cuda.reset_peak_memory_stats() + arm_started = time.perf_counter() + chunks, native, rng, rows = [], {}, {}, [] + for sample, (state, prior) in enumerate(zip(states, priors, strict=True)): + for text_index, text in enumerate(texts): + # Same native preprocessing as predict; no sampling here. + batch = model.preprocess( + { + "observation.state": torch.from_numpy( + policy_state(state, prior, include_previous_target=True).astype( + np.float32 + ) + ), + IMAGE_KEY: torch.from_numpy(BLACK.copy()).permute(2, 0, 1).float() + / 255, + "task": text, + } + ) + for key, value in batch.items(): + if isinstance(value, torch.Tensor): + native[f"{sample}:{text_index}:{key}"] = value.detach().cpu().numpy() + if ( + text_index + and key.startswith("observation.") + and "language" not in key + ): + np.testing.assert_array_equal( + native[f"{sample}:0:{key}"], + native[f"{sample}:{text_index}:{key}"], + ) + for noise in seeds: + torch.manual_seed(noise) + identity = f"{sample}:{text_index}:{noise}" + rng[identity + ":cpu"] = torch.get_rng_state().numpy() + rng[identity + ":cuda"] = torch.cuda.get_rng_state().cpu().numpy() + call_started = time.perf_counter() + chunk = model.predict(text, state, BLACK, prior) + latency = time.perf_counter() - call_started + rng[identity + ":cpu-after"] = torch.get_rng_state().numpy() + rng[identity + ":cuda-after"] = torch.cuda.get_rng_state().cpu().numpy() + report["inference_calls"] += 1 + assert chunk.shape == (H, 15) and np.isfinite(chunk).all() + if not chunks: + torch.manual_seed(noise) + repeat = model.predict(text, state, BLACK, prior) + report["inference_calls"] += 1 + np.savez_compressed( + root / f"{arm}-smoke.npz", prediction=chunk, repeat=repeat + ) + np.testing.assert_array_equal(chunk, repeat) + note(root, f"{arm}: same-input smoke exact; shape H10x15 finite.") + chunks.append(chunk) + rows.append( + dict( + sample=sample, + text_index=text_index, + noise_seed=noise, + latency_s=latency, + ) + ) + # Persist each completed call, including partially completed states. + np.savez_compressed(root / f"{arm}-predictions.npz", prediction=chunks) + np.savez_compressed(root / f"{arm}-native.npz", **native) + np.savez_compressed(root / f"{arm}-rng.npz", **rng) + write(root / f"{arm}-rows.json", rows) + note( + root, + f"{arm}: completed {sample + 1}/{sample_count} states; " + f"calls={report['inference_calls']}", + ) + cube = np.asarray(chunks, dtype=np.float64).reshape( + sample_count, text_count, noise_count, H, 15 + )[..., legs] + target = targets[..., legs].astype(np.float64) + prior = priors[..., legs].astype(np.float64) + metrics = [] + for sample in range(sample_count): + for text_index in range(text_count): + pred = cube[sample, text_index] + for horizon in (E, H): + p = pred[:, :horizon] + # Average unordered noise pairs, preserving state and text identity. + variance = [ + np.mean((p[a] - p[b]) ** 2) + for a in range(noise_count) + for b in range(a + 1, noise_count) + ] + metrics.append( + dict( + sample=sample, + text_index=text_index, + horizon=horizon, + within_noise_rms_rad=float(np.sqrt(np.mean(variance))), + paired_text_rms_rad=float( + np.sqrt(np.mean((p - cube[sample, 0, :, :horizon]) ** 2)) + ), + teacher_mse_rad2=float( + np.mean((p - target[sample, None, :horizon]) ** 2) + ) + if text_index < backward_count + else None, + persistence_mse_rad2=float( + np.mean((prior[sample] - target[sample, :horizon]) ** 2) + ), + target_offset_rms_rad=float( + np.sqrt(np.mean((p - prior[sample]) ** 2)) + ), + ) + ) + report["arms"][arm] = dict( + metrics=metrics, + elapsed_s=time.perf_counter() - arm_started, + gpu_peak_bytes=torch.cuda.max_memory_allocated(), + median_latency_s=float(np.median([r["latency_s"] for r in rows])), + ) + write(root / "report.json", report) + note( + root, + f"{arm}: offline grid complete, elapsed={report['arms'][arm]['elapsed_s']:.2f}s", + ) + del model + torch.cuda.empty_cache() + assert report["inference_calls"] == call_budget + assert source_witness() == witness + assert all(sha256(Path(p)) == digest for p, digest in hashes.items()) + report.update(stage="complete", elapsed_s=time.perf_counter() - started) + (root / "run.stage").write_text("complete") + (root / "run.exit-code").write_text("0") + note( + root, + f"## Complete\n{call_budget} inference calls; 0 updates; 0 physics steps.\n" + "Full predictions, native tensors and RNG states saved; " + "per-state metrics in report.json.", + ) + except BaseException: + report["stage"] = "failed" + (root / "run.exit-code").write_text("1") + raise + finally: + write(root / "report.json", report) + + +LANGUAGE_TEXTS = ("向后走", "往后退一点", "朝身后退着走", "向后迈步", "往身后走", "往后退步") +LANGUAGE_EVAL = [ + ("original", "向后走"), + ("augmented", "朝身后退着走"), + ("fresh-a", "向身后迈几步"), + ("fresh-b", "往后方走一段"), +] + + +def prepare_language_pair(args: argparse.Namespace) -> None: + """Pair native dataset replicas; change only backward task strings. + + spec: both arms share source-frame exposure and physical normalization. + link: docs/project/vla-language-coverage-2026-10-15.md + """ + from collections import Counter + + from lerobot.datasets.dataset_tools import merge_datasets, modify_tasks + from lerobot.datasets.io_utils import write_stats + from lerobot.datasets.sampler import EpisodeAwareSampler + + root = args.root + root.mkdir(parents=True, exist_ok=False) + assert shutil.disk_usage(root).free > 15 * 2**30 + source = dataset(args.replay) + original = json.loads((args.replay.parent / "data-audit.json").read_text())["episodes"] + assert source.num_episodes == len(original) == 134 and source.num_frames == 31400 + source_hashes = {str(p): sha256(p) for p in args.replay.rglob("*") if p.is_file()} + backward = [i for i, e in enumerate(original) if e["family"] == "backward"] + assert len(backward) == 15 + replay = [] + for family in HEAD_TASKS + MOTION_TASKS: + if family != "backward": + chosen = [i for i, e in enumerate(original) if e["family"] == family][:2] + assert len(chosen) == 2 and all(original[i]["frames"] == 200 for i in chosen) + replay.extend(chosen) + selected = sorted(backward + replay) + episodes = [r * 134 + i for r in range(6) for i in selected] + assert len(episodes) == 210 + config = { + "source_files": source_witness(), + "source_dataset_sha256": source_hashes, + "initial_sha256": sha256(args.initial / "model.safetensors"), + "external_sha256": { + str(p): sha256(p) + for p in ( + args.teacher, + args.upstream / "scripts/infer_policy.py", + args.upstream / "src/mjlab_microduck/robot/microduck/scene.xml", + *[p for p in args.initial.iterdir() if p.is_file()], + ) + }, + "argv": sys.argv, + "train_texts": LANGUAGE_TEXTS, + "eval_texts": LANGUAGE_EVAL, + "eval_bodies": [730, 760, 761], + "selected_source_episodes": selected, + "selected_episodes": episodes, + "max_updates": 2004, + "formal_steps_per_arm": 1000, + "discarded_smoke_per_arm": 2, + "batch_size": 8, + "optimizer_seed": 42, + "state_dim": 51, + "horizon": H, + "execute_steps": E, + "physics_paused": True, + } + write(root / "language-config.json", config) + note( + root, + "# v25 language coverage pair\n" + "SHOULD: identical numeric data, native sampling and effective normalization; " + "only backward text changes.\n" + "SHOULD: each arm discards 2 smoke updates and restarts from the same initial " + "weight for 1000 updates.\n" + "SHOULD: 22 scheduled main physics trials per model, including old-task " + "regression; no visual or real-time claim.", + ) + copies = [] + for replica in range(6): + path = root / "replicas" / f"source-{replica}" + shutil.copytree(args.replay, path) + copies.append(dataset(path)) + control = root / "dataset-control" + merge_datasets( + copies, + "duckmind/language-control", + output_dir=control, + concatenate_data=False, + concatenate_videos=False, + ) + augmented = root / "dataset-augmented" + shutil.copytree(control, augmented) + modify_tasks( + dataset(augmented), + episode_tasks={ + r * 134 + i: text for r, text in enumerate(LANGUAGE_TEXTS) for i in backward + }, + ) + # modify_tasks rewrites parquet in place, so discard its stale hf_dataset object. + # Repair bookkeeping stats too; physical episode values are unchanged by replication. + source_arrays = { + k: np.asarray(source.hf_dataset[k]) + for k in ("observation.state", "action", "timestamp", "frame_index", "task_index") + } + offset = np.cumsum([0] + [e["frames"] for e in original]) + arm_arrays, audit_rows, hashes = {}, {}, {} + for arm in ("control", "augmented"): + path = root / ("dataset-" + arm) + ds = dataset(path) + assert ds.num_episodes == 804 and ds.num_frames == 188400 + arrays = { + k: np.asarray(ds.hf_dataset[k]) for k in (*source_arrays, "episode_index", "index") + } + arm_arrays[arm] = arrays + bookkeeping, declared = [], [] + for replica in range(6): + for i, episode in enumerate(original): + ep = replica * 134 + i + start, stop = replica * 31400 + offset[i], replica * 31400 + offset[i + 1] + for key in ("observation.state", "action", "timestamp", "frame_index"): + np.testing.assert_array_equal( + arrays[key][start:stop], source_arrays[key][offset[i] : offset[i + 1]] + ) + assert np.all(arrays["episode_index"][start:stop] == ep) + np.testing.assert_array_equal(arrays["index"][start:stop], np.arange(start, stop)) + ids = np.unique(arrays["task_index"][start:stop]) + assert len(ids) == 1 + text = str(ds.meta.tasks.iloc[int(ids[0])].name) + expected = ( + LANGUAGE_TEXTS[replica] + if arm == "augmented" and i in backward + else episode["text"] + ) + assert text == expected + declared.append({**episode, "text": text, "source_episode": i, "replica": replica}) + bookkeeping.append( + { + k: arrays[k][start:stop].reshape(-1, 1) + for k in ("index", "episode_index", "task_index") + } + ) + precise_numeric_stats(path, ds, bookkeeping) + # Six identical copies have exactly the original physical distribution. Preserve + # canonical float64 stats rather than introducing aggregation roundoff near zero. + for key in ("observation.state", "action", "timestamp", "frame_index", IMAGE_KEY): + ds.meta.stats[key] = { + k: np.array(v, copy=True) for k, v in source.meta.stats[key].items() + } + ds.meta.stats[key]["count"] *= 6 + write_stats(ds.meta.stats, path) + processor_audit(args.initial, path) + audit_rows[arm] = declared + hashes[arm] = {str(p): sha256(p) for p in path.rglob("*") if p.is_file()} + note( + root, + f"{arm}: 804 replicas /188400 frames; full numerical and resolved-text audit passed.", + ) + for key in arm_arrays["control"]: + if key != "task_index": + np.testing.assert_array_equal(arm_arrays["control"][key], arm_arrays["augmented"][key]) + + def video_hashes(path: Path) -> list[str]: + return sorted(sha256(p) for p in path.rglob("*.mp4")) + + assert video_hashes(control) == video_hashes(augmented) == sorted(video_hashes(args.replay) * 6) + # A native sampler reconstruction is evidence of planned exposure, not a captured + # training hook. Its seed/epoch are independent of the model's diffusion RNG. + sampling = {} + for arm in ("control", "augmented"): + ds = LeRobotDataset( + "duckmind/" + arm, + root=root / ("dataset-" + arm), + episodes=episodes, + video_backend="pyav", + ) + sampler = EpisodeAwareSampler( + ds.meta.episodes["dataset_from_index"], + ds.meta.episodes["dataset_to_index"], + episode_indices_to_use=ds.episodes, + shuffle=True, + seed=42, + absolute_to_relative_idx=ds.absolute_to_relative_idx, + ) + relative = list(sampler)[:8000] + global_indices = np.asarray(ds.hf_dataset["index"])[relative].astype(int) + ep_indices = np.asarray(ds.hf_dataset["episode_index"])[relative].astype(int) + tasks = [audit_rows[arm][i]["text"] for i in ep_indices] + sampling[arm] = { + "global_indices": global_indices.tolist(), + "text_counts": dict(Counter(tasks)), + "family_counts": dict(Counter(audit_rows[arm][i]["family"] for i in ep_indices)), + } + assert sampling["control"]["global_indices"] == sampling["augmented"]["global_indices"] + write(root / "sampling.json", sampling) + # evaluate uses a canonical physical dataset for processors and train-only probes. + (root / "dataset").symlink_to(control.name) + write( + root / "data-audit.json", + { + "episodes": audit_rows["control"], + "frames": 188400, + "arm_episodes": audit_rows, + "dataset_sha256": hashes, + "source_dataset_sha256": source_hashes, + "independent_source_episodes": 134, + "selected_source_episodes": 35, + }, + ) + raw = root / "teacher" + raw.mkdir() + report = [] + sim = Simulation(args.upstream, args.teacher) + try: + for seed in config["eval_bodies"]: + task = SELECTED["backward"] + preparation = sim.prepare(task, seed) + initial_witness(root, sim, seed, task, preparation) + sim.command(task) + trace = expert_rollout(sim, 200) + trace.save(raw / f"{seed}-backward.npz") + quality = outcome(task, trace.metrics, 200, trace_gaze(sim, trace.physics)) + report.append({"body": seed, "ticks": len(trace.actions), "quality": quality}) + write(raw / "report.json", report) + note( + root, + f"Teacher body={seed}: ticks={len(trace.actions)}, accepted={quality['accepted']}", + ) + assert all(row["quality"]["accepted"] for row in report), report + finally: + sim.close() + assert all(sha256(Path(p)) == value for p, value in source_hashes.items()) + (root / "prepare.exit-code").write_text("0") + note( + root, + "Preparation complete. Zero model inference and zero optimizer updates; " + "independent preflight required before training.", + ) + + +def run_language_pair(args: argparse.Namespace) -> None: + root, course = args.root, args.course + cfg = json.loads((root / "language-config.json").read_text()) + audit = json.loads((root / "data-audit.json").read_text()) + assert cfg["source_files"] == source_witness() + assert cfg["initial_sha256"] == sha256(args.initial / "model.safetensors") + assert all(sha256(Path(p)) == value for p, value in cfg["external_sha256"].items()) + assert (root / "prepare.exit-code").read_text().strip() == "0" + assert ( + json.loads((root.parent / "independent-preflight-audit.json").read_text())["stage"] + == "passed" + ) + for hashes in audit["dataset_sha256"].values(): + assert all(sha256(Path(p)) == value for p, value in hashes.items()) + assert not (root / "run.pid").exists() + (root / "run.pid").write_text(str(os.getpid())) + + def stage(name: str) -> None: + (root / "run.stage").write_text(name) + note(root, "## " + name) + + def train(arm: str, steps: int, label: str) -> Path: + command = [ + sys.executable, + "-c", + "from duckmind_training.train import main; main()", + "--dataset", + str(root / ("dataset-" + arm)), + "--model", + str(args.initial), + "--output", + str(root / label), + "--steps", + str(steps), + "--batch-size", + "8", + "--chunk-size", + str(H), + "--episodes", + *map(str, cfg["selected_episodes"]), + ] + note(root, f"Native trainer argv: {command!r}") + subprocess.run(command, check=True) + checkpoint = root / label / "checkpoints" / f"{steps:06d}" / "pretrained_model" + processor_audit(checkpoint, root / ("dataset-" + arm)) + return checkpoint + + def physical(checkpoint: Path, label: str) -> None: + args.course = replace(course, tasks=("backward",), eval_seeds=tuple(cfg["eval_bodies"])) + evaluate(args, checkpoint, label + "-backward", texts={"backward": LANGUAGE_EVAL}) + regression = tuple(t for t in HEAD_TASKS + MOTION_TASKS if t != "backward") + args.course = replace(course, tasks=regression, eval_seeds=(730,)) + evaluate( + args, + checkpoint, + label + "-regression", + texts={name: [("original", SELECTED[name].texts[0])] for name in regression}, + ) + + try: + for arm in ("control", "augmented"): + stage("smoke-" + arm) + checkpoint = train(arm, 2, "smoke-" + arm) + args.course = replace(course, tasks=("backward",), eval_seeds=(730,)) + evaluate( + args, + checkpoint, + "smoke-eval-" + arm, + smoke=True, + texts={"backward": [("augmented", LANGUAGE_TEXTS[2])]}, + ) + stage("baseline") + physical(args.initial, "baseline") + for arm in ("control", "augmented"): + stage("train-" + arm) + checkpoint = train(arm, 1000, "model-" + arm) + stage("evaluate-" + arm) + physical(checkpoint, arm) + stage("verify") + assert all(sha256(Path(p)) == value for p, value in cfg["external_sha256"].items()) + assert cfg["source_files"] == source_witness() + assert cfg["initial_sha256"] == sha256(args.initial / "model.safetensors") + for hashes in audit["dataset_sha256"].values(): + assert all(sha256(Path(p)) == value for p, value in hashes.items()) + stage("complete") + (root / "run.exit-code").write_text("0") + except BaseException: + (root / "run.exit-code").write_text("1") + raise + + +SMALL_TEXTS = { + "stand": ("保持站立", "站好别动", "在原地站稳", "保持站姿", "站着保持平衡", "停在原地站好"), + "forward": ("向前走", "往前走一点", "朝前面走", "向前迈步", "往前方走", "往前走几步"), + "backward": LANGUAGE_TEXTS, +} +SMALL_COURSE = Curriculum( + "v27-small", + tuple(SMALL_TEXTS), + tuple(SMALL_TEXTS), + (700, 730), + (), + (700, 730, 770, 771), + {"stand": "请在这里站稳不动", "forward": "朝前方迈出几步", "backward": "朝后方迈出几步"}, + 4000, + 0, + 0, +) + + +def prepare_small(args: argparse.Namespace) -> None: + """Build six matched-reset trajectories and balanced language replicas. + + spec: only teacher actions label training; all texts of one physical trajectory stay together. + why: this is a fitting diagnostic, not a broad replacement checkpoint. + link: docs/project/vla-small-curriculum-2026-10-16.md + """ + root = args.root + root.mkdir(parents=True, exist_ok=False) + assert shutil.disk_usage(root).free > 12 * 2**30 + (root / "teacher").mkdir() + cfg = { + "source_files": source_witness(), + "initial": str(args.initial), + "initial_sha256": sha256(args.initial / "model.safetensors"), + "teacher_sha256": sha256(args.teacher), + "external_sha256": { + str(p): sha256(p) + for p in ( + args.teacher, + args.upstream / "scripts/infer_policy.py", + args.upstream / "src/mjlab_microduck/robot/microduck/scene.xml", + *[p for p in args.initial.iterdir() if p.is_file()], + ) + }, + "texts": SMALL_TEXTS, + "curriculum": asdict(SMALL_COURSE), + "budget_updates": 4002, + "normalization": "true small training dataset; zero-control changes stats only", + "argv": sys.argv, + } + write(root / "small-config.json", cfg) + note( + root, + "# v27 small language curriculum\n" + "SHOULD: six physical episodes, 36 language replicas, 7200 frames; shared resets.\n" + "SHOULD: 2 discarded smoke +4000 formal updates; native checkpoints every1000.\n" + "SHOULD: new true float64 stats, zero unchanged weights; black51/H10/E5.\n" + "Four seconds is the trained duration. No visual, realtime or broad capability claim.", + ) + report = {"stage": "started", "episodes": []} + numeric = [] + sim = Simulation(args.upstream, args.teacher) + try: + # No teacher screening of evaluation bodies: keep their student results honest. + for seed in SMALL_COURSE.train_seeds: + for name, texts in SMALL_TEXTS.items(): + task = SELECTED[name] + preparation = sim.prepare(task, seed) + initial_witness(root, sim, seed, task, preparation) + sim.command(task) + trace = expert_rollout(sim, 200) + identity = f"train-{seed}-{name}" + trace.save(root / "teacher" / f"{identity}.npz") + gaze = trace_gaze(sim, trace.physics) + quality = outcome(task, trace.metrics, 200, gaze) + row = { + "id": identity, + "body": seed, + "task": name, + "quality": quality, + "command": asdict(task), + "ticks": len(trace.actions), + "wrong_goals": { + n: outcome(SELECTED[n], trace.metrics, 200, gaze)["accepted"] + for n in SMALL_TEXTS + if n != name + }, + } + report["episodes"].append(row) + write(root / "teacher/report.json", report) + note(root, f"Teacher {identity}: accepted={quality['accepted']}") + assert quality["accepted"] and not any(row["wrong_goals"].values()), row + states = policy_state( + np.asarray(trace.states), + np.asarray(trace.previous_targets), + include_previous_target=True, + ) + actions = np.asarray(trace.actions) + np.testing.assert_array_equal(states[1:, 36:], actions[:-1]) + for text in texts: + numeric.append( + { + "observation.state": states, + "action": actions, + "texts": [text] * 200, + "family": name, + } + ) + finally: + sim.close() + assert len(numeric) == 36 + report["stage"] = "complete" + write(root / "teacher/report.json", report) + prepare_data(args, numeric) + assert source_witness() == cfg["source_files"] + (root / "prepare.exit-code").write_text("0") + + +def small_fit(args: argparse.Namespace, checkpoint: Path, label: str) -> None: + """Every trained text, both bodies and six periods; fixed noise across paraphrases.""" + root = args.root + out = root / (label + "-fit") + out.mkdir(exist_ok=False) + processor_audit(checkpoint, root / "dataset") + ds = dataset(root / "dataset", chunks=True) + model = SmolPolicy(checkpoint) + ticks = (0, 5, 10, 20, 60, 140) + legs = [i for i, n in enumerate(JOINT_NAMES) if n.startswith(("left_", "right_"))] + rows, predictions, states, targets = [], [], [], [] + started = time.perf_counter() + try: + for episode in range(36): + physical = episode // 6 + family = SMALL_COURSE.tasks[physical % 3] + for tick in ticks: + item = ds[episode * 200 + tick] + assert int(item["episode_index"]) == episode and not item["action_is_pad"].any() + assert item["task"] == SMALL_TEXTS[family][episode % 6] + state, target = item["observation.state"].numpy(), item["action"].numpy() + assert not item[IMAGE_KEY].any() + for noise in (27001, 27002): + torch.manual_seed(noise) + call_start = time.perf_counter() + pred = model.predict(item["task"], state[:36], BLACK, state[36:]) + assert pred.shape == (H, 15) and np.isfinite(pred).all() + predictions.append(pred) + states.append(state) + targets.append(target) + rows.append( + { + "episode": episode, + "physical_episode": physical, + "task": family, + "tick": tick, + "text": item["task"], + "noise": noise, + "latency_s": time.perf_counter() - call_start, + "leg_e5_mse": float(np.mean((pred[:E, legs] - target[:E, legs]) ** 2)), + "leg_h10_mse": float(np.mean((pred[:, legs] - target[:, legs]) ** 2)), + "persistence_leg_e5_mse": float( + np.mean((state[36:][legs] - target[:E, legs]) ** 2) + ), + } + ) + # Preserve partial diagnostics on interruption, not only a final aggregate. + write(out / "rows.json", rows) + np.savez_compressed( + out / "predictions.npz", prediction=predictions, state=states, target=targets + ) + assert len(rows) == 432 + write( + out / "report.json", + { + "stage": "complete", + "rows": rows, + "model_sha256": sha256(checkpoint / "model.safetensors"), + "calls": len(rows), + "training_states_only": True, + "elapsed_s": time.perf_counter() - started, + }, + ) + note( + root, + f"{label} fit:432 calls, {time.perf_counter() - started:.1f}s; teacher states only.", + ) + finally: + del model + torch.cuda.empty_cache() + + +def run_small(args: argparse.Namespace) -> None: + root = args.root + cfg = json.loads((root / "small-config.json").read_text()) + audit = json.loads((root / "data-audit.json").read_text()) + assert (root / "prepare.exit-code").read_text().strip() == "0" + assert cfg["source_files"] == source_witness() + assert all(sha256(Path(p)) == v for p, v in cfg["external_sha256"].items()) + assert cfg["initial_sha256"] == sha256(args.initial / "model.safetensors") + assert all(sha256(Path(p)) == digest for p, digest in audit["dataset_sha256"].items()) + assert not (root / "run.pid").exists() + (root / "run.pid").write_text(str(os.getpid())) + + def stage(name: str) -> None: + (root / "run.stage").write_text(name) + note(root, f"## {name}") + + def train(steps: int, label: str) -> Path: + command = [ + sys.executable, + "-c", + "from duckmind_training.train import main; main()", + "--dataset", + str(root / "dataset"), + "--model", + str(args.initial), + "--output", + str(root / label), + "--steps", + str(steps), + "--batch-size", + "8", + "--chunk-size", + str(H), + "--save-freq", + "1000", + ] + note(root, f"Native trainer argv: {command!r}") + subprocess.run(command, check=True) + checkpoint = root / label / "checkpoints" / f"{steps:06d}" / "pretrained_model" + processor_audit(checkpoint, root / "dataset") + return checkpoint + + def physical(checkpoint: Path, phase: str, *, final: bool = False, smoke: bool = False) -> None: + selected = argparse.Namespace(**vars(args)) + selected.course = replace( + SMALL_COURSE, + eval_seeds=(SMALL_COURSE.eval_seeds if final else SMALL_COURSE.train_seeds), + ) + texts = {n: [("original", t[0])] for n, t in SMALL_TEXTS.items()} + if final: + texts = { + n: [("original", t[0]), ("variant", t[-1]), ("unseen", SMALL_COURSE.unseen[n])] + for n, t in SMALL_TEXTS.items() + } + evaluate(selected, checkpoint, phase, smoke=smoke, texts=texts, forward_steps=200) + + try: + stage("smoke-train") + smoke_checkpoint = train(2, "model-smoke") + stage("smoke-evaluate") + physical(smoke_checkpoint, "smoke", smoke=True) + stage("zero-fit") + small_fit(args, root / "zero-checkpoint", "zero") + stage("zero-evaluate") + physical(root / "zero-checkpoint", "zero-eval") + stage("train") + final = train(4000, "model") + for step in (1000, 2000, 4000): + checkpoint = root / "model/checkpoints" / f"{step:06d}" / "pretrained_model" + stage(f"fit-{step}") + small_fit(args, checkpoint, f"step-{step}") + stage(f"evaluate-{step}") + physical(checkpoint, f"step-{step}", final=step == 4000) + stage("verify") + assert source_witness() == cfg["source_files"] + assert all(sha256(Path(p)) == v for p, v in cfg["external_sha256"].items()) + assert all(sha256(Path(p)) == digest for p, digest in audit["dataset_sha256"].items()) + assert cfg["initial_sha256"] == sha256(args.initial / "model.safetensors") + assert final.exists() + stage("complete") + (root / "run.exit-code").write_text("0") + except BaseException: + (root / "run.exit-code").write_text("1") + raise + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "stage", + choices=( + "prepare", + "train-evaluate", + "calibrate-gaze", + "check-starts", + "diagnose-backward", + "diagnose-correction-fit", + "diagnose-feedback", + "diagnose-text-noise", + "diagnose-language-fit", + "prepare-backward-corrections", + "finalize-backward-preparation", + "train-backward-pair", + "prepare-language-pair", + "train-language-pair", + "prepare-small", + "train-small", + ), + ) + for name in ("root", "replay", "initial", "upstream", "teacher"): + parser.add_argument("--" + name, type=Path, required=True) + parser.add_argument("--course", choices=tuple(COURSES), default="heads") + parser.add_argument("--diagnosis-root", type=Path) + args = parser.parse_args() + args.course = ( + SMALL_COURSE if args.stage in ("prepare-small", "train-small") else COURSES[args.course] + ) + { + "prepare": prepare, + "train-evaluate": run, + "calibrate-gaze": calibrate_gaze, + "check-starts": check_starts, + "diagnose-backward": diagnose_backward, + "diagnose-correction-fit": diagnose_correction_fit, + "diagnose-feedback": diagnose_feedback, + "diagnose-text-noise": diagnose_text_noise, + "diagnose-language-fit": diagnose_text_noise, + "prepare-backward-corrections": prepare_backward_corrections, + "finalize-backward-preparation": finalize_backward_preparation, + "train-backward-pair": run_backward_pair, + "prepare-language-pair": prepare_language_pair, + "train-language-pair": run_language_pair, + "prepare-small": prepare_small, + "train-small": run_small, + }[args.stage](args) + + +if __name__ == "__main__": + main() diff --git a/training/experiments/run-backward-curriculum.sh b/training/experiments/run-backward-curriculum.sh new file mode 100644 index 0000000..b9391d9 --- /dev/null +++ b/training/experiments/run-backward-curriculum.sh @@ -0,0 +1,13 @@ +#!/bin/sh +set -eu +cd "$(dirname "$0")/.." +export PYTHONPATH="$PWD/src:$PWD/../policy/src" +OLD="$HOME/rustprojects/compforge/duckmind/.worktrees/vla-training/training" +export HF_HOME="$OLD/artifacts/huggingface" HF_HUB_OFFLINE=1 MUJOCO_GL=egl OMP_NUM_THREADS=2 PYTHONUNBUFFERED=1 +exec "$OLD/.venv/bin/python" experiments/head_curriculum.py "$1" --course motion \ + --root /srv/duckmind-training/v21-owner/run \ + --replay /srv/duckmind-training/v19-owner/run/dataset \ + --initial /srv/duckmind-training/v19-owner/run/model/checkpoints/004000/pretrained_model \ + --diagnosis-root /srv/duckmind-training/v20-owner/run \ + --upstream "$HOME/rustprojects/pollen-robotics/microduck_rl" \ + --teacher "$HOME/myprojects/mono-robot/MicroduckSim/policies/velstand.onnx" diff --git a/training/experiments/run-correction-fit-curriculum.sh b/training/experiments/run-correction-fit-curriculum.sh new file mode 100644 index 0000000..1addcb7 --- /dev/null +++ b/training/experiments/run-correction-fit-curriculum.sh @@ -0,0 +1,14 @@ +#!/bin/sh +set -eu +cd "$(dirname "$0")/.." +export PYTHONPATH="$PWD/src:$PWD/../policy/src" +OLD="$HOME/rustprojects/compforge/duckmind/.worktrees/vla-training/training" +export HF_HOME="$OLD/artifacts/huggingface" HF_HUB_OFFLINE=1 MUJOCO_GL=egl OMP_NUM_THREADS=2 PYTHONUNBUFFERED=1 +ROOT=/srv/duckmind-training/v22-owner/run +[ ! -e "$ROOT" ] || { echo "Refusing existing output: $ROOT" >&2; exit 1; } +trap 'code=$?; if [ "$code" -ne 0 ] && [ -d "$ROOT" ]; then printf "%s\n" "$code" > "$ROOT/run.exit-code"; fi' EXIT +"$OLD/.venv/bin/python" experiments/head_curriculum.py diagnose-correction-fit --course motion \ + --root "$ROOT" --replay /srv/duckmind-training/v21-owner/run/dataset \ + --initial /srv/duckmind-training/v19-owner/run/model/checkpoints/004000/pretrained_model \ + --upstream "$HOME/rustprojects/pollen-robotics/microduck_rl" \ + --teacher "$HOME/myprojects/mono-robot/MicroduckSim/policies/velstand.onnx" diff --git a/training/experiments/run-feedback-curriculum.sh b/training/experiments/run-feedback-curriculum.sh new file mode 100644 index 0000000..2d13fd0 --- /dev/null +++ b/training/experiments/run-feedback-curriculum.sh @@ -0,0 +1,14 @@ +#!/bin/sh +set -eu +cd "$(dirname "$0")/.." +export PYTHONPATH="$PWD/src:$PWD/../policy/src" +OLD="$HOME/rustprojects/compforge/duckmind/.worktrees/vla-training/training" +export HF_HOME="$OLD/artifacts/huggingface" HF_HUB_OFFLINE=1 MUJOCO_GL=egl OMP_NUM_THREADS=2 PYTHONUNBUFFERED=1 +ROOT=/srv/duckmind-training/v23-owner/run +[ ! -e "$ROOT" ] || { echo "Refusing existing output: $ROOT" >&2; exit 1; } +trap 'code=$?; if [ "$code" -ne 0 ] && [ -d "$ROOT" ]; then printf "%s\n" "$code" > "$ROOT/run.exit-code"; fi' EXIT +"$OLD/.venv/bin/python" experiments/head_curriculum.py diagnose-feedback --course motion \ + --root "$ROOT" --replay /srv/duckmind-training/v21-owner/run/dataset \ + --initial /srv/duckmind-training/v19-owner/run/model/checkpoints/004000/pretrained_model \ + --upstream "$HOME/rustprojects/pollen-robotics/microduck_rl" \ + --teacher "$HOME/myprojects/mono-robot/MicroduckSim/policies/velstand.onnx" diff --git a/training/experiments/run-head-curriculum.sh b/training/experiments/run-head-curriculum.sh new file mode 100644 index 0000000..47f64f7 --- /dev/null +++ b/training/experiments/run-head-curriculum.sh @@ -0,0 +1,13 @@ +#!/bin/sh +set -eu +cd "$(dirname "$0")/.." +export PYTHONPATH="$PWD/src:$PWD/../policy/src" +OLD="$HOME/rustprojects/compforge/duckmind/.worktrees/vla-training/training" +export HF_HOME="$OLD/artifacts/huggingface" HF_HUB_OFFLINE=1 MUJOCO_GL=egl OMP_NUM_THREADS=2 PYTHONUNBUFFERED=1 +ROOT=/srv/duckmind-training/v18-owner/run +exec "$OLD/.venv/bin/python" experiments/head_curriculum.py "$1" \ + --root "$ROOT" \ + --replay /srv/duckmind-training/v17/dataset-union \ + --initial /srv/duckmind-training/v17/model-mixed/checkpoints/001000/pretrained_model \ + --upstream "$HOME/rustprojects/pollen-robotics/microduck_rl" \ + --teacher "$HOME/myprojects/mono-robot/MicroduckSim/policies/velstand.onnx" diff --git a/training/experiments/run-language-curriculum.sh b/training/experiments/run-language-curriculum.sh new file mode 100644 index 0000000..d94d341 --- /dev/null +++ b/training/experiments/run-language-curriculum.sh @@ -0,0 +1,12 @@ +#!/bin/sh +set -eu +cd "$(dirname "$0")/.." +export PYTHONPATH="$PWD/src:$PWD/../policy/src" +OLD="$HOME/rustprojects/compforge/duckmind/.worktrees/vla-training/training" +export HF_HOME="$OLD/artifacts/huggingface" HF_HUB_OFFLINE=1 MUJOCO_GL=egl OMP_NUM_THREADS=2 PYTHONUNBUFFERED=1 +exec "$OLD/.venv/bin/python" experiments/head_curriculum.py "$1" --course motion \ + --root /srv/duckmind-training/v25-owner/run \ + --replay /srv/duckmind-training/v21-owner/run/dataset \ + --initial /srv/duckmind-training/v21-owner/run/zero-checkpoint \ + --upstream "$HOME/rustprojects/pollen-robotics/microduck_rl" \ + --teacher "$HOME/myprojects/mono-robot/MicroduckSim/policies/velstand.onnx" diff --git a/training/experiments/run-language-fit-curriculum.sh b/training/experiments/run-language-fit-curriculum.sh new file mode 100644 index 0000000..183f718 --- /dev/null +++ b/training/experiments/run-language-fit-curriculum.sh @@ -0,0 +1,15 @@ +#!/bin/sh +set -eu +cd "$(dirname "$0")/.." +export PYTHONPATH="$PWD/src:$PWD/../policy/src" +OLD="$HOME/rustprojects/compforge/duckmind/.worktrees/vla-training/training" +export HF_HOME="$OLD/artifacts/huggingface" HF_HUB_OFFLINE=1 MUJOCO_GL=egl OMP_NUM_THREADS=2 PYTHONUNBUFFERED=1 +ROOT=/srv/duckmind-training/v26-owner/run +[ ! -e "$ROOT" ] || { echo "Refusing existing output: $ROOT" >&2; exit 1; } +trap 'code=$?; if [ "$code" -ne 0 ] && [ -d "$ROOT" ]; then printf "%s\n" "$code" > "$ROOT/run.exit-code"; fi' EXIT +"$OLD/.venv/bin/python" experiments/head_curriculum.py diagnose-language-fit --course motion \ + --root "$ROOT" --replay /srv/duckmind-training/v21-owner/run/dataset \ + --initial /srv/duckmind-training/v21-owner/run/zero-checkpoint \ + --diagnosis-root /srv/duckmind-training/v25-owner/run \ + --upstream "$HOME/rustprojects/pollen-robotics/microduck_rl" \ + --teacher "$HOME/myprojects/mono-robot/MicroduckSim/policies/velstand.onnx" diff --git a/training/experiments/run-motion-curriculum.sh b/training/experiments/run-motion-curriculum.sh new file mode 100644 index 0000000..a24278f --- /dev/null +++ b/training/experiments/run-motion-curriculum.sh @@ -0,0 +1,13 @@ +#!/bin/sh +set -eu +cd "$(dirname "$0")/.." +export PYTHONPATH="$PWD/src:$PWD/../policy/src" +OLD="$HOME/rustprojects/compforge/duckmind/.worktrees/vla-training/training" +export HF_HOME="$OLD/artifacts/huggingface" HF_HUB_OFFLINE=1 MUJOCO_GL=egl OMP_NUM_THREADS=2 PYTHONUNBUFFERED=1 +ROOT=/srv/duckmind-training/v19-owner/run +exec "$OLD/.venv/bin/python" experiments/head_curriculum.py "$1" --course motion \ + --root "$ROOT" \ + --replay /srv/duckmind-training/v18-owner/run/dataset \ + --initial /srv/duckmind-training/v18-owner/run/model/checkpoints/002000/pretrained_model \ + --upstream "$HOME/rustprojects/pollen-robotics/microduck_rl" \ + --teacher "$HOME/myprojects/mono-robot/MicroduckSim/policies/velstand.onnx" diff --git a/training/experiments/run-small-curriculum.sh b/training/experiments/run-small-curriculum.sh new file mode 100644 index 0000000..a34e159 --- /dev/null +++ b/training/experiments/run-small-curriculum.sh @@ -0,0 +1,12 @@ +#!/bin/sh +set -eu +cd "$(dirname "$0")/.." +export PYTHONPATH="$PWD/src:$PWD/../policy/src" +OLD="$HOME/rustprojects/compforge/duckmind/.worktrees/vla-training/training" +export HF_HOME="$OLD/artifacts/huggingface" HF_HUB_OFFLINE=1 MUJOCO_GL=egl OMP_NUM_THREADS=2 PYTHONUNBUFFERED=1 +exec "$OLD/.venv/bin/python" experiments/head_curriculum.py "$1" --course motion \ + --root /srv/duckmind-training/v27-owner/run \ + --replay /srv/duckmind-training/v21-owner/run/dataset \ + --initial /srv/duckmind-training/v25-owner/run/model-augmented/checkpoints/001000/pretrained_model \ + --upstream "$HOME/rustprojects/pollen-robotics/microduck_rl" \ + --teacher "$HOME/myprojects/mono-robot/MicroduckSim/policies/velstand.onnx" diff --git a/training/experiments/run-text-noise-curriculum.sh b/training/experiments/run-text-noise-curriculum.sh new file mode 100644 index 0000000..0bb329a --- /dev/null +++ b/training/experiments/run-text-noise-curriculum.sh @@ -0,0 +1,14 @@ +#!/bin/sh +set -eu +cd "$(dirname "$0")/.." +export PYTHONPATH="$PWD/src:$PWD/../policy/src" +OLD="$HOME/rustprojects/compforge/duckmind/.worktrees/vla-training/training" +export HF_HOME="$OLD/artifacts/huggingface" HF_HUB_OFFLINE=1 MUJOCO_GL=egl OMP_NUM_THREADS=2 PYTHONUNBUFFERED=1 +ROOT=/srv/duckmind-training/v24-owner/run +[ ! -e "$ROOT" ] || { echo "Refusing existing output: $ROOT" >&2; exit 1; } +trap 'code=$?; if [ "$code" -ne 0 ] && [ -d "$ROOT" ]; then printf "%s\n" "$code" > "$ROOT/run.exit-code"; fi' EXIT +"$OLD/.venv/bin/python" experiments/head_curriculum.py diagnose-text-noise --course motion \ + --root "$ROOT" --replay /srv/duckmind-training/v21-owner/run/dataset \ + --initial /srv/duckmind-training/v19-owner/run/model/checkpoints/004000/pretrained_model \ + --upstream "$HOME/rustprojects/pollen-robotics/microduck_rl" \ + --teacher "$HOME/myprojects/mono-robot/MicroduckSim/policies/velstand.onnx" diff --git a/training/src/duckmind_training/correction_quality.py b/training/src/duckmind_training/correction_quality.py index 1203d88..dacb971 100644 --- a/training/src/duckmind_training/correction_quality.py +++ b/training/src/duckmind_training/correction_quality.py @@ -2,7 +2,7 @@ import numpy as np -from duckmind_training.contract import FPS +from duckmind_training.contract import FPS, Task, episode_success def recovery_outcome(metrics: list[dict], expected_steps: int) -> dict: @@ -39,3 +39,84 @@ def recovery_outcome(metrics: list[dict], expected_steps: int) -> dict: "tail_forward_m": tail_forward, "terminal": end, } + + +def stationary_head_outcome(task: Task, metrics: list[dict], expected_steps: int) -> dict: + """Require a stationary body and sustained head pitch/yaw task tracking. + + spec: compute error before temporal averaging, and never average across joints. + Opposite head commands or oscillations must not count as standing at neutral. + Uncommanded neck/roll can compensate for balance; report them, not as zero targets. + link: docs/project/vla-head-actions-2026-10-13.md + """ + keys = ("neck_pitch_delta", "head_pitch_delta", "head_yaw_delta", "head_roll_delta") + tail = np.array([[m[k] for k in keys] for m in metrics[-FPS:]]) + mean = tail.mean(0) + errors = np.abs(tail - np.array(task.head)).mean(0) + required_axes = sorted({1, 2} | {i for i, value in enumerate(task.head) if value}) + checks = { + "complete": len(metrics) - 1 == expected_steps and expected_steps >= 2 * FPS, + "stable": all(m["stable"] for m in metrics), + "stationary": bool(max(np.hypot(m["forward_m"], m["left_m"]) for m in metrics) < 0.08), + "heading": all(abs(m["yaw_rad"]) < 0.5 for m in metrics), + "head_tracking": bool((errors[required_axes] < 0.15).all()), + "direction": all(mean[i] * np.sign(v) > 0.1 for i, v in enumerate(task.head) if v), + } + return { + "accepted": all(checks.values()), + "checks": checks, + "required_axes": [keys[i] for i in required_axes], + "tail_head_mean_rad": mean.tolist(), + "tail_head_error_rad": errors.tolist(), + "legacy_success": episode_success(task, metrics[-1], checks["stable"]), + } + + +def locomotion_outcome(task: Task, metrics: list[dict], expected_steps: int) -> dict: + """Sustained backward/turn/stop tasks; stop requires a moving-start experiment witness. + + spec: use accumulated yaw and whole-path drift, not wrapped endpoint heading or net speed. + case: opposite turns, sideways walking, oscillation and a late fall cannot pass. + link: docs/project/vla-motion-curriculum-2026-10-13.md + """ + if task.name == "backward": + # The original forward rubric applies in the commanded backward direction. + mirrored = [{**row, "forward_m": -row["forward_m"]} for row in metrics] + return recovery_outcome(mirrored, expected_steps) + if task.name not in ("turn_left", "turn_right", "stop_forward", "stop_backward"): + raise ValueError(f"unsupported locomotion task: {task.name}") + tail = metrics[-FPS - 1 :] + drift = max(float(np.hypot(m["forward_m"], m["left_m"])) for m in metrics) + checks = { + "complete": len(metrics) - 1 == expected_steps and expected_steps >= 2 * FPS, + "stable": all(m["stable"] for m in metrics), + "tail_upright": bool( + np.mean([m["gravity_z"] for m in tail]) < -0.9 + and np.mean([m["trunk_z_m"] for m in tail]) > 0.1 + ), + } + yaw = float(metrics[-1]["yaw_rad"] - metrics[0]["yaw_rad"]) + tail_yaw = float(tail[-1]["yaw_rad"] - tail[0]["yaw_rad"]) + if task.twist[2]: + sign = np.sign(task.twist[2]) + checks.update( + turn_direction=bool(sign * yaw > 0.2), + tail_turn=bool(sign * tail_yaw > 0.15), + limited_translation=drift < 0.20, + ) + else: + checks.update( + stopped=float(metrics[-1]["tail_speed_mps"]) < 0.06, + limited_travel=drift < 0.3, + original_heading=all(abs(m["yaw_rad"]) < 0.5 for m in metrics), + ) + return { + "accepted": all(checks.values()), + "checks": checks, + "steps": len(metrics) - 1, + "max_displacement_m": drift, + "yaw_rad": yaw, + "tail_yaw_rad": tail_yaw, + "tail_speed_mps": float(metrics[-1]["tail_speed_mps"]), + "terminal": metrics[-1], + } diff --git a/training/src/duckmind_training/simulation.py b/training/src/duckmind_training/simulation.py index 6d24a23..f279967 100644 --- a/training/src/duckmind_training/simulation.py +++ b/training/src/duckmind_training/simulation.py @@ -12,6 +12,14 @@ from duckmind_training.contract import IMAGE_SIZE, JOINT_NAMES, Task, state_vector, wire_positions +class PreparationError(ValueError): + """An invalid physical start, with evidence retained before origin reset.""" + + def __init__(self, message: str, evidence: dict): + super().__init__(message) + self.evidence = evidence + + class Simulation: def __init__(self, upstream: Path, teacher: Path): spec = importlib.util.spec_from_file_location( @@ -100,16 +108,39 @@ def mark_origin(self) -> None: self.yaw_travel = 0.0 self.previous_yaw = self.origin_yaw - def prepare(self, task: Task, seed: int) -> None: + def prepare(self, task: Task, seed: int) -> dict: self.reset(seed) + preparation = {"initial_twist": list(task.initial_twist), "metrics": []} if any(task.initial_twist): self.expert.set_vel_cmd(*task.initial_twist) + preparation["metrics"].append(self.metrics()) for _ in range(100): self.teacher_step() + preparation["metrics"].append(self.metrics()) + # Preserve moving-start evidence before mark_origin clears the speed window. + tail = preparation["metrics"][-51:] + signed_progress = (tail[-1]["forward_m"] - tail[0]["forward_m"]) * np.sign( + task.initial_twist[0] + ) + preparation["signed_tail_progress_m"] = float(signed_progress) + if task.initial_twist[0] and ( + signed_progress <= 0.03 or not all(m["stable"] for m in tail) + ): + raise PreparationError( + f"moving start failed: task={task.name} seed={seed} " + f"signed_progress={signed_progress:.6f}m " + f"stable_tail={all(m['stable'] for m in tail)}", + preparation, + ) # Stop examples start from an actually moving body, never a stationary reset. if not self.metrics()["stable"] or np.mean(self.tail_speeds) < 0.06: - raise ValueError(f"expert cannot prepare moving start for {task.name}") + raise PreparationError( + f"moving start too slow or unstable: task={task.name} seed={seed} " + f"speed={np.mean(self.tail_speeds):.6f}m/s", + preparation, + ) self.mark_origin() + return preparation def command(self, task: Task) -> None: self.expert.head_offset[:] = task.head diff --git a/training/src/duckmind_training/train.py b/training/src/duckmind_training/train.py index 47a93dc..7220b59 100644 --- a/training/src/duckmind_training/train.py +++ b/training/src/duckmind_training/train.py @@ -27,7 +27,10 @@ def main() -> None: parser.add_argument("--steps", type=int, default=1000) parser.add_argument("--batch-size", type=int, default=8) parser.add_argument("--chunk-size", type=int, choices=range(1, 101), default=50) + parser.add_argument("--save-freq", type=int, default=500) args = parser.parse_args() + if args.save_freq < 1: + parser.error("--save-freq must be positive") if args.output.exists(): raise FileExistsError(args.output) metadata = json.loads((args.dataset / "meta/info.json").read_text()) @@ -81,7 +84,7 @@ def main() -> None: num_workers=2, env_eval_freq=0, log_freq=10, - save_freq=min(args.steps, 500), + save_freq=min(args.steps, args.save_freq), seed=42, ) ) diff --git a/training/tests/test_head_quality.py b/training/tests/test_head_quality.py new file mode 100644 index 0000000..d86adaf --- /dev/null +++ b/training/tests/test_head_quality.py @@ -0,0 +1,63 @@ +"""Wrong-direction and transient poses cannot earn stationary task success.""" + +import json + +from duckmind_training.contract import TASKS +from duckmind_training.correction_quality import stationary_head_outcome + +TASK = {t.name: t for t in TASKS} + + +def trajectory(yaw: float = 0.0, pitch: float = 0.0) -> list[dict]: + return [ + dict( + stable=True, + forward_m=0.0, + left_m=0.0, + yaw_rad=0.0, + neck_pitch_delta=0.0, + head_pitch_delta=pitch, + head_yaw_delta=yaw, + head_roll_delta=0.0, + ) + for _ in range(201) + ] + + +def test_tasks_require_distinct_head_pose() -> None: + for name, yaw, pitch in [ + ("stand", 0, 0), + ("look_left", 0.3, 0), + ("look_right", -0.3, 0), + ("look_up", 0, -0.4), + ("look_down", 0, 0.25), + ]: + metrics = trajectory(yaw, pitch) + for other in ("stand", "look_left", "look_right", "look_up", "look_down"): + result = stationary_head_outcome(TASK[other], metrics, 200) + assert result["accepted"] == (name == other) + json.dumps(result) # Run reports must also accept NumPy-derived checks. + + +def test_tail_error_cannot_cancel_or_hide_another_joint() -> None: + metrics = trajectory() + for i, row in enumerate(metrics): + row["head_yaw_delta"] = 0.3 if i % 2 else -0.3 + assert not stationary_head_outcome(TASK["stand"], metrics, 200)["accepted"] + assert not stationary_head_outcome(TASK["look_left"], trajectory(0.3, 0.25), 200)["accepted"] + + +def test_fall_excursion_or_incomplete_rollout_cannot_pass() -> None: + for key, value in [("stable", False), ("forward_m", 0.1), ("yaw_rad", 0.6)]: + metrics = trajectory(0.3) + metrics[20][key] = value + assert not stationary_head_outcome(TASK["look_left"], metrics, 200)["accepted"] + assert not stationary_head_outcome(TASK["look_left"], trajectory(0.3)[:50], 200)["accepted"] + + +def test_teacher_neck_compensation_is_not_an_unrequested_zero_target() -> None: + metrics = trajectory(pitch=-0.28) + for row in metrics: + row["neck_pitch_delta"] = -0.23 + assert stationary_head_outcome(TASK["look_up"], metrics, 200)["accepted"] + assert not stationary_head_outcome(TASK["look_down"], metrics, 200)["accepted"] diff --git a/training/tests/test_motion_quality.py b/training/tests/test_motion_quality.py new file mode 100644 index 0000000..5f4edd5 --- /dev/null +++ b/training/tests/test_motion_quality.py @@ -0,0 +1,69 @@ +"""Motion rubrics reject wrong direction, transient motion, and endpoint cancellation.""" + +import json + +from duckmind_training.contract import TASKS +from duckmind_training.correction_quality import locomotion_outcome + +TASKS_BY_NAME = {task.name: task for task in TASKS} + + +def motion(forward: float = 0, yaw: float = 0, speed: float = 0) -> list[dict]: + return [ + dict( + stable=True, + forward_m=forward * i / 200, + left_m=0, + yaw_rad=yaw * i / 200, + tail_speed_mps=speed, + gravity_z=-1.0, + trunk_z_m=0.12, + ) + for i in range(201) + ] + + +def test_backward_requires_signed_progress_and_a_stable_tail() -> None: + task = TASKS_BY_NAME["backward"] + good = motion(forward=-0.4) + assert locomotion_outcome(task, good, 200)["accepted"] + assert not locomotion_outcome(task, motion(forward=0.4), 200)["accepted"] + for index, row in enumerate(good): + row["left_m"] = 0.4 * index / 200 + assert not locomotion_outcome(task, good, 200)["accepted"] + + +def test_turns_require_continuing_direction_and_limited_drift() -> None: + left, right = TASKS_BY_NAME["turn_left"], TASKS_BY_NAME["turn_right"] + good = motion(yaw=4.0) # accumulated yaw must not wrap at pi + result = locomotion_outcome(left, good, 200) + assert result["accepted"] + json.dumps(result) + assert not locomotion_outcome(right, good, 200)["accepted"] + for row in good[150:]: + row["yaw_rad"] = 3.0 + assert not locomotion_outcome(left, good, 200)["accepted"] + good = motion(yaw=4.0) + good[40]["left_m"] = 0.25 + assert not locomotion_outcome(left, good, 200)["accepted"] + + +def test_stop_speed_cannot_be_replaced_by_net_displacement() -> None: + task = TASKS_BY_NAME["stop_forward"] + assert locomotion_outcome(task, motion(speed=0.01), 200)["accepted"] + assert not locomotion_outcome(task, motion(speed=0.1), 200)["accepted"] + rows = motion(speed=0.01) + rows[50]["forward_m"] = 0.31 + assert not locomotion_outcome(task, rows, 200)["accepted"] + + +def test_incomplete_and_fallen_motion_cannot_pass() -> None: + for name, rows in [ + ("backward", motion(forward=-0.4)), + ("turn_left", motion(yaw=4.0)), + ("stop_backward", motion(speed=0.01)), + ]: + task = TASKS_BY_NAME[name] + assert not locomotion_outcome(task, rows[:60], 200)["accepted"] + rows[30]["stable"] = False + assert not locomotion_outcome(task, rows, 200)["accepted"]