Harden private A100 SFT smoke preflight#3
Conversation
A100 SFT smoke runtime follow-up (2026-07-14)Status: PREFLIGHT_BLOCKED. This is not
Must Fix implementation
Archived OpenSpec: Validation
Runtime detection
Complete sanitized preflight JSON{"blockers":["MAX_SEQUENCE_LENGTH_EXCEEDED"],"config":{"allow_heavy_training":true,"base_model_public_id":"Qwen/Qwen2.5-7B-Instruct","bf16":true,"config_sha256":"2b0631a1ae11c3bff56313356c633bd844e4182a527bc125d876ff4e6ef3a8a6","dataset_split":"train","dtype_is_bfloat16":true,"fp16":false,"gradient_accumulation_steps":1,"gradient_checkpointing":true,"local_files_only":true,"logging_steps":1,"lora_policy_valid":true,"low_cpu_mem_usage":true,"max_seq_length":1024,"max_steps":1,"max_train_rows":2,"minimum_free_disk_gib":20.0,"per_device_train_batch_size":1,"private_file":{"git_ignored":true,"git_tracked":false,"nonsymlink":true,"under_private_runtime":true},"report_to_empty":true,"save_strategy":"no","seed":42,"tf32":true,"torch_dtype_is_bfloat16":true,"trust_remote_code":false,"use_cache":false},"dataset":{"manifest_file":"manifest_public_sample.json","manifest_id":"public-sample-20260619T090925Z","manifest_sha256":"f866c173795e97953b1dec85611b405867d0a29497910282f99d399f109cda95","non_train_rows_selected":0,"selected_row_count":2,"selected_row_ids_sha256":"0fdcbfe62867b87ededaf2c99a122438cb4eb89ef214c4e02fca703805738409","selected_split":"train","sft_file":"sft_public_sample.jsonl","sft_sha256":"4b677420f766555c04199f15f69f41f3b3ad36ad3cd5c33d2b40b0e3f8573587"},"git":{"commit_sha":"f214b9be551e364742c2b4a07c341f5062dca552","tracked_worktree_clean":true},"gpu":{"bf16_supported":true,"compute_capability":"8.0","compute_process_count":0,"cuda_version":"12.9","explicit_selection":true,"free_memory_gib":78.734,"idle_verified":true,"minimum_free_memory_gib":35.0,"minimum_memory_gib":35.0,"name":"NVIDIA A100-SXM4-80GB","total_memory_gib":79.151,"visible_device_count":1},"model":{"geometry_matches_qwen2_5_7b":true,"local_files_only":true,"minimum_weight_bytes":12884901888,"public_id":"Qwen/Qwen2.5-7B-Instruct","snapshot_revision_sha256":"2a1aaaf30e79bc9817f385e41260f822b4533f570a62f7253afc536e8f724363","stable_fingerprints":{"config.json":"7463bb0ea78315365e6c6b74de4e73bbcc8359dfb0c5a737584e077d42c0b03c","generation_config.json":"3a8f9087e486054c8a4a08dae2e5a3ba62e23da212b5b8c08bc42cb983c3459f","merges.txt":"599bab54075088774b1733fde865d5bd747cbcc7a547c5bc12610e874e26f5e3","model.safetensors.index.json":"624bf7c47cd12468fdc16e38a47cf4f19e0415b859a223ba3c027eed2f0e1028","tokenizer.json":"c0382117ea329cdf097041132f6d735924b697924d6f6fc3945713e96ce87539","tokenizer_config.json":"5b5d4f65d0acd3b2d56a35b56d374a36cbc1c8fa5cf3b3febbbfabf22f359583","vocab.json":"ca10d7e9fb3ed18575dd1e277a2579c16d108e32f27439684afa0e10b1440910"},"total_weight_bytes":15231271888,"weight_inventory":[{"name":"model-00001-of-00004.safetensors","size":3945441440},{"name":"model-00002-of-00004.safetensors","size":3864726352},{"name":"model-00003-of-00004.safetensors","size":3864726424},{"name":"model-00004-of-00004.safetensors","size":3556377672}]},"objective":{"assistant_target_present":false,"max_sequence_length":1024,"prompt_labels_masked":false,"records_checked":0},"output":{"blockers":[],"free_disk_gib":10142733.748,"minimum_free_disk_gib":20.0,"output_path_sha256":"6d5c0189521d027b29cdb81e4a139a419519312c4cd9a1ae6ac32630fde3b368","ready":true,"root_path_sha256":"c26d3afb706ca1995065c67df889a9c850ddf18072679c265574ec6bcfca84da","writable":true},"ready":false,"runtime":{"missing":[],"pip_check":"ok","python":"3.11.4","python_requirement":">=3.10","versions":{"accelerate":"1.13.0","datasets":"5.0.0","peft":"0.18.1","torch":"2.10.0+cu129","transformers":"5.5.0","trl":"1.5.1"}},"schema_version":"voice2task-sft-preflight-v1","status":"blocked"}Repository status after push is clean and synchronized with the remote branch. No full training, DPO, GRPO, prediction, evaluation, lockbox access, or model-improvement claim was performed. |
Real 1-step runtime evidence — postcondition failure (2026-07-15)Authoritative external status: The earlier
A fresh shared preflight returned exit Exactly one real {"adapter_files":[{"name":"README.md","sha256":"088fc697ae52cb14f1e808e5c967c8be13cb0256da184ad882dd41ba8e95c617","size":5262},{"name":"adapter_config.json","sha256":"5212eea727806033f85531cabb70ca59f6c93152aa3f783c543e169314644760","size":1029},{"name":"adapter_model.safetensors","sha256":"13fce8410c1c21d86c1dc628c9f20e9477294c707d1fe7235e4a111834a8fab7","size":20214760}],"clean_evaluation":{"acquisition_source_status":"UNAVAILABLE","authoritatively_bound_binding_count":0,"clean_population_status":"NOT_MATERIALIZED","execution_readiness":false,"freeze_authorized":false,"human_acceptance_status":"NOT_RECORDED","protocol_freeze_status":"NOT_FROZEN"},"observed_optimizer_steps":1,"smoke_status":"SMOKE_COMPLETED","stage":"sft","train_result_metrics":{"adapter_state_digest_after":"a2395bb971e32f30cf760f2da2fc9b06e727c82a2ffac6b753ac7b0b21f7105d","adapter_state_digest_before":"ba19fe10ce78816dbb3e3944dda2ae6d3e354277ad0ce1cad58279ec1ed68a52","adapter_tensor_count":224,"all_adapter_tensors_finite":true,"changed_adapter_tensor_count":112,"epoch":0.5,"train_loss":0.0034358606208115816,"train_runtime":0.9902,"train_samples_per_second":1.01,"train_steps_per_second":1.01,"trainable_parameter_count":5046272},"training_budget":{"configured_max_steps":1,"effective_batch_size":1,"gradient_accumulation_steps":1,"num_train_epochs":1.0,"observed_optimizer_steps":1,"per_device_train_batch_size":1,"scheduler_max_steps":1,"step_matched_not_token_matched":true,"step_matching_unit":"optimizer_steps","target_tokens_per_single_pass":112,"target_tokens_seen_estimate":56,"target_tokens_seen_status":"estimated_from_label_tokens_and_step_budget","theoretical_examples_seen":1,"train_row_count":2},"training_rows_used":2,"training_status":"training_completed"}Additional verified postconditions:
Blocking postconditionCLI stdout was not one JSON document:
Therefore the embedded metadata value Sanitized ready preflight JSON{"blockers":[],"config":{"allow_heavy_training":true,"base_model_public_id":"Qwen/Qwen2.5-7B-Instruct","bf16":true,"config_sha256":"1a3932231d9c6df01167024bbfaf81ad549bf20ac672c15c5ee5b4a7b889f734","dataset_split":"train","dtype_is_bfloat16":true,"fp16":false,"gradient_accumulation_steps":1,"gradient_checkpointing":true,"local_files_only":true,"logging_steps":1,"lora_policy_valid":true,"low_cpu_mem_usage":true,"max_seq_length":1077,"max_steps":1,"max_train_rows":2,"minimum_free_disk_gib":20.0,"per_device_train_batch_size":1,"private_file":{"git_ignored":true,"git_tracked":false,"nonsymlink":true,"under_private_runtime":true},"report_to_empty":true,"save_strategy":"no","seed":42,"tf32":true,"torch_dtype_is_bfloat16":true,"trust_remote_code":false,"use_cache":false},"dataset":{"manifest_file":"manifest_public_sample.json","manifest_id":"public-sample-20260619T090925Z","manifest_sha256":"f866c173795e97953b1dec85611b405867d0a29497910282f99d399f109cda95","non_train_rows_selected":0,"selected_row_count":2,"selected_row_ids_sha256":"0fdcbfe62867b87ededaf2c99a122438cb4eb89ef214c4e02fca703805738409","selected_split":"train","sft_file":"sft_public_sample.jsonl","sft_sha256":"4b677420f766555c04199f15f69f41f3b3ad36ad3cd5c33d2b40b0e3f8573587"},"git":{"commit_sha":"f214b9be551e364742c2b4a07c341f5062dca552","tracked_worktree_clean":true},"gpu":{"bf16_supported":true,"compute_capability":"8.0","compute_process_count":0,"cuda_version":"12.9","explicit_selection":true,"free_memory_gib":78.734,"idle_verified":true,"minimum_free_memory_gib":35.0,"minimum_memory_gib":35.0,"name":"NVIDIA A100-SXM4-80GB","total_memory_gib":79.151,"visible_device_count":1},"model":{"geometry_matches_qwen2_5_7b":true,"local_files_only":true,"minimum_weight_bytes":12884901888,"public_id":"Qwen/Qwen2.5-7B-Instruct","snapshot_revision_sha256":"2a1aaaf30e79bc9817f385e41260f822b4533f570a62f7253afc536e8f724363","stable_fingerprints":{"config.json":"7463bb0ea78315365e6c6b74de4e73bbcc8359dfb0c5a737584e077d42c0b03c","generation_config.json":"3a8f9087e486054c8a4a08dae2e5a3ba62e23da212b5b8c08bc42cb983c3459f","merges.txt":"599bab54075088774b1733fde865d5bd747cbcc7a547c5bc12610e874e26f5e3","model.safetensors.index.json":"624bf7c47cd12468fdc16e38a47cf4f19e0415b859a223ba3c027eed2f0e1028","tokenizer.json":"c0382117ea329cdf097041132f6d735924b697924d6f6fc3945713e96ce87539","tokenizer_config.json":"5b5d4f65d0acd3b2d56a35b56d374a36cbc1c8fa5cf3b3febbbfabf22f359583","vocab.json":"ca10d7e9fb3ed18575dd1e277a2579c16d108e32f27439684afa0e10b1440910"},"total_weight_bytes":15231271888,"weight_inventory":[{"name":"model-00001-of-00004.safetensors","size":3945441440},{"name":"model-00002-of-00004.safetensors","size":3864726352},{"name":"model-00003-of-00004.safetensors","size":3864726424},{"name":"model-00004-of-00004.safetensors","size":3556377672}]},"objective":{"assistant_target_present":true,"max_sequence_length":1077,"maximum_observed_tokens":1077,"prompt_labels_masked":true,"records_checked":2},"output":{"blockers":[],"free_disk_gib":10142325.843,"minimum_free_disk_gib":20.0,"output_path_sha256":"3016709988fdcbd111576e8e41a21782e357575052879c9897de01cd98ad541b","ready":true,"root_path_sha256":"c26d3afb706ca1995065c67df889a9c850ddf18072679c265574ec6bcfca84da","writable":true},"ready":true,"runtime":{"missing":[],"pip_check":"ok","python":"3.11.4","python_requirement":">=3.10","versions":{"accelerate":"1.13.0","datasets":"5.0.0","peft":"0.18.1","torch":"2.10.0+cu129","transformers":"5.5.0","trl":"1.5.1"}},"schema_version":"voice2task-sft-preflight-v1","status":"ready"}Stage 2 remains blocked and was not started. Fixing stdout and authorizing any second real smoke are separate future decisions. |
Bounded stdout-contract fix closeout
Must Fix implementation
TDD and review
Final validation
Runtime truth boundaryNo second real smoke was run in this fix. The prior authorized one-step process still remains Runtime Postcondition Failed because its complete stdout was not a single JSON document. Although the prior run produced genuine one-step adapter-update evidence, this PR does not describe it as No new GPU/model probe or preflight was run. The prior sanitized preflight and one-run evidence remain in the earlier audit comment: #3 (comment) Clean-evaluation truth remains unchanged: source unavailable, authoritative binding count 0, human acceptance not recorded, protocol not frozen, clean population not materialized, No full training, DPO, GRPO, prediction, evaluation, lockbox access, or model-improvement claim was performed. |
Real 1-step A100 SFT smoke rerun after CLI stdout fix (2026-07-15)Authoritative status: SMOKE_COMPLETED.
Fresh gates before the only launch
Only authorized launch
Independent review and validation
No prediction, evaluation, DPO, GRPO, public test, lockbox, clean-evaluation population access, extra seed, or metric experiment was run. This is bounded infrastructure-smoke evidence only; it is not a quality, held-out, generalization, safety, or production-readiness result. |
边界
4c99e8d6cf1fe2782b2760e36437ff5fae164dff41c53b8eceaafef339e8984e861f16f749e85f383a4bf38a574ed81534eaea1103b491b137da3049Must Fix 实现位置
Public result 严格 allowlist
src/voice2task/training.py:359_public_sft_preflight_resultsrc/voice2task/training.py:460public_training_resultsrc/voice2task/cli/train.py:100CLI 只输出重建后的单个 JSON。Descriptor-relative output claim
src/voice2task/training.py:860_open_bound_output_parentsrc/voice2task/training.py:1036_claim_sft_output_directoryShared GPU idle/free-memory gate
src/voice2task/training.py:1240_probe_sft_gpuGPU_FREE_MEMORY_INSUFFICIENT、GPU_BUSY、GPU_OCCUPANCY_PROBE_FAILED。src/voice2task/training.py:4552real runner 入口和权重加载紧前均复检。Assistant-only objective
src/voice2task/training.py:3944assistant indices 独立从 tokenizer offsets/span 构造。src/voice2task/training.py:3998验证 prompt 全部 -100、assistant 目标非空且连续、labels/input_ids 对齐、三张量等长。Adapter 真实更新证据
src/voice2task/training.py:2341before/after adapter tensor snapshot。src/voice2task/training.py:2379digest、changed tensor count、finite 证据。src/voice2task/training.py:2401smoke postconditions 和 adapter 文件 SHA-256。新增重点回归
test_public_training_result_is_exact_allowlist_for_arbitrary_absolute_pathstest_public_training_result_rebuilds_preflight_without_unknown_nested_private_fieldstest_public_training_result_rejects_private_paths_in_every_allowed_string_containertest_output_claim_blocks_root_same_path_inode_exchange_without_external_mutationtest_output_claim_blocks_parent_exchange_to_external_symlink_with_zero_external_mutationtest_output_claim_checks_parent_path_identity_after_open_before_leaf_mkdirtest_output_claim_rejects_concurrent_final_leaf_creationtest_gpu_probe_reports_free_memory_and_idle_aggregatetest_gpu_probe_blocks_insufficient_free_memory_or_busy_devicetest_gpu_probe_fails_closed_when_occupancy_probe_failstest_sft_preflight_rejects_one_prompt_label_leak_even_with_other_masked_tokenstest_real_sft_rechecks_gpu_idle_state_before_model_weight_loadtest_adapter_state_evidence_requires_real_finite_changetest_smoke_postconditions_require_adapter_update_evidenceOpenSpec
openspec/specs/supervised-contract-tuning/spec.md已同步。openspec/changes/archive/2026-07-14-enable-private-a100-sft-smoke-v1/openspec list --json:active changes = 0。验证
PYTHONPATH=src pytest -q→ 1413 passedPYTHONPATH=src ruff check .→ PASSOPENSPEC_TELEMETRY=0 openspec validate --all --strict→ 15 passed, 0 failedPYTHONPATH=src python scripts/check_current_truth_surface.py→ exit 0git diff --check→ PASSslot_error_analysis.py:574copy_backed_shadow_interface.py:357本次触及的
training.py/cli/train.py无诊断、无新增错误。Stop hook
本机
~/.codex/hooks.json已移除会因缺少tree-sitter而 exit 1 的graph-on-stop.js子 hook,保留 Hivemindstop.js;使用合法 hook JSON payload 直接验证stop_hook_exit=0。这是本机配置修复,不属于仓库提交。Committed-HEAD preflight
Exit code:
1。stdout 为单个脱敏 JSON,stderr 无第二个结果:{ "blockers": [ "CONFIG_FILE_MISSING" ], "config": {}, "dataset": {}, "git": {}, "gpu": {}, "model": {}, "objective": {}, "output": {}, "ready": false, "runtime": {}, "schema_version": "voice2task-sft-preflight-v1", "status": "blocked" }.gitignore保护,未提交。CONFIG_FILE_MISSING。最终状态
git status --short:空git diff --check:PASSexecution_readiness=false。