drewOrc · Sep 30, 2026 · Sep 30, 2026 · Sep 30, 2026 · Sep 30, 2026
diff --git a/DEVLOG.md b/DEVLOG.md
@@ -4,6 +4,36 @@
 
 ---
 
+## 2026-09-30：AC1b 兩次嘗試失敗；修正各 seed 選不同聚合時的畫圖缺陷，新增嘗試帳本
+
+### 本次工作 / 執行摘要
+- 狀態不變：RQ1 到 RQ5 完成；**Tier 1 驗收未完成，AC1b 尚未通過**。
+- **AC1b 第 1 次**（`3992840ddb3f`，2026-09-29 07:40 到 14:24 UTC）：訓練、AC2、pilot 全過；Haiku 前的 `count_tokens` 在 Anthropic 認證事故期間連 5 次 HTTP 503，花費 US$0。comparison 機械判定 FAIL；Drew 判定為基礎設施事故、不視為 AC1b 失敗，帳本記為 `INFRASTRUCTURE INTERRUPTED`，同一 id 與身分續跑。
+- **AC1b 第 2 次**（續跑，2026-09-30 01:13 到 01:31 UTC）：Haiku 8,600/8,600、US$3.178751、0 parse failed、20 筆預測與原 run 不同；analysis、latency、cost 完成；`make figures` 失敗：ModernBERT k=100 三個 seed 在 validation 選了 argmax、summed、summed，`figures.selected_aggregation` 直接丟 ValueError。原始 run 三個 seed 都選 argmax，所以這條路徑從沒跑過。判 FAIL（程式缺陷）。
+- **修正**：找過所有讀聚合方式的地方。`analysis` 本來就讓每個 seed 用自己 validation 選的聚合（`final`、`diagnostics` 逐 seed 算好再合併）；`report` 的聚合說明已會列出逐 seed 選擇；`cost`、`comparison`、學習曲線、router 圖、門檻圖都讀 `final` 或 `diagnostics`，不受影響。只有 risk-coverage 圖假設三個 seed 同一種。`curves.json` 對每種聚合只存三 seed 的 mean 與 std，沒有逐 seed 曲線，而補存會改動原始 `curves.json` 的位元組，所以改成：seed 一致的 k 維持一個面板；不一致的 k 每種被選到的聚合各一個面板（固定 argmax、summed 順序），標題寫出哪些 seed 的 final router 用它；README 圖說在這種情況自動加一句說明。沒有任何選擇用到 test，也沒有固定成某一種。
+- **帳本**：`docs/ac1b/attempts.json` 記錄每次嘗試（commit、id、UTC 起訖、結果與原因、實際 Haiku 支出），`docs/ac1b/attempt-<n>/` 放 comparison 與事故說明。comparison 與 README 的預算改成三部分：原始實驗 US$3.19（固定）、每次嘗試支出與原因、reproduction-validation 累計。PLAN §5.1 預算規則補一行：每次嘗試各自獨立 US$5 上限（Drew，2026-09-30）。
+- **聚合選擇只列不判**（第 3 次嘗試實跑前寫定，PLAN §5.1 實跑前補充追加一行）：comparison 新增一節，對每個有 final router 的組（25 組）列出原始與重跑每個 seed 在 validation 選的 8 類聚合與是否相同，狀態固定 `LISTED, NOT JUDGED`，不計入 REVIEW、不影響判定。用第 2 次重跑輸出實測：7 組不同，其中 ModernBERT k=100 原始 argmax/argmax/argmax、重跑 argmax/summed/summed（seeds 43、44 改選），可解釋 small-only 91.87 → 91.42 的 REVIEW；REVIEW 計數仍是 192。
+- **帳本結果分類（PR #20 審查後）**：依 Drew 2026-09-29 的決定（由協調者轉達），第 1 次記為 `INFRASTRUCTURE INTERRUPTED`（`failure_kind: infrastructure`），附外部事故證據（status.claude.com 事故、2026-09-29 14:21 UTC、503 credential validation failed）與決定原文；comparison 的機械判定 FAIL 另存為 `comparison_verdict`，證據檔不改寫。收緊條件：`failure_kind` 只能是 infrastructure、program defect、ac2；INTERRUPTED 只允許 infrastructure 且需證據與決定；同一 infrastructure 錯誤類型連續第二次必須是 `INFRASTRUCTURE BLOCKED`（AC1b 暫停）；程式缺陷只能是 FAIL。AC1b 最終判定仍只有 PASS 或 FAIL：INTERRUPTED 不算失敗也不算通過，只有某次嘗試 PASS 才完成驗收。risk-coverage 混合面板標題改為「mean of 3 seeds; final router of seed …」。
+- 下一步依 Drew 決定：合併後在新 commit 完整重跑 AC1b（新的一次嘗試，新 id、新 journal、獨立 US$5）；不拿第 2 次的輸出配新 commit 的圖。
+
+### 核心發現 / 數據
+- 用第 2 次重跑輸出的副本跑 `figures` 與 `report`（`--results-root`、`--readme-out`）：4 張圖與 report 都完成；risk-coverage 為 (a) k=10 summed、(b) k=100 argmax（seed 42）、(c) k=100 summed（seeds 43、44）。
+- 第 2 次重跑中其他聚合不一致的組：BERT k=10、25、50、100，ModernBERT k=25，OOS 0 消融 k=100，TF-IDF k=5、10、25。
+- 原始結果上 `make figures`、`make report` 兩次，`results/` 與 README 生成區塊逐位元組不變。
+
+### Blockers / 遇到的問題
+- (無)
+
+### Next
+- [ ] 合併後在新 commit 完整重跑 AC1b（第 3 次嘗試），結果寫進 `docs/ac1b/attempts.json`
+
+### Files / Budget
+- 新增：`src/tinyrouter/ac1b_ledger.py`；`tests/test_ac1b_ledger.py`；`docs/ac1b/`（`attempts.json`、`README.md`、`attempt-1/`、`attempt-2/`）
+- 修改：`src/tinyrouter/figures.py`、`report.py`、`comparison.py`；`tests/test_figures.py`、`test_report.py`、`test_comparison.py`；`README.md`、`docs/PLAN.md`、`DEVLOG.md`
+- API 花費：本次 US$0。AC1b 累計（reproduction-validation）：US$3.178751；原始實驗成本維持 US$3.19
+
+---
+
 ## 2026-09-29（深夜，七）：PR #19 審查修正（R1 到 R6），AC1b 實跑前固定比較範圍
 
 ### 本次工作 / 執行摘要

diff --git a/README.md b/README.md
@@ -237,7 +237,25 @@ Training on Apple MPS is not bit-for-bit deterministic, so a rerun is not expect
 
 A clean clone is not a clean cache: the base models and the CLINC150 files come from the shared Hugging Face cache (`~/.cache/huggingface` by default). That does not affect correctness, since every model is pinned to a revision and every data file is checked against its SHA-256, but to download everything afresh point the cache at an empty directory, for example `HF_HOME=/tmp/tinyrouter-hf make reproduce`.
 
-**Status: AC1b has not been run yet.** Until it has passed, the status is RQ1 to RQ5 complete, Tier 1 acceptance not complete.
+**Status: AC1b has not passed yet.** It has been attempted twice at `3992840ddb3f`: attempt 1 was interrupted by an external authentication incident (recorded as infrastructure-interrupted, neither a failure nor a pass), and attempt 2 failed on a defect in the figure code when seeds chose different 8-way aggregations; every attempt is recorded in [`docs/ac1b/`](docs/ac1b/README.md). Until it has passed, the status is RQ1 to RQ5 complete, Tier 1 acceptance not complete.
+
+AC1b budget. Every attempt has its own US$5 Haiku cap, enforced per reproduction id:
+
+<!-- BEGIN AC1b budget, generated from docs/ac1b/attempts.json; do not edit by hand -->
+- Original experiment (AC6, fixed): US$3.19
+- AC1b attempts (reproduction-validation, each capped at US$5):
+
+| attempt | reproduction id | result | reason | Haiku spend |
+|---|---|---|---|---|
+| 1 | `3992840ddb3f` | INFRASTRUCTURE INTERRUPTED | infrastructure: Haiku count_tokens returned HTTP 503 credential validation failed 5 times during an Anthropic authentication incident; no classification request sent | US$0.000000 |
+| 2 | `3992840ddb3f` | FAIL | program defect: make figures stopped because seeds chose different 8-way aggregations for modernbert/k100 (argmax, summed, summed); Haiku 8600/8600, 0 parse failures, 20 predictions differ | US$3.178751 |
+
+- Reproduction-validation total (all AC1b attempts): US$3.178751
+
+Reproduction-validation spend is never reported as original experiment cost and does not change the AC6 conclusion (Haiku spend at most US$5).
+
+INFRASTRUCTURE INTERRUPTED is neither an AC1b failure nor a pass; AC1b acceptance is completed only by an attempt whose result is PASS.
+<!-- END AC1b budget -->
 
 ## Reproducibility notes
 

diff --git a/docs/PLAN.md b/docs/PLAN.md
@@ -127,9 +127,11 @@ TinyRouter 問三件事：**小模型要多少標註資料才夠？它知不知
 
 - 判定範圍（偏離原始 mean ± std 標 `REVIEW REQUIRED`）固定為：README 首屏數字；README router 表（LLM-only，以及 ModernBERT k=10 與 k=100 的 small-only、hybrid 目標 2% 與 5%、oracle，各報 8 類準確率、OOS recall、高信心 OOS 誤派、LLM 呼叫率）；學習曲線（兩個 encoder、每個 k 的 8 類準確率與 OOS recall）；`ablation_comparison`（OOS 250 對 0 的 AUROC、AUPRC 與 router 行為）；所有門檻診斷統計。
 - **只列不判**：延遲（`results/efficiency/*`）、k=100 的訓練時間與峰值記憶體、成本模型（`results/cost/cost.json`）隨機器變動。comparison 列出原始值、重跑值與差異，狀態固定為 `LISTED, NOT JUDGED`，不計入 `REVIEW REQUIRED`，也不影響判定。
+- **聚合選擇只列不判**（2026-09-30，第 2 次嘗試觀察到差異後追加，第 3 次嘗試實跑前寫定）：每個有 final router 的組，comparison 列出原始與重跑每個 seed 在 validation 選的 8 類聚合方式與是否相同，狀態固定為 `LISTED, NOT JUDGED`，不計入 `REVIEW REQUIRED`，也不影響判定。
 - 續跑只限同一身分：每一步記錄 HEAD、`uv.lock` 與 `configs/` 的 SHA-256、Python、torch 與 transformers 版本；同一個 reproduction 目錄若由不同身分開始，拒絕續跑，必須換新 id，所以一份 comparison 不會混到兩個 commit 的輸出（第 1 條的落實方式）。
 
 **預算規則**：原始 AC6 實驗的 Haiku 花費固定是 US$3.19（完整 run US$3.18 加 smoke），這個數字不再變動。AC1b 的 Haiku 是另一次、明確標記為 reproduction-validation 的執行，有獨立的 US$5 上限，花費寫在 reproduction 目錄自己的紀錄裡；兩者不得混稱為原始實驗成本，也不相加。
+- 每次 AC1b 嘗試各自獨立 US$5 上限（Drew，2026-09-30）。嘗試帳本：`docs/ac1b/attempts.json`（人讀版 `docs/ac1b/README.md`，各次證據在 `docs/ac1b/attempt-<n>/`）。
 
 ## 6. 風險
 

diff --git a/docs/ac1b/README.md b/docs/ac1b/README.md
@@ -0,0 +1,35 @@
+# AC1b attempts
+
+AC1b (docs/PLAN.md section 5.1) is the whole study rerun from a clean clone with `make setup && make reproduce`. This directory records every attempt, whether it passed or not, so that a failed attempt and its cost are never hidden by a later one.
+
+- `attempts.json`: the ledger, one entry per attempt: commit, reproduction id, start and end (UTC), result and reason, actual Haiku spend. `python -m tinyrouter.ac1b_ledger` checks it and rewrites the budget block below and in the top-level README; `tests/test_ac1b_ledger.py` fails when either is stale.
+- `attempt-<n>/`: the attempt's `comparison.md` as written by `make reproduce`, and `INCIDENT.md`, a note on why it stopped. Evidence files are kept as written: attempt 1's `INCIDENT.md` says it started "about 07:45" UTC; the ledger uses 07:40:02 UTC, the start of the first step in `steps.json`, which is authoritative.
+
+Results (validated by `tinyrouter.ac1b_ledger`):
+
+- `PASS` or `FAIL`: the comparison's verdict under the frozen criteria. `comparison_verdict` always records that mechanical verdict, and the evidence is never rewritten.
+- `INFRASTRUCTURE INTERRUPTED`: allowed only with `failure_kind: infrastructure`, external incident evidence (incident, time, source, error type) and the decision that classified it. INFRASTRUCTURE INTERRUPTED is neither an AC1b failure nor a pass; AC1b acceptance is completed only by an attempt whose result is PASS. Attempt 1 is recorded this way under Drew's decision of 2026-09-29, while its comparison verdict stays FAIL.
+- `INFRASTRUCTURE BLOCKED`: required when the same infrastructure error type recurs on the next attempt. AC1b is then paused, not resumed again.
+- `failure_kind` is one of `infrastructure`, `program defect`, `ac2`; a program defect can only be `FAIL`.
+
+Budget rules (Drew, 2026-09-29 and 2026-09-30):
+
+- The original AC6 experiment cost is fixed at US$3.19. No AC1b attempt changes it, and reproduction-validation spend is never reported as original experiment cost or used to revisit the AC6 conclusion (Haiku spend at most US$5).
+- Every AC1b attempt has its own US$5 cap. `make reproduce` writes the Haiku journal under `reproduction/<id>/results` and runs it with `MAX_USD=5`; the cap counts only that journal, so an attempt at a new commit starts at US$0. Attempt 2 resumed attempt 1 under the same id and identity and so shared its journal and cap (attempt 1 spent US$0).
+- A comparison never mixes the outputs of two commits. After a code fix, AC1b is run again in full at the new commit.
+
+<!-- BEGIN AC1b budget, generated from docs/ac1b/attempts.json; do not edit by hand -->
+- Original experiment (AC6, fixed): US$3.19
+- AC1b attempts (reproduction-validation, each capped at US$5):
+
+| attempt | reproduction id | result | reason | Haiku spend |
+|---|---|---|---|---|
+| 1 | `3992840ddb3f` | INFRASTRUCTURE INTERRUPTED | infrastructure: Haiku count_tokens returned HTTP 503 credential validation failed 5 times during an Anthropic authentication incident; no classification request sent | US$0.000000 |
+| 2 | `3992840ddb3f` | FAIL | program defect: make figures stopped because seeds chose different 8-way aggregations for modernbert/k100 (argmax, summed, summed); Haiku 8600/8600, 0 parse failures, 20 predictions differ | US$3.178751 |
+
+- Reproduction-validation total (all AC1b attempts): US$3.178751
+
+Reproduction-validation spend is never reported as original experiment cost and does not change the AC6 conclusion (Haiku spend at most US$5).
+
+INFRASTRUCTURE INTERRUPTED is neither an AC1b failure nor a pass; AC1b acceptance is completed only by an attempt whose result is PASS.
+<!-- END AC1b budget -->
diff --git a/docs/ac1b/attempt-1/INCIDENT.md b/docs/ac1b/attempt-1/INCIDENT.md
@@ -0,0 +1,13 @@
+# AC1b attempt 1: interrupted by an external authentication incident
+
+- Reproduction id: `3992840ddb3f` (clean clone of `main` at `3992840ddb3f`, `uv.lock` SHA-256 prefix `d6b5940a86b5c28c`).
+- Started: 2026-09-29 about 07:45 UTC. Failed: 2026-09-29 about 14:24 UTC.
+- Completed and passed before the failure (all at `3992840ddb3f`): original-haiku, ac2, pilot-lr, pilot-steps, baselines, curve-bert, curve-modernbert, oos-ablation, verify-logits.
+  - AC2 rerun: 96.60 / 96.36 / 96.49 percent (seeds 42 / 43 / 44), all at or above 95.7.
+  - Pilots selected the same values as the original run (lr 5e-5 for both encoders, S_min 400).
+- Failed step: `llm`. The pre-run token count (`messages.count_tokens`) returned HTTP 503 `InternalServerError` with the message "credential validation failed" on all 5 attempts. No classification request was sent.
+- Cost of attempt 1: US$0.00 (token counting is not billed; the reproduction journal holds no records).
+- Cause: an Anthropic incident, "Elevated errors on claude.ai, Claude Code and Claude Cowork", Investigating since 2026-09-29 14:21 UTC, described as failed requests and being asked to sign in again (status.claude.com). The API component was listed as operational, but requests from this machine received 503 during the incident window.
+- Verdict of attempt 1 under the frozen criteria: FAIL (flow did not complete). Drew's decision (2026-09-29): this is an infrastructure availability failure, not a failure of training or of the protocol; resume after the incident with the same reproduction id and identity. If the same authentication failure recurs, pause and mark AC1b infrastructure-blocked.
+- Before resuming (2026-09-30 01:13 UTC): a free token count succeeded; HEAD, working tree, and `uv.lock` unchanged; the reproduction LLM journal was empty, so no prediction is paid for twice. No hyperparameter, threshold, or result file was changed.
+- Evidence kept: `comparison.md`, `comparison.json`, `steps.json`, and `run.log` from attempt 1 (no API key in any of them).
diff --git a/docs/ac1b/attempt-1/comparison.md b/docs/ac1b/attempt-1/comparison.md
@@ -0,0 +1,53 @@
+# AC1b comparison
+
+**Verdict: FAIL** (0 item(s) REVIEW REQUIRED). Criteria: docs/PLAN.md section 5.1 (frozen 2026-09-29). FAIL only when the flow or AC2 fails.
+
+## Flow
+
+| step | status | commit | completion line(s) |
+|---|---|---|---|
+| original-haiku | PASS | 3992840ddb3f | `verified 1/1 release files` |
+| ac2 | PASS | 3992840ddb3f | `wrote reproduction/3992840ddb3f/results/ac2.json: PASS` |
+| pilot-lr | PASS | 3992840ddb3f | `wrote reproduction/3992840ddb3f/results/pilots/lr.json` |
+| pilot-steps | PASS | 3992840ddb3f | `wrote reproduction/3992840ddb3f/results/pilots/steps.json` |
+| baselines | PASS | 3992840ddb3f | `completed 36/36 baseline points` |
+| curve-bert | PASS | 3992840ddb3f | `completed 18/18 encoder points (bert)` |
+| curve-modernbert | PASS | 3992840ddb3f | `completed 18/18 encoder points (modernbert)` |
+| oos-ablation | PASS | 3992840ddb3f | `completed 3/3 ablation points` |
+| verify-logits | PASS | 3992840ddb3f | `OK 75 archive(s) match reproduction/3992840ddb3f/results/logits-manifest.json` |
+| llm | FAIL | 3992840ddb3f | none |
+
+## AC2 (threshold 95.7%, every seed): PASS
+
+| seed | rerun (%) | original (%) | passed |
+|---|---|---|---|
+| 42 | 96.60 | 96.51 | True |
+| 43 | 96.36 | 96.40 | True |
+| 44 | 96.49 | 96.22 | True |
+
+## Haiku (reproduction-validation run)
+
+```json
+{
+  "reached": false,
+  "checks_passed": false,
+  "problems": [
+    "no reproduction run"
+  ]
+}
+```
+
+## Budget (separate, never summed)
+
+- original AC6 experiment: US$3.19 (fixed)
+- reproduction-validation: US$None (cap US$5)
+
+## Pilots
+
+| pilot | original | rerun | status |
+|---|---|---|---|
+| lr | {'bert': 5e-05, 'modernbert': 5e-05} | {'bert': 5e-05, 'modernbert': 5e-05} | OK |
+| steps | 400 | 400 | OK |
+
+Numbers: not reached (the analysis step did not finish).
+
diff --git a/docs/ac1b/attempt-2/INCIDENT.md b/docs/ac1b/attempt-2/INCIDENT.md
@@ -0,0 +1,11 @@
+# AC1b attempt 2: stopped by a figure-code defect
+
+Written 2026-09-30 from the attempt's `run.log`, `steps.json` and `comparison.md` (the evidence directory had no incident note for this attempt).
+
+- Reproduction id: `3992840ddb3f`, the same id and identity as attempt 1 (commit `3992840ddb3f`, same `uv.lock`, `configs/`, Python, torch and transformers). It resumed attempt 1: the nine steps that had passed were skipped because their outputs still had the recorded SHA-256.
+- Started (resumed `llm` step): 2026-09-30 01:13:40 UTC. Failed: 2026-09-30 01:31:53 UTC.
+- Passed in this attempt: llm (Haiku 8,600/8,600, 0 parse failures, 20 predictions different from the original run), verify-llm, analysis, bench-cpu, llm-latency, cost.
+- Haiku spend: US$3.178751, under the US$5 cap of this reproduction id (attempt 1 spent US$0 on the same journal).
+- Failed step: `figures`. `figures.selected_aggregation` raised `ValueError: modernbert/k100: seeds chose different aggregations ['argmax', 'summed', 'summed']; pick one first`. In the original run all three ModernBERT k=100 seeds chose argmax on validation, so this path had never run.
+- Verdict under the frozen criteria (docs/PLAN.md 5.1): FAIL (the flow did not complete). This is a defect in the program, not an infrastructure failure.
+- Drew's decision, relayed by the coordinator (2026-09-30): fix the defect, then run AC1b again in full at the new commit as a new attempt with its own US$5 cap. The outputs of this attempt are not combined with figures or a report from another commit.