research: attention budget measured + Agent Loop Engineering rubric (25/28, gate FAIL)

docs/reportcards/collection.json

@@ -208,5 +208,47 @@
       "source": "PR #47 + PR #291, curl 200"
     },
     "next_lever": "Item 9: the false-green benchmark \u2014 how often the loop reports A when a stronger judge disagrees"
+  },
+  {
+    "id": "rc0007",
+    "delivery": "Part A (@skills, arXiv:2608.12610): residency_audit.py on the live library \u2014 221 entries, 176 distinct = 1.76x the paper's argued 100-slot bound; resident index 62,010 chars (~15.5k tok) vs 2.95M chars of bodies = 47.6x; 17 phantom symlinks, 25 duplicate names, 130/176 overlong descriptions. trigger_reliability.py runs the paper's OWN named future work: top-1 routing 100% (N=10), 100% (N=40), 83.3% (N=80), 44.4% (N=176, 9 usable trials, CI 19-73%) \u2014 subject gpt-oss-120b, probe generator a different family, declared optimistic-bound bias, failed calls excluded from the denominator. TWO self-corrections shipped as commits: a YAML block-scalar parse bug made my first audit numbers wrong (86.6x/72 overlong -> 47.6x/130) AND corrupted the experiment's ground truth (three targets got the identical probe 'Hey, can you pull'); and an eyeballed '~190 MCP tools' was retracted for a counted 132. Part B (Agent Loop Engineering lecture): encoded 16 sections as docs/rubrics/agent-loop-engineering.yml + scripts/audit_loop_rubric.py with observed-only evidence and unmeasured-blocks-the-gate. First run 23/23 -> went back for claims chosen because I expected them to fail -> 25/28, gate FAIL. Closed one gap for real: MemoryStore.success_rate() (in-flight excluded, blocked_safety counts as failure, None not 0.0 when nothing finished, 6 tests). 606 tests pass, ruff clean, 5 commits on branch research/attention-budget.",
+    "objective": "Make loop-engineering-anything's public architecture as honest and reviewable as its engine already is",
+    "created_at": "2026-08-21T11:45:03-05:00",
+    "key_results": [
+      {
+        "text": "A paper's unmeasured central claim becomes measured",
+        "target": "a curve with CIs from a real corpus",
+        "score": 1.0,
+        "evidence": "research/attention-budget/results.json, per-trial records tracked"
+      },
+      {
+        "text": "A lecture's taxonomy becomes a gate that can fail us",
+        "target": "rubric as data + observed-evidence checker",
+        "score": 1.0,
+        "evidence": "docs/rubrics/AUDIT.md, 25/28 gate FAIL, exit 1"
+      },
+      {
+        "text": "My own errors are caught and published, not buried",
+        "target": "self-corrections shipped as commits",
+        "score": 1.0,
+        "evidence": "382759f parse bug, 5f910e0 count retraction, both with corrected numbers in the README"
+      }
+    ],
+    "request": "study the @skills paper + atskills repo: 1. apply it 2. extend to other agentic resource management with 4-window research 3. deep R&D on top 3 areas 4. long-form article. THEN: eval/research/10X the Agent Loop Engineering lecture, make it TRUE for /loop-anything, and write another article.",
+    "headline": "Two rubrics scored; both first answers were wrong",
+    "brief": "I measured the thing a new paper admits nobody has measured: how reliably an AI picks the right tool as you install more. Perfect at 40 tools, 83% at 80, 44% at your 176. Then I turned a lecture's checklist into an automatic test of your engine. It scored 23 out of 23, which was the bug \u2014 I wrote the questions knowing the answers. Adding the ones I expected to fail gave an honest 25 of 28.",
+    "growth": [
+      "C:met:Treated my own 23/23 as evidence of a bad rubric rather than a good engine, and went looking for questions I expected to fail",
+      "P:met:Derived the success-rate definition from first principles about what would corrupt it \u2014 excluded in-flight runs and counted safety blocks as failures, so the metric cannot rise when the safety gate fires"
+    ],
+    "why": "Both sources assert without verifying \u2014 the paper says its central number is unmeasured, the lecture says it is not a standard. Acting on either as fact would have been the exact false-green this repo exists to prevent. The 23/23 moment proves the risk is not hypothetical: I produced a perfect score on my own system within an hour of starting.",
+    "needle": {
+      "name": "verifiable claims about this engine backed by an executable probe",
+      "before": "0",
+      "after": "28",
+      "better": "up",
+      "source": "scripts/audit_loop_rubric.py"
+    },
+    "next_lever": "Human takeover of a running loop \u2014 the one red item with a reason but no plan; it is the only miss I cannot defend as a design divergence"
   }
 ]

docs/rubrics/AUDIT.md

@@ -0,0 +1,115 @@
+# Agent Loop Engineering — conformance audit
+
+Rubric: `docs/rubrics/agent-loop-engineering.yml` — 31 items drawn from *Agent Loop Engineering — 讲座总结* (DataApplab / AI聘 (info@aipin.io), received 2026-08-20).
+
+That source is a **lecture summary; taxonomy, no verification against a running system**. Every claim below is scored against shipped code or a test that was actually executed. No evidence means no.
+
+**Conformance: 25/28 (89%)** verifiable claims implemented · 3 declared gaps · 0 unmeasured · gate **FAIL**
+
+## Implemented — with observed evidence
+
+- **L1-loop-not-oneshot** · 2 — why an agent needs a loop
+  - claim: The system iterates plan→act→observe→evaluate rather than answering once.
+  - evidence: `src/loopeng/loop/controller.py:75`
+- **L2-goal** · 3 — Goal
+  - claim: The loop carries an explicit goal and a definition of done.
+  - evidence: `src/loopeng/config.py:89`
+- **L3-state** · 3 — State
+  - claim: Progress lives outside the model: steps taken, results, environment.
+  - evidence: `src/loopeng/memory/store.py:196`
+- **L4-policy** · 3 — Policy
+  - claim: Something decides the next action from the current state.
+  - evidence: `src/loopeng/adapters/base.py:103`
+- **L5-action-space** · 3 — Action space
+  - claim: The agent's available actions are declared, not open-ended.
+  - evidence: `src/loopeng/adapters/base.py:52`
+- **L6-observation** · 3 — Observation
+  - claim: Every action returns structured environment feedback.
+  - evidence: `src/loopeng/adapters/base.py:16`
+- **L7-evaluation** · 3/10 — Evaluation is the controller
+  - claim: A verifiable evaluator decides progress, not the maker's self-report.
+  - evidence: `src/loopeng/adapters/base.py:62`
+- **L8-maker-not-checker** · 10 — evaluation must be trustworthy
+  - claim: The thing that builds is not the thing that grades.
+  - evidence: `tests/test_maker_checker.py -> 30 passed in 0.16s`
+- **L9-reflection** · 4/6 — Plan-Execute-Observe-Reflect, Self-Reflection
+  - claim: Why the last attempt scored what it did is carried into the next attempt.
+  - evidence: `src/loopeng/adapters/base.py:66`
+- **L10-replan-on-plateau** · 4 — the plan is not immutable
+  - claim: Feedback can force a change of strategy, not just another attempt.
+  - evidence: `src/loopeng/config.py:100`
+- **L11-retry-transient-only** · 8 — Retry vs Recovery
+  - claim: Only retryable (infrastructure) failures are retried; not every error.
+  - evidence: `src/loopeng/adapters/base.py:142`
+- **L12-recovery-state** · 8 — Recovery keeps enough state to resume
+  - claim: A failed change can be rolled back rather than restarting from zero.
+  - evidence: `tests/test_checkpoint.py -> 2 passed in 0.50s`
+- **L13-exit-success** · 9 — explicit exits: success
+  - claim: The loop stops when the goal is verifiably met.
+  - evidence: `src/loopeng/loop/convergence.py:29`
+- **L14-exit-budget** · 9/14 — explicit exits: budget (iterations, tokens, wall clock)
+  - claim: The loop stops on a spent budget, and the budget has more than one dimension.
+  - evidence: `src/loopeng/loop/convergence.py:38`
+- **L15-exit-giveup** · 9 — explicit exits: give up and report
+  - claim: Repeated non-progress ends the run and reports failure instead of looping.
+  - evidence: `src/loopeng/loop/convergence.py:36`
+- **L16-safety-terminal** · 13 — permission control
+  - claim: A safety failure is terminal and unbypassable, whatever the score.
+  - evidence: `src/loopeng/loop/convergence.py:30`
+- **L17-permission-boundary** · 13 — which tools/data the agent may touch is limited
+  - claim: Execution is jailed and shell metacharacters are refused.
+  - evidence: `src/loopeng/adapters/safety.py:5`
+- **L18-human-in-the-loop** · 12 — Human-in-the-Loop for high-risk actions
+  - claim: High-risk completion requires a human, and the caller cannot self-approve.
+  - evidence: `src/loopeng/config.py:127`
+- **L19-hitl-unbypassable** · 12 — the gate must actually hold
+  - claim: An unattended run cannot pre-confirm its own result.
+  - evidence: `tests/test_run_contract.py::test_contract_can_never_disable_the_human_gate -> 4 passed in 0.04s`
+- **L20-multi-agent-graph** · 11 — Loop becomes Graph with many agents
+  - claim: Multiple agents are coordinated as a dependency graph, cycles refused.
+  - evidence: `src/loopeng/orchestration/coordinator.py:6`
+- **L21-observability** · 13 — every iteration leaves enough log/trace to debug
+  - claim: A run is reconstructable after the fact from recorded evidence.
+  - evidence: `src/loopeng/autonomous/report.py:37`
+- **L22-token-accounting** · 14 — Token economics: measure tokens, time, iterations, success
+  - claim: Token cost, wall time and iteration count are recorded per run.
+  - evidence: `src/loopeng/proof.py:108`
+- **L23-cost-never-faked** · 14 — measurement must be real to be useful
+  - claim: An unavailable cost is omitted, never estimated into the record.
+  - evidence: `tests/test_proof.py -> 7 passed in 0.09s`
+- **L29-success-rate-metric** · 14 — measure tokens, time, iteration count AND task success rate
+  - claim: Task success RATE is computed across runs, not just per-run outcome.
+  - evidence: `tests/test_success_rate.py -> 6 passed in 0.09s`
+- **L31-error-memory** · 13 — record errors so the agent does not repeat the same mistake
+  - claim: Failures that recur across runs are surfaced to the next attempt.
+  - evidence: `src/loopeng/memory/store.py:277`
+
+## NOT implemented
+
+- **L27-react-interleaving** · 5 — ReAct: reasoning and acting alternate at fine granularity
+  - claim: The loop interleaves reason->act->observe per tool call, rather than one coarse generate/judge/refactor turn.
+  - evidence: `pattern /class ReActLoop|def reason_then_act/ not found in 76 file(s)`
+  - why it is still a miss: Deliberate divergence, not an oversight: ReAct puts the actor in charge of judging its own next step, and this engine's load-bearing rule is that the maker never grades. Recorded as a MISS rather than reclassified as a pass, because a rubric that lets the author explain failures away scores nothing.
+- **L28-human-takeover** · 12 — a human may take the task over directly
+  - claim: A human can seize control of a running loop, not merely approve or reject its result.
+  - evidence: `pattern /def takeover|def handoff_to_human/ not found in 76 file(s)`
+  - why it is still a miss: A real gap. The gate is end-of-run approval; there is no control channel into a loop already in flight. Worth building; not built.
+- **L30-manager-worker** · 11 — Manager-Worker and hierarchical multi-agent structures
+  - claim: A manager agent decomposes a goal and assigns sub-tasks to worker agents.
+  - evidence: `pattern /class ManagerAgent|def assign_subtask/ not found in 76 file(s)`
+  - why it is still a miss: Divergence: coordination is an explicit dependency DAG with cycle rejection, which is inspectable before anything runs, rather than a manager agent deciding at runtime. Still a miss against the lecture's claim.
+
+## Declared gaps — the engine does NOT do these, and says so
+
+Each probe passes while the gap is real and fails the moment it silently closes, so this list cannot quietly go stale.
+
+- **L24-typed-failure-taxonomy** · 8 — classify the error before choosing a response
+  - claim: Failures are classified into a named taxonomy (referee-unavailable, adapter-contract, human-gate-timeout).
+  - status: **confirmed absent** — Only infra-vs-clean is distinguished. External eval item 3.
+- **L25-branching-search** · 7 — Tree of Thoughts
+  - claim: The loop explores multiple candidate paths and prunes them.
+  - status: **confirmed absent** — Single-path refine with a dimension pivot. Deliberate: branching multiplies cost.
+- **L26-trace-schema** · 13 — observability as a first-class trace
+  - claim: Runs emit a structured trace (spans/trace ids), not just a report.
+  - status: **confirmed absent** — Reports and proof packs exist; a span-level trace schema does not.
+