Skip to content

benchmark: aggregate EdgeBench fix round for the next 24h run - #6624

Open
songoow wants to merge 15 commits into
mainfrom
edgebenchtest
Open

songoow wants to merge 15 commits into
mainfrom
edgebenchtest

Conversation

@songoow

@songoow songoow commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Purpose

Prepare one branch for the next EdgeBench experiment. It combines refused-checkpoint handoff resilience, bounded full capture preparation, initial-delivery guidance and evidence-based research followthrough.

A refused handoff keeps the original wake running and records a bounded diagnostic. Full capture preparation shares a 90-second window bounded by the original trial phase. Planning compares a feasible candidate with the strongest available starting point and separates informative evaluation from replacing the best artifact.

Research followthrough

Stride reviews return the prior decision, its declared Todo targets, canonical work state and later observations. Outcome citations are replayed against complete scoped history at writeback: display caps and an omitted optional typed-progress attachment cannot invalidate real work evidence. Old or unrelated evidence cannot establish execution of the declared work. Negative and unknown outcomes remain legal.

Legacy descriptive Todo deltas remain readable with unresolved linkage; overlong references cannot alias another Todo. Structured start packets reuse the shared planning guidance, while standalone messages retain the full instruction. No new benchmark acceptance switch or score threshold is introduced; stride remains opt-in and historical decision identities are preserved.

Validation

Focused validation for the new changes:

  • 40 TypeScript replan/stride tests passed, including capped history, ordinary work evidence and legacy declarations.
  • 2 isolated real File/SQLite CLI continuation tests passed, including negative-result review and rejected old-evidence recovery.
  • 2 planning-packet/standalone-message checks and the focused output-guidance recognition check passed.
  • Control-plane TypeScript typecheck, focused Python lint and diff checks passed.
  • Structured guidance copies reduced from four/six to one/two in compact/detailed guided packets. Standalone messages remain self-contained; the bounded first-introduction allowance documents their retained content.

Full CI was not awaited. No live model inference, benchmark launch or score-improvement claim is part of this change. Real model adoption, sustained operation and research benefit remain to be measured. Packaged frontend adoption of the added review detail is not qualified here.

This remains an experiment-preparation branch; it has not been merged into main. The original checkpoint and first-delivery components remain traceable to #6618 and #6621.

Branch repairs after the first launch attempt (fb6f060, a0fa279)

A first paired launch on head f7756ff lost every LoopX arm within two minutes on both hosts: RuntimeError("Native task prompt is unavailable") in SForgeWorker.format_run_cmd, before the first turn. The provider (decf67c) and the workspace-instruction install were unchanged; the discriminator against the earlier run that completed 11 rounds was the launcher's umask 077. The worker re-uploads the feedback prompt from a host temp file, the native copy preserves that mode and lands the file root-owned, so a 0600 prompt was unreadable by the container's agent user. a0fa27981 sets the prompt mode explicitly before upload and adds a regression test that runs format_run_cmd under umask 077 (fails on the previous head with mode 0600, passes now). The checkpoint declaration upload was already chowned and re-moded after upload and is unaffected.

fb6f060e7 repairs the branch's own kernel-static-checks failure. The base/head differential copies cli_output_semantics.py out of the package and measures a base checkout that predates the planner guidance constant: the relative import had no parent, and an absolute import fails on that base. A missing owner is now observed as "guidance absent". The bootstrap JSON pack carries its rendered message, so the expanded guidance lands twice there (planner field plus the labelled block message-only consumers need); measured one-time growth is 2,821 chars, and that surface's allowance is raised to 3,000 with the measured value recorded in the parity fixture. Other surfaces keep their limits.

Validation on a0fa279 (source checkout, uv run --extra test): cli-output-budget-regression-smoke ok; 176 cli_output control-plane tests; 49 EdgeBench prompt tests including the new umask case; 322 SForge runtime, checkpoint-handoff and warm-start tests; 40 TypeScript replan/stride tests. Full CI on this head was not awaited at the time of writing.

Planning-wake retry (b7c47c9)

The second paired launch on f7756ff got past the prompt but every LoopX arm still ended after one wake: the model stream disconnected during the planning wake (eight failed reconnects, turn.failed), the entry raised "Planning failed; execution was not started", and the trial closed with no submission. Explore activation, stride and the checkpoint declaration had all succeeded. prepare_entry now retries a planning wake whose native process exited non-zero without a plan or a harness error, up to three attempts inside the same phase budget, and records every attempt in planning-attempts.json beside the planning receipt. Timeouts, harness-side failures, a zero exit without a plan and an exhausted budget still fail once; a validated or blocked plan is never re-run. This is a default harness behavior change for the loopx-planned entry and is covered by three new entry tests (58 task-entry tests, 323 runtime/handoff/prompt tests pass locally).

loopx-agent and others added 7 commits October 16, 2026 15:08
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: song <liusongstep@gmail.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
…urce wake

A fresh_replan handoff whose packet is refused (unparseable, outside this
wake's capture root, or rejected by the typed selector) raised out of the
wake loop and ended the source session; the next resume then reused the
ended wake's socket and capture root. The refusal is now recorded as a
bounded, unreleased handoff event and the uninterrupted original wake keeps
running. A refusal raised after the decision holder was entered, or after
the source exited, still re-raises.

Wake receipts also keep a bounded error message beside error_kind, so a
failed wake is diagnosable without an out-of-band incident file.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: song <liusongstep@gmail.com>
Brings the shared planning first-delivery guidance and the stride review
questions that distinguish initial deliverable quality from later gain.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: song <liusongstep@gmail.com>
Gives both full controller capture hooks one shared 90-second window,
capped by the original trial phase, so a mature research boundary is not
missed inside the old 30-second window.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: song <liusongstep@gmail.com>
songoow and others added 8 commits October 16, 2026 17:02
The shared first-delivery guidance adds fixed decision text to four
agent-facing CLI surfaces. The frozen 96-row differential fixture measures
+2,340 JSON characters, +3,496 for explicit command-pack detail and +578
Markdown; the guidance is one bounded block with no per-unit growth and no
line change. Declare that one-time allowance for the four surfaces and teach
the authoring-input observation to recognize the exact guidance, so the
budget keeps characterizing cost instead of hiding the clause.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: song <liusongstep@gmail.com>
Return canonical planned work with the previous decision and replay cited outcome evidence against complete scoped history at writeback. Display limits and optional typed-progress attachments no longer exclude valid work evidence; descriptive legacy Todo deltas remain readable. Preserve unknown and negative outcomes, historical decision identities and off-mode isolation.

Keep initial-delivery guidance in the shared planner for structured packets and expand it for standalone messages. Qualify the bounded standalone growth while removing repeated full guidance. Cover late/ordinary evidence, unavailable sources, legacy deltas and the real File/SQLite continuation path.

Signed-off-by: song <liusongstep@gmail.com>
Document complete-source citation replay, readable legacy Todo declarations, unknown/negative outcomes and compact versus standalone guidance delivery. Record the bounded output tradeoff without claiming model adoption or score improvement.

Signed-off-by: song <liusongstep@gmail.com>
Reuse the canonical frozen-request digest matcher and retain the original source commit and focused validation cases. This integrates only PR #6623 into edgebenchtest; main remains unchanged.

Signed-off-by: song <liusongstep@gmail.com>
Signed-off-by: song <liusongstep@gmail.com>
…ckout

The base/head differential copies cli_output_semantics.py out of the
package and measures a base checkout that predates the planner guidance
constant. A relative import there has no parent, and an absolute import
fails on that base; both broke kernel-static-checks on this branch. Treat
a missing owner as "guidance absent" and never let a None match.

Bootstrap JSON packs carry their rendered message, so the expanded
guidance lands twice (planner field plus the labelled block message-only
consumers need). Measured growth is 2,821 chars; raise that one-time
allowance to 3,000 and record the measured value in the parity fixture.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: song <liusongstep@gmail.com>
…uncher umask

The worker re-uploads the feedback prompt from a host temp file and the
native copy preserves that file's mode while landing it root-owned. A
launcher running under umask 077 therefore delivered a 0600 prompt that
the container's agent user could not read, and every heartbeat arm failed
at format_run_cmd with "Native task prompt is unavailable" before its
first turn. Set the prompt mode explicitly before upload and cover the
umask case.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: song <liusongstep@gmail.com>
One planning wake whose native process exited non-zero without a plan or
a harness error stranded whole paired launches: the model stream
disconnected during planning, the entry raised "Planning failed" and the
trial ended before execution with no submission. Retry such a wake up to
three times inside the same phase budget and record every attempt beside
the planning receipt. Timeouts, harness failures and exhausted budgets
still fail once, and a validated or blocked plan is never re-run.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: song <liusongstep@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants