Skip to content

fix(planning): preserve initial quality in shared delivery guidance - #6621

Open
songoow wants to merge 2 commits into
mainfrom
codex/initial-quality-research-continuation-20261016
Open

songoow wants to merge 2 commits into
mainfrom
codex/initial-quality-research-continuation-20261016

Conversation

@songoow

@songoow songoow commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Problem and delivered behavior

An agent can optimize for an earlier first signal by presenting an unnecessarily weak artifact, then describe recovery from that low start as useful long-horizon gain. Shared product planning and enabled stride review should distinguish initial deliverable quality, latency/cost and improvement beyond the retained best.

The common Goal planner now carries first_delivery_guidance, also rendered into guided start prompts and consumed by the existing todo plan packet. It asks for a strong feasible first deliverable with coherent implementation/validation, preserves useful diagnostic probes and original acceptance/feedback/authority limits, and rejects indefinite preparation as the objective.

The existing TypeScript stride review questions distinguish initial quality from later gain, anchor comparisons to the retained best artifact, ask whether subsequent work implemented the proposed change, and bind observations to artifact versions, inputs, budgets and feedback availability. Snapshot existence is explicitly separated from restored execution.

Ownership and behavior boundaries

  • Existing owners: goals/start_contract.py owns the shared planning text; work_items/stride_policy.ts owns the existing optional decision-value questions. This adds no Python state-decision owner, strategy registry or second evaluator.
  • Specification basis: the Planning Contract in docs/reference/protocols/loopx-goal-command-v0.md and the iterative strategy boundary in the stride RFC, at base f6cf9583794eb8c4cca41fa832a1c2f150059583.
  • Intentional default guidance change: ordinary and fine-grained Goal/task planning receive the initial-quality guidance. The new string is an additive field of the existing planner projection.
  • Stride remains default-off. When enabled, the same three questions gain evidence distinctions; no decision schema, valid action, settlement/admission gate or numerical pivot rule changes.
  • This is Agent-consumed product behavior, not a machine guarantee that an artifact meets a score. Model adoption, initial-quality noninferiority and later breakthroughs require separately admitted experiments.
  • No benchmark-specific prompt, restored feedback-first flag, additional submission allowance, deadline extension, guard relaxation or model run is included.
  • The bounded companion refactor shares one initial-quality string across planning entrypoints. Public docs and the bilingual stride checkpoint disclose the distinction.

Original guidance validation

Source environment: isolated worktree and temporary Effect runtime, Python 3.12.13 / Node 22.23.1.

  • Before implementation, the new public-planning assertion failed on the absent guidance and the stride assertion failed on the missing initial-versus-later distinction.
  • Python: 114 passed, covering task planning, guided-start projections, fine-grained turns and stride observation.
  • TypeScript: 20 passed, covering stride proposals, off isolation, unchanged legal actions, stale/foreign evidence rejection and source-replayed Explore pagination.
  • npm run typecheck:control-plane, Ruff, changed-diff semantic advisory, full semantic vocabulary smoke and git diff --check: passed.
  • Selected public-boundary scan: 7 files clean. The disposable checkout has no registered research Goal; registry-boundary qualification is not claimed.
  • Real CLI packet/readback is exercised without starting a host loop or spending quota. These checks do not validate a real model's subsequent choices.
  • No new frontend interaction is introduced: existing guided-start/CLI consumers receive the shared contract. Browser interaction, hosted CI and real-model runs are not claimed as passed.

The larger experimental goal remains open: first submission quality versus native under a fixed time/cost protocol, then additional verified gain while a concurrently running native baseline stalls. No merge or benchmark launch is performed by this PR.

Output qualification repair

The original head failed the native base/head CLI differential. The unchanged 96-row public fixture measured +2340 characters for default JSON, +3496 for explicit command-pack detail and +578 for bootstrap Markdown, with no extra lines. The planner, guided transaction and prompt/message entrypoints retain their existing consumers. No instruction or compatibility field was removed for size.

The existing test-only authoring observation now recognizes the complete frozen guidance. Only its absent-to-complete transition on the affected start/bootstrap carriers receives bounded 2400/3600/600 character and byte allowances. Already-present, missing, incomplete, other-surface and over-limit cases still fail; absolute, line, semantic and scale gates stay unchanged. This extends the private probe observation, not a runtime vocabulary, permission or quality classifier.

Repair validation: Python3.12.13 / Node22.22.3; 145 differential/planning tests pass, including clause deletion and one-time/surface/boundary counterexamples. The actual CLI differential passes all 96 unchanged base/head rows against f6cf958, with 11 explicit review signals. Ruff, advisory then full semantic-vocabulary and whitespace checks pass. Original failures remain failures under their old limits. Original guidance validation above is retained evidence, not relabeled as a rerun. Hosted CI and independent review must evaluate this new head.

Related refactor: reuse the existing authoring-input qualification owner; no new framework or parallel product decision. The complete decision text and its diagnostic, cost, acceptance and authority clauses are unchanged by this repair. No frontend behavior, real model, benchmark launch, scoring criterion or merge authority changes. Aggregate experiment consumers still need deliberate adoption of the new source; this push is not their qualification.

Signed-off-by: song <liusongstep@gmail.com>
@songoow

songoow commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator Author

Folded into edgebenchtest (commit c1eb30315) as the experiment-preparation aggregate branch, so the next EdgeBench 24h run can pin one commit. Content is unchanged: da01f996c is an ancestor of that head. Leaving this PR open for its own review; no merge authority is claimed here.

Signed-off-by: song <liusongstep@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant