Skip to content

feat: simplify harness authoring and preview native output - #215

Merged
goldmedal merged 6 commits into
mainfrom
codex/harness-authoring-contract
Oct 7, 2026
Merged

goldmedal merged 6 commits into
mainfrom
codex/harness-authoring-contract

Conversation

@goldmedal

@goldmedal goldmedal commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator

Harness authors can now start with one profile containing an inline prompt, without creating a placeholder semantic context or managing an intermediate IR file. warble check, preview, and build carry that profile through validation, inspection, and native-file output for plain Claude Code headless and interactive targets. The native CLI is started separately by the user.

Changes

  • Make context optional for behaviors with no context requirements, while rejecting missing required context and ambiguous shorthand. Extracting an inline behavior into a reusable component preserves its meaning.
  • Preview the actual emitted instruction files, generated permissions, and authored file/field origins, including brief replacement and slot selection. Runtime-added instructions and conversation state are explicitly outside the preview.
  • Refuse unsupported or ineffective authored configuration, protect existing output directories, and clean up failed new output when directory ownership can be verified. Context-free IR is rejected by bound-session targets, including pi.
  • Keep profile paths in compile errors, use neutral context-free agent descriptions, and explain unsupported context-free targets and runtime-injected parameters before output.
  • Update the authoring guides and add a self-contained code-review example with an explicit input inventory, review scope, and synthetic diff.
  • Document the boundary between prompt guidance, target enforcement, and host responsibilities. General workflow DSLs and new stateful runners remain deferred; existing integration paths are retained.

Compatibility

The IR moves from 0.8 to 0.9 to represent an absent context binding. Compiler, readers, package declarations, and fixtures are updated together. Regenerate stored IR and use matching readers; 0.8 artifacts are not accepted by 0.9 readers. The new author commands support only plain Claude Code file targets; other targets and explicit host contracts continue through compile/dispatch. Prompt instructions are not substitutes for tool permissions or execution isolation.

Profile component mounts and profile context declarations now reject unknown fields that were previously ignored. Remove stray fields; unsupported non-null mount config is also rejected. Custom tier names remain supported through dispatch-time model mappings.

Validation

Local verification covers Rust lint/test/build/release and fresh rustdoc (877 passing, 2 existing ignored), SDK (375 passing), Codex (259 passing), BIRD (661 passing, 8 expected external-fixture skips), documentation generation/drift/release checks and production build, publish checks, fixture/workflow checks, and the release plan. Author-flow tests compare preview with emitted files and cover missing context, diagnostics, output collisions, and cleanup failures. A regression demonstrates that pi rejects context-free IR before writing output; all 11 pi dispatch tests pass. GitHub checks on 59750a5 completed with 12 passing checks and 4 expected release-only skips; no failures or pending checks.

Two bounded real-model authoring exercises were completed separately from the deterministic gates. They support the usability observations, not a general review-accuracy or runtime-safety certification. No release or merge is part of this PR.

Support single-file harness profiles with optional context and IR 0.9 across readers. Add check, preview and build commands with native prompt provenance, strict diagnostics and guarded failed-output cleanup. Update authoring contracts, tutorials, fixtures and regression coverage.
Separate prompt guidance from enforceable target and host requirements. Defer new workflow DSL and runner state, retain existing runtime contracts, and document evidence required to reopen scope. Align roadmap, composition and direct-session support descriptions.

@goldmedal goldmedal left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review

Verdict: approve with two small fixes requested. CI is green on head 15ebf9d (rust ×2, ts ×2, bird-eval ×2, doc, docs-site, publish-check, fixture-contract, workflows). I rebuilt the CLI at this head, ran the two new author-flow test files (16 passed), and probed the new paths by hand; notes below.

P2 — fix before merge

  1. compile parse errors lost their file-path prefix. project_needs_hub (cli/src/lib.rs) runs before compilation in run_compile and compile_project_to_ir and returns the bare serde string. Repro: components: 3 now prints error: profile components must be a list; a YAML syntax error prints error: found unexpected end of stream at line 6 column 1… with no file named. Before this PR the same path produced failed to parse <profile.yml>: …. check/preview are unaffected because they validate first. Suggest wrapping the error in project_needs_hub (or at its two call sites) with the profile path.

  2. Context-free agent files carry five blank lines between the frontmatter and ## respond. In dispatcher/claude-code-cli/src/emit/agent.rs the empty preamble and empty prompt_section() are still pushed together with their separators. Cosmetic, but preview advertises exact bytes and this is the first output the new tutorial shows. Suggest skipping empty parts when joining.

P3 — nits / follow-ups

  1. vercel rejects context-free IR only via the raw serde error (invalid type: null, expected struct ContextBinding at line 5 column 29), and context_free_unsupported_shapes_and_targets_refuse_before_output pins that string. Every other target emits a purposeful "context-free IR is not supported by …" message. An explicit check in dispatcher/vercel/src/ir.rs plus a test update would make the targets consistent.
  2. Shorthand without description synthesizes analytical skill that renders no render blocks (outcome: none). as the subagent description. Claude Code uses that field for delegation, and it is misleading for a text helper. Consider requiring description in the shorthand, or deriving it from the prompt's first line.
  3. Shorthand tier is free text. tier: medium compiles and only fails at dispatch. parse_component could validate against strong|cheap|orchestrator and give a source-located error.
  4. The context-free check treats every params[].source as a context requirement, but authoring.md defines source: runtime-injected as runtime-supplied, not context. The resulting message ("requires context for parameter … source 'runtime-injected'") would be wrong. Masked today because the emitter also rejects such params, so this is wording only.
  5. Check-order nit: a context-free profile mounting generate_dashboard reports the missing answer_query composition mount before the context requirement. Accurate, just not the most useful first error.

Compatibility notes (confirm intent; not defects)

  • ProfileComponentMount and ProfileContext are now deny_unknown_fields. External profiles with stray mount or context keys that were silently ignored now fail compile (verified with foo: bar on a mount). authoring.md documents this; the PR body only mentions config. Worth one line in the PR body since the squash commit feeds the changelog.
  • Downstream readers that hard-code warble_ir_version: "0.8" or pin @warble/ir-spec 0.8.x (e.g. the WrenAI genbi app) will need a coordinated bump when they move to this release. Nothing breaks until they do.
  • packages/ir-spec is excluded from release-please, so the manual 0.9.0 bump matches how 0.7 → 0.8 was done.

Verification performed

  • cargo build --bin warble; cargo test -p warble-cli --test author_commands --test authoring_tutorial → 16 passed.
  • Probes with the built binary: compile first-harness (null root and component bindings, IR 0.9); plain dispatch to claude-code:headless, codex:interactive and vercel; manifest; check/preview/build on first-harness, code-review and the bound demo-agent; a mixed bound + inline-shorthand profile; build refusals on an existing file, an existing directory and a nested new path; empty profile, duplicate ids, use + prompt, {{project}} without context, hub mount without context, unknown mount/context field, non-null mount config.
  • Not exercised: produce-session on context-free IR, the interactive target's launch pins, live model runs.

@goldmedal
goldmedal marked this pull request as ready for review October 7, 2026 07:27
@goldmedal
goldmedal merged commit dbaf92c into main Oct 7, 2026
16 checks passed
@goldmedal
goldmedal deleted the codex/harness-authoring-contract branch October 7, 2026 07:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant