2.4.3: clarify shape for vague prompt requests + R8 echo gate - #38
Conversation
Reported: `prompt:` on "develop a app that allows to perfrom a full schedule for
my meeting a proper PM would" returned that sentence as the task, again as its
only real constraint, and again as the output format, padded with two scope
guards, typos intact.
Root cause: default prompt v2 is fully deterministic - ground_prompt_v2_output
discards the model draft and renders from the source clauses. A request with four
splittable clauses (the CSV case) decomposes well; one unsplittable clause has
nothing to decompose, so the renderer emitted it everywhere and padded to reach
the 3-constraint minimum. The existing echo detector cannot catch this:
is_weak_prompt_echo returns False once output has target structure (V32), and the
v2 finalizer always emits structure.
- Clarify shape. is_underspecified() (fewer than 2 real source clauses) switches
to a render that names the open questions - platform, scope, data source,
success criteria - and tells the agent to settle them before building. Adds no
requirements: every line asserts only that something was not stated. V33
anti-invention still holds (verified: no agenda/calendar/database/framework).
- Language normalization. Surfaced text gets article agreement and a fixed,
audited typo map, so "a app ... perfrom" no longer repeats in every section.
Exceptions preserved ("a user", "a unique", "a one-off", "a utility").
- R8 rubric item. Fails any output whose constraints hold nothing beyond a
restatement of the task plus boilerplate, or whose output_format re-quotes the
request. Disqualifying on its own, so a valid-looking echo cannot pass at any
score. Machine-checkable; no judge needed. Rubric is now 8 items, pass >= 7.
- The reported request is a permanent fixed-set case.
Two self-caught metric errors on the way (SPEC B39/B40): coverage-of-candidate
flagged good CSV decomposition, and per-item coverage flagged long requests whose
task derives from sentence 1. Final metric is coverage-of-task, judged set-wise on
constraints. Validated across all 14 fixed cases: ZERO false verdicts.
Live on qwen3.5:4b: the reported input now yields the clarify shape (5.87s), the
CSV case is byte-identical to before (8.89s, 4 real constraints).
Updated test_v33_sparse_request to the new shape while keeping the anti-invention
assertions it existed to protect.
ruff clean, 485 pytest (+26). SPEC V44/V45/V46, T30, B38-B40.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cfea93b5c2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "passed": bool(score is not None and score >= 6 and not hard_fail), | ||
| # R8 is disqualifying on its own: an output that merely restates the | ||
| # request is not a converted prompt no matter what else it scores. | ||
| "passed": bool(score is not None and score >= 7 and not hard_fail and r8), |
There was a problem hiding this comment.
When a candidate fails only R8, this line marks that sample as failed, but _style_summary() merely records it in passed_outputs and pass_rate; evaluate_gate() never reads either and instead checks the median score, invention count, and R1 rate. In a 14-case run, one 7/8 echo normally leaves the median unchanged, so the top-level release gate can still pass despite R8 being described as disqualifying. Add a v2 per-output/pass-rate condition to the aggregate gate.
Useful? React with 👍 / 👎.
|
|
||
| Such a request has no structure to surface, so the ordinary render would | ||
| repeat it as task, constraint, and output format (see _MIN_SOURCE_CLAUSES).""" | ||
| return len(source_clauses(cleaned)) < _MIN_SOURCE_CLAUSES |
There was a problem hiding this comment.
Any request that produces one parser clause is treated as underspecified, even when it is complete—for example, write a command-line tool in Rust that prints SHA-256 hashes for every file under the supplied directory. The clarify render then falsely says the language, scope, data source, and success criteria were not stated and requires approval before implementation. Clause count measures syntactic decomposability rather than which details are missing, so common one-action prompts are contradicted; detect absent dimensions instead of using this blanket threshold.
Useful? React with 👍 / 👎.
| to perfrom ..." would otherwise appear verbatim in three sections. This only | ||
| corrects spelling/agreement; it never changes meaning or adds requirements.""" | ||
| cleaned = str(text or "") | ||
| cleaned = _TYPO_RE.sub(lambda m: _match_case(m.group(0), _TYPO_FIXES[m.group(0).lower()]), cleaned) |
There was a problem hiding this comment.
When the source contains a legitimate identifier or filename matching the typo map—for example, rename the variable teh to result or read shedule.json—this unconditional substitution changes the identifier before rendering. The resulting task refers to the or schedule.json, potentially directing the coding agent to edit the wrong symbol or file. Avoid normalizing protected/code-like tokens, including text in code spans, rather than applying typo fixes to every word.
Useful? React with 👍 / 👎.
| if len(task_words) < 3: | ||
| return "" |
There was a problem hiding this comment.
For a short task such as <task>Fix bug.</task>, this early return declares that no section repeats the task without examining the output format or constraints. An output that repeats Fix bug. in <output_format> and supplies only the same restatement plus boilerplate constraints therefore receives R8=true and can score 8/8. Use exact or short-task-specific comparison instead of exempting the short vague prompts this rubric is intended to catch.
Useful? React with 👍 / 👎.
2 participants
2.4.3 —
prompt:handles vague requests, and the gate can no longer green-light an echoThe report
prompt:ondevelop a app that allows to perfrom a full schedule for my meeting a proper PM wouldreturned:The request, three times, typos intact — plus two generic scope lines.
Root cause
Default prompt v2 is fully deterministic:
ground_prompt_v2_outputdiscards the model draft and renders from source clauses. A request with four splittable clauses (the CSV case) decomposes well; one unsplittable clause has nothing to decompose, so the renderer emitted it in every section and padded to reach the 3-constraint minimum.The existing echo detector could never catch it:
is_weak_prompt_echoreturnsFalseas soon as output carries target structure (V32), and the v2 finalizer always emits structure. Measured:word_overlap(src→out) = 1.00,is_weak_prompt_echo = False.And the release gate passed it — 5/7 machine-checkable, no hard fail, 7/7 under a lenient judge. The rubric measured structure and non-invention but never asked whether anything was added.
Fixed
is_underspecified()(fewer than 2 real source clauses) switches to a render naming the open questions — platform, scope, data source, success criteria — and instructing the agent to settle them first. It adds no requirements: every line asserts only that something was not stated. V33 anti-invention holds (asserted: no agenda / calendar / database / framework leaks in).a app … perfromdoesn't repeat in every section. Known exceptions preserved (a user,a unique,a one-off,a utility).output_formatre-quotes the request. Disqualifying on its own, so a structurally valid echo cannot pass at any score. Machine-checkable — no judge. Rubric is now 8 items, pass ≥ 7.vague_unparseable_scheduler).Two self-caught metric errors
Recorded as SPEC B39/B40 rather than quietly fixed:
Final metric: coverage of the task, judged set-wise on constraints — fail only when nothing survives after discarding restatements and boilerplate. Validated across all 14 fixed cases: zero false verdicts.
Verification
validate()passestest_v33_sparse_requestwas updated to the new shape while keeping the anti-invention assertions it existed to protect.ruff clean · 485 pytest (+26) · SPEC V44/V45/V46, T30, B38–B40.
🤖 Generated with Claude Code