Repository navigation
Fix 34 audited findings, and add the tests that would have caught them - #5
Merged
Merged
Conversation
… caught them
Six auditors read the codebase along different lines, every finding then had to survive an adversarial pass whose job was to refute it, and several were reproduced by running the code rather than reasoning about it. 34 survived. This fixes them.
The worst one collapsed a page that had already rendered. `asSettled` in lib/content-parallel.ts is named for settling and raced raw promises instead, so when one of four copy chunks failed — and the README measures malformed JSON at roughly one run in two — the rejection escaped compose.ts, reached the route as a terminal error, and the client hid the exports and the edit box on a page whose nav, hero, testimonials, pricing, FAQ and footer had all arrived and painted. The fix is in the helper that owns the contract: it now yields `{status}` like `Promise.allSettled`, and it is a plain function returning a generator rather than an `async function*`, because a generator body does not run until its first `next()` and a rejection landing in that gap is unhandled. The second round's promise gets its handler where it is created, not at the await site — attaching it at the await produces an unhandled rejection in the window between, which is demonstrable.
The retry underneath it could not fix the failure it exists for. `finish_reason` was never read and attempt two re-sent the identical `max_tokens`, so a truncated reply truncated again at the same place, burning the budget twice. Truncation is now detected and the second attempt gets a larger cap and a complaint the model can act on; other malformed-JSON causes keep the original budget.
Two whole features were dead. `add` arrived well-formed from both edit layers and nothing consumed it, so the log printed `add@1.00 faq` and the page did not change — and that made `remove` one-way. The token readout summed a field that was never written, so `· N tok · N evals` was structurally unreachable.
The stream consumer had no error path at all: no `try`, no `finally`, no `response.ok`. A rejected fetch left Generate disabled until a reload, and a non-200 was fed to the NDJSON parser as if it were events.
On the MCP server, a tool argument of `toString` or `constructor` reached a plain object lookup and threw out of the handler, killing the process for the agent on the other end — reproduced over real stdio, where ids 2 and 3 went unanswered. Every client-keyed lookup now goes through `Object.hasOwn`, the module comment states that as the rule, and batch requests are handled rather than advertised and refused.
Nothing capped a prompt, at any layer. 2000 characters now, at both routes, chosen against the longest example any locale ships (91). Both routes validate what they destructure instead of trusting it.
Smaller, all real: `:first-child` was being rewritten to `:first` by an alternation matching a prefix, so those rules were dropped from every HTML export; the JSX exporter trimmed each text node independently and ate the space before a byline; the HTML title deleted `&`, `<` and `>` rather than escaping them; AGENTS.md told the receiving agent the theme tokens sit on the outermost element when they sit on its first child; the theme swatches carried their selected state in colour alone.
Four files had no tests: lib/compose.ts, lib/edit.ts, lib/export.ts, and the erasable-syntax constraint that lets scripts/mcp.mjs import lib/*.ts under Node's type stripping — invisible to the suite until `npm run mcp` breaks. All four now have them, and CI drives the MCP server over a real pipe, which is the only thing that exercises the transport rather than the handler.
The contrast test constrained six token pairs while the registry paints text on three more. Extended; no theme fails. The README claimed in all eight languages that app/registry.tsx contains no hex at all — it contains three, the macOS traffic lights in the mocked screenshot. The claim was what was wrong, and it now names its exception.
Two findings turned out to be arguments rather than bugs and were deliberately not "fixed". Demo mode was not made to answer 「不要定价了」 by adding a pricing block, because the fixture has no pricing and the live system would return `unclear`; demo mode exists to replay what really happens, so it now says so through `blockedBy` instead. And the removal guard rejects the request rather than removing a block the archetype requires.
One bug was found while wiring the parts together rather than by the audit: `applyEdit` sent `variants` without declaring it, so after any restyle the next edit described the page as it looked two edits ago and Jev answered about a page that no longer existed. `run` had the same shape with `locale`, harmless only because the dictionary changes identity alongside it. lib/page.test.ts already guarded this class for the dictionary; it now guards it for everything a callback puts in a request body, which is how both were caught.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Six auditors read the codebase along different lines. Every finding then had to survive an adversarial pass whose only job was to refute it, and several were reproduced by running the code rather than reasoning about it. 34 survived. This fixes them, in one PR as requested.
The one that mattered most
A single failed copy chunk threw away a page that had already rendered.
asSettledinlib/content-parallel.tsis named for settling and raced raw promises instead. The README measures malformed LLM JSON at roughly one run in two, so one of four chunks failing is an ordinary outcome — and when it did, the rejection escapedcompose.ts, reached the route as a terminal error, and the client hid the exports and the edit box on a page whose nav, hero, testimonials, pricing, FAQ and footer had all arrived and painted.The verifier reproduced it end to end with a stubbed transport:
Fixed in the helper that owns the contract rather than at the call site:
asSettlednow yields{status}likePromise.allSettled. It is also a plain function returning a generator rather than anasync function*, because a generator body does not run until its firstnext()and a rejection landing in that gap is unhandled. The second Jev round's promise gets its.catchwhere it is created — attaching it at the await site produces an unhandled rejection in the window between, which is demonstrable and was demonstrated.The retry could not fix the failure it exists for
finish_reasonwas never read, and attempt two re-sent the identicalmax_tokens. A truncated reply truncated again at the same place, burning the budget twice. The shipped fixtures show theplacechunk already at 1067–1098 of 1300 tokens on the happy path — under 18% headroom.Truncation is now detected; attempt two gets a larger cap and a complaint the model can act on. Other malformed-JSON causes keep the original budget, so a fenced reply does not buy tokens it will not use.
Two features were dead
addwas never applied. It arrived well-formed from both edit layers and nothing consumed it, so the log printedadd@1.00 faq · 3ms— which reads as success — and the page did not change. That also maderemoveone-way.steps.reduce()summed a fieldsetStepsnever wrote, so· N tok · N evalscould only ever render zeros.The stream consumer had no error path
No
try, nofinally, noresponse.ok. A rejected fetch left Generate disabled until a page reload; a non-200 was fed to the NDJSON parser as if it were events. All three closed, plus atryaround the per-lineJSON.parse.MCP: a tool argument could kill the server
toStringorconstructorreached a plain object lookup and threw out of the handler. Reproduced over real stdio — id 1 answered, then:ids 2 and 3 never came back. Every client-keyed lookup now goes through
Object.hasOwn, the whole file was scanned for siblings of the pattern, and the module comment states the rule so the next table added is not a new hole. Batch requests are now handled rather than advertised and refused, and an id the server can plainly see is echoed instead of discarded.Verified after the fix:
Nothing capped a prompt, at any layer
A 1 MB prompt became a ~17 MB upstream body. Now 2000 characters at both routes — chosen against the longest example any locale ships (91, in
pt.json) — returned as a 400, never a truncation, withmaxLengthon both inputs so the field cannot accept what the route will reject. Both routes now validate what they destructure.Smaller, all real
:first-childwas rewritten to:firstby an alternation matching a prefix, so those rules were dropped from every HTML export&,<and>instead of escaping themAGENTS.mdtold the receiving agent the 17 theme tokens sit on the outermost element; they sit on its first childaria-pressedand a weight cue now, no restylepickedThemewhile returningprompt, a leftover whose fallback was unreachablescripts/sources.mjsread only SSEdata:lines, so a spec-compliant JSON reply was reported as no answerFour things had no tests at all
lib/compose.ts,lib/edit.ts,lib/export.ts, and the erasable-syntax constraint that letsscripts/mcp.mjsimportlib/*.tsunder Node's type stripping — a type annotation Node cannot erase breaksnpm run mcpwith nothing in the suite noticing. All four now have them, using Node's ownstripTypeScriptTypesrather than a reimplementation.CI gained one step that drives the MCP server over a real pipe.
lib/mcp.test.tscallshandle()directly and would stay green through a transport that never starts.npm run sourcesis deliberately not in CI — it needs the network, and the step comment says so.The contrast test constrained six token pairs while the registry paints text on three more (
accent-soft, and accent text on both hero-gradient stops). Extended; no theme fails, minimum 4.54 oncoral. One correction to the finding itself, recorded in the test: it claimed lighteningwarm.accentSoftdrops the ratio — lightening raises it to 4.732; the pair is fragile in the other direction.Two findings were arguments, not bugs — deliberately not "fixed"
unclearfor that phrase on that page. Demo mode exists to replay what really happens, so it now says so throughblockedByinstead of performing an outcome the real system never produces.One bug the audit did not find
Wiring the six groups together surfaced it:
applyEditsentvariantswithout declaring it as a dependency, so after any restyle the next edit described the page as it looked two edits ago and Jev answered a question about a page that no longer existed. Nothing threw.runhad the same shape withlocale, harmless today only because the dictionary changes identity alongside it.lib/page.test.tsalready guarded this exact class for the dictionary — that rule is why the file exists. It now guards it for everything a callback puts in a request body, which is how both were caught. Mutation-verified in both directions.Verification
End to end through the real editor with Playwright: generate, edit twice, retheme, export the bundle. Zero page errors. The edit box is disabled during the stream and enabled after; 14 buttons now carry
aria-pressed.Every test added was mutation-verified. The agents reported the mutations that stayed green as well as the ones that went red — including one test whose name promises more than it checks, and two regex guards that are redundant in isolation. Those are in the commit history rather than hidden.
Reported, not fixed — out of the scope this PR was given
catalog.validate()does not check props deeply. Probed:{type:"Pricing", props:{tiers:"免费"}}returnssuccess: true, so the file whose header calls itself "the component contract" does not enforce that contract at runtime.layoutRules()readscontent.features?.lengthandcontent.tiers?.length, so a string field votes on layout by its character count —"免费"is length 2 and electspricing_multi.lib/themes.ts:126still answers an unknown theme key withforestsilently, andapp/registry.tsx:19reaches it with model output typedz.string().app/registry.tsxcould render fromvar(--border)/var(--muted); underinkandterminalthe mock window currently keeps red/amber/green chrome while everything around it is monochrome. The README now documents the exception rather than the code changing.