Skip to content

Fix 34 audited findings, and add the tests that would have caught them - #5

Merged
houko merged 1 commit into
mainfrom
fix/audit-sweep
Sep 21, 2026
Merged

houko merged 1 commit into
mainfrom
fix/audit-sweep

Conversation

@houko

@houko houko commented Sep 20, 2026

Copy link
Copy Markdown
Member

Six auditors read the codebase along different lines. Every finding then had to survive an adversarial pass whose only job was to refute it, and several were reproduced by running the code rather than reasoning about it. 34 survived. This fixes them, in one PR as requested.

The one that mattered most

A single failed copy chunk threw away a page that had already rendered.

asSettled in lib/content-parallel.ts is named for settling and raced raw promises instead. The README measures malformed LLM JSON at roughly one run in two, so one of four chunks failing is an ordinary outcome — and when it did, the rejection escaped compose.ts, reached the route as a terminal error, and the client hid the exports and the edit box on a page whose nav, hero, testimonials, pricing, FAQ and footer had all arrived and painted.

The verifier reproduced it end to end with a stubbed transport:

EVENT plan / partial frame / content identity / select / partial identity /
partial features / partial commerce / partial social / THREW: 两次都没拿到合法 JSON
last painted spec: Nav HeroCentered Testimonials Pricing FAQ Footer   ← discarded

Fixed in the helper that owns the contract rather than at the call site: asSettled now yields {status} like Promise.allSettled. It is also a plain function returning a generator rather than an async function*, because a generator body does not run until its first next() and a rejection landing in that gap is unhandled. The second Jev round's promise gets its .catch where it is created — attaching it at the await site produces an unhandled rejection in the window between, which is demonstrable and was demonstrated.

The retry could not fix the failure it exists for

finish_reason was never read, and attempt two re-sent the identical max_tokens. A truncated reply truncated again at the same place, burning the budget twice. The shipped fixtures show the place chunk already at 1067–1098 of 1300 tokens on the happy path — under 18% headroom.

Truncation is now detected; attempt two gets a larger cap and a complaint the model can act on. Other malformed-JSON causes keep the original budget, so a fenced reply does not buy tokens it will not use.

Two features were dead

  • add was never applied. It arrived well-formed from both edit layers and nothing consumed it, so the log printed add@1.00 faq · 3ms — which reads as success — and the page did not change. That also made remove one-way.
  • The token readout was structurally unreachable. steps.reduce() summed a field setSteps never wrote, so · N tok · N evals could only ever render zeros.

The stream consumer had no error path

No try, no finally, no response.ok. A rejected fetch left Generate disabled until a page reload; a non-200 was fed to the NDJSON parser as if it were events. All three closed, plus a try around the per-line JSON.parse.

MCP: a tool argument could kill the server

toString or constructor reached a plain object lookup and threw out of the handler. Reproduced over real stdio — id 1 answered, then:

TypeError: Cannot read properties of undefined (reading 'bg')
    at themeVars (lib/themes.ts:130:20) at Object.run (lib/mcp.ts:195:17)

ids 2 and 3 never came back. Every client-keyed lookup now goes through Object.hasOwn, the whole file was scanned for siblings of the pattern, and the module comment states the rule so the next table added is not a new hole. Batch requests are now handled rather than advertised and refused, and an id the server can plainly see is echoed instead of discarded.

Verified after the fix:

initialize -> 2025-11-25
BATCH -> 2 responses, ids [2, 3]
  theme_tokens{"theme":"toString"} -> forest | matched: false

Nothing capped a prompt, at any layer

A 1 MB prompt became a ~17 MB upstream body. Now 2000 characters at both routes — chosen against the longest example any locale ships (91, in pt.json) — returned as a 400, never a truncation, with maxLength on both inputs so the field cannot accept what the route will reject. Both routes now validate what they destructure.

$ curl -X POST /api/generate -d '{"prompt":"xxx…3000 chars"}'   → 400
$ curl -X POST /api/edit     -d 'not json'                      → 400

Smaller, all real

  • :first-child was rewritten to :first by an alternation matching a prefix, so those rules were dropped from every HTML export
  • the JSX exporter trimmed each text node independently and ate the space before a testimonial byline
  • the HTML title deleted &, < and > instead of escaping them
  • AGENTS.md told the receiving agent the 17 theme tokens sit on the outermost element; they sit on its first child
  • theme swatches and the locale toggle carried selected state in colour alone — aria-pressed and a weight cue now, no restyle
  • the HTML export title branched on pickedTheme while returning prompt, a leftover whose fallback was unreachable
  • scripts/sources.mjs read only SSE data: lines, so a spec-compliant JSON reply was reported as no answer

Four things had no tests at all

lib/compose.ts, lib/edit.ts, lib/export.ts, and the erasable-syntax constraint that lets scripts/mcp.mjs import lib/*.ts under Node's type stripping — a type annotation Node cannot erase breaks npm run mcp with nothing in the suite noticing. All four now have them, using Node's own stripTypeScriptTypes rather than a reimplementation.

CI gained one step that drives the MCP server over a real pipe. lib/mcp.test.ts calls handle() directly and would stay green through a transport that never starts. npm run sources is deliberately not in CI — it needs the network, and the step comment says so.

The contrast test constrained six token pairs while the registry paints text on three more (accent-soft, and accent text on both hero-gradient stops). Extended; no theme fails, minimum 4.54 on coral. One correction to the finding itself, recorded in the test: it claimed lightening warm.accentSoft drops the ratio — lightening raises it to 4.732; the pair is fragile in the other direction.

Two findings were arguments, not bugs — deliberately not "fixed"

  • Demo mode was not made to answer 「不要定价了」 by adding a pricing block. The zh fixture has no pricing, and the live system would return unclear for that phrase on that page. Demo mode exists to replay what really happens, so it now says so through blockedBy instead of performing an outcome the real system never produces.
  • The removal guard rejects a request that would drop an archetype-required block rather than quietly removing it.

One bug the audit did not find

Wiring the six groups together surfaced it: applyEdit sent variants without declaring it as a dependency, so after any restyle the next edit described the page as it looked two edits ago and Jev answered a question about a page that no longer existed. Nothing threw. run had the same shape with locale, harmless today only because the dictionary changes identity alongside it.

lib/page.test.ts already guarded this exact class for the dictionary — that rule is why the file exists. It now guards it for everything a callback puts in a request body, which is how both were caught. Mutation-verified in both directions.

Verification

npx tsc --noEmit     exit 0
npx vitest run       20 files, 347 tests, all passing   (was 13 files, 250)
npm run build        compiled successfully
npm run sources      8/8 endpoints answered
npm run mcp          real stdio: handshake, batch, and the crash case now answered
unzip -t bundle.zip  6 files, no errors

End to end through the real editor with Playwright: generate, edit twice, retheme, export the bundle. Zero page errors. The edit box is disabled during the stream and enabled after; 14 buttons now carry aria-pressed.

Every test added was mutation-verified. The agents reported the mutations that stayed green as well as the ones that went red — including one test whose name promises more than it checks, and two regex guards that are redundant in isolation. Those are in the commit history rather than hidden.

Reported, not fixed — out of the scope this PR was given

  • catalog.validate() does not check props deeply. Probed: {type:"Pricing", props:{tiers:"免费"}} returns success: true, so the file whose header calls itself "the component contract" does not enforce that contract at runtime.
  • layoutRules() reads content.features?.length and content.tiers?.length, so a string field votes on layout by its character count — "免费" is length 2 and elects pricing_multi.
  • lib/themes.ts:126 still answers an unknown theme key with forest silently, and app/registry.tsx:19 reaches it with model output typed z.string().
  • The three traffic-light hex literals in app/registry.tsx could render from var(--border)/var(--muted); under ink and terminal the mock window currently keeps red/amber/green chrome while everything around it is monochrome. The README now documents the exception rather than the code changing.

… caught them

Six auditors read the codebase along different lines, every finding then had to survive an adversarial pass whose job was to refute it, and several were reproduced by running the code rather than reasoning about it. 34 survived. This fixes them.

The worst one collapsed a page that had already rendered. `asSettled` in lib/content-parallel.ts is named for settling and raced raw promises instead, so when one of four copy chunks failed — and the README measures malformed JSON at roughly one run in two — the rejection escaped compose.ts, reached the route as a terminal error, and the client hid the exports and the edit box on a page whose nav, hero, testimonials, pricing, FAQ and footer had all arrived and painted. The fix is in the helper that owns the contract: it now yields `{status}` like `Promise.allSettled`, and it is a plain function returning a generator rather than an `async function*`, because a generator body does not run until its first `next()` and a rejection landing in that gap is unhandled. The second round's promise gets its handler where it is created, not at the await site — attaching it at the await produces an unhandled rejection in the window between, which is demonstrable.

The retry underneath it could not fix the failure it exists for. `finish_reason` was never read and attempt two re-sent the identical `max_tokens`, so a truncated reply truncated again at the same place, burning the budget twice. Truncation is now detected and the second attempt gets a larger cap and a complaint the model can act on; other malformed-JSON causes keep the original budget.

Two whole features were dead. `add` arrived well-formed from both edit layers and nothing consumed it, so the log printed `add@1.00 faq` and the page did not change — and that made `remove` one-way. The token readout summed a field that was never written, so `· N tok · N evals` was structurally unreachable.

The stream consumer had no error path at all: no `try`, no `finally`, no `response.ok`. A rejected fetch left Generate disabled until a reload, and a non-200 was fed to the NDJSON parser as if it were events.

On the MCP server, a tool argument of `toString` or `constructor` reached a plain object lookup and threw out of the handler, killing the process for the agent on the other end — reproduced over real stdio, where ids 2 and 3 went unanswered. Every client-keyed lookup now goes through `Object.hasOwn`, the module comment states that as the rule, and batch requests are handled rather than advertised and refused.

Nothing capped a prompt, at any layer. 2000 characters now, at both routes, chosen against the longest example any locale ships (91). Both routes validate what they destructure instead of trusting it.

Smaller, all real: `:first-child` was being rewritten to `:first` by an alternation matching a prefix, so those rules were dropped from every HTML export; the JSX exporter trimmed each text node independently and ate the space before a byline; the HTML title deleted `&`, `<` and `>` rather than escaping them; AGENTS.md told the receiving agent the theme tokens sit on the outermost element when they sit on its first child; the theme swatches carried their selected state in colour alone.

Four files had no tests: lib/compose.ts, lib/edit.ts, lib/export.ts, and the erasable-syntax constraint that lets scripts/mcp.mjs import lib/*.ts under Node's type stripping — invisible to the suite until `npm run mcp` breaks. All four now have them, and CI drives the MCP server over a real pipe, which is the only thing that exercises the transport rather than the handler.

The contrast test constrained six token pairs while the registry paints text on three more. Extended; no theme fails. The README claimed in all eight languages that app/registry.tsx contains no hex at all — it contains three, the macOS traffic lights in the mocked screenshot. The claim was what was wrong, and it now names its exception.

Two findings turned out to be arguments rather than bugs and were deliberately not "fixed". Demo mode was not made to answer 「不要定价了」 by adding a pricing block, because the fixture has no pricing and the live system would return `unclear`; demo mode exists to replay what really happens, so it now says so through `blockedBy` instead. And the removal guard rejects the request rather than removing a block the archetype requires.

One bug was found while wiring the parts together rather than by the audit: `applyEdit` sent `variants` without declaring it, so after any restyle the next edit described the page as it looked two edits ago and Jev answered about a page that no longer existed. `run` had the same shape with `locale`, harmless only because the dictionary changes identity alongside it. lib/page.test.ts already guarded this class for the dictionary; it now guards it for everything a callback puts in a request body, which is how both were caught.
@houko
houko merged commit 4a93993 into main Sep 21, 2026
1 check passed
@houko
houko deleted the fix/audit-sweep branch September 21, 2026 06:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant