Repository navigation
Decide the copy language in the layer that knows it, and fail out loud - #1
Merged
Merged
Conversation
Two defects found by running the thing against a dead endpoint and then against a real one, rather than reasoning about either. They land together because the failure UI and the language plumbing share the same state and the same render tree. ## The language was one question that should have been two "Which language should this site be written in" went to Jev as a single Choice. Measured over ten requests it answered an English one with `zh` at 0.76 confidence, was right but shaky at 0.63 on another, and came back under 0.85 on three of the ten. Persistently low confidence is the project's own signal that the wrong party was asked, and here the question was two stacked: - which script the request is written in — kana, Hangul and Han are three regexes, and Latin script cannot be narrowed further from the text, so the editor's locale decides. The client already sent it; live mode threw it away. - whether the request asks for a different language than it is written in — that is in the sentence, and only a reader can see it. Split, the override question separated its two populations by 0.03 against 0.85 over eleven requests without misreading one, and the follow-up answered every override case at 1.00. Both ride in the existing planning round, so there is no extra round trip. Simplified versus traditional stays with Jev: it is about market and word choice, not script. Getting the decision right was half of it. The copy layer's field schema is written in Chinese down to its `字` counts, and with the language rule up in the preamble the model followed the schema instead of the instruction — a French request produced 476 Chinese characters, a German one 429. Moving the rule after the schema fixed French and not German; repeating it in the user turn as well brought French, German, Korean and English to zero. Three field descriptions were also demanding a Chinese site on their own: a team member's `name` was specified as "a Chinese name", prices as `¥`, figures in `万`. The rendered page had the same assumption in three more places. Testimonials wrapped every language in 「」, the exported HTML always claimed `zh-CN`, and the editor shell did too — which made `:lang(zh) .quoted` match an English site nested inside it, so the quote styles keyed off the element's own language now. Verified in a browser across six languages: de gets „…", fr « … », zh and ja 「…」, en and ko curly doubles. ## A dead run looked like a slow one Pointing LLM_BASE_URL at a closed port leaves Jev working and the copy layer failing mid-stream, which is the shape a real outage takes. `fetch failed` named neither the service nor the knob to turn; both fetch sites now name the layer, the endpoint and the environment variable, with hints for the statuses that have an obvious cause. Seven skeletons kept pulsing after the run had died, so the page read as still loading forever; a failure freezes them and dims the preview. The theme picker, export buttons and edit box stayed live with nothing behind them, and are hidden while a failure shows. Tests cover all of it and every one was mutation-verified: seventeen deliberate breakages, seventeen red.
Both are generated artefacts that were produced by hand once and then went stale in silence. Three fixtures sat in the repo showing Chinese copy on German, Spanish and French sites, and the screenshots taken from them showed the same thing in three translated READMEs — for anyone without keys, that recording is the product. `npm run record` re-records a fixture against a live server. The prompts live in the script, so a recording is reproducible rather than whatever one person typed that day, and it refuses to write a run that errored, that came back in a different language than it asked for, or whose copy drifted into another script. `npm run capture` re-shoots a language's three images from a demo-mode server. It reads the theme keys off the page rather than hardcoding them — the UI labels are localised, and the list it would have hardcoded was already wrong — and clips each tile against the rendered site's own box, because the panel above it changes height with the locale. Two things the encoders forced: ffmpeg as shipped on macOS decodes WebP but does not encode it, so frames go through cwebp and webpmux; and montage wants a font it does not have on a headless box, so tiles are appended instead. Also drops `npm run lint`. `next lint` was removed in Next 16, so it has been exiting 1 with "Invalid project directory"; CI never ran it, which is why nobody noticed. A script that promises a check and errors is worse than no script.
… claim All eight fixtures are fresh runs against the fixed pipeline, so the demo now shows what the system actually does. Each is checked on the way in: the language matches what was asked for, and no copy has drifted into another script. German, Spanish and French previously carried 212, 191 and 203 Chinese characters, and the screenshots taken from them showed that in three translated READMEs. The 24 images are re-captured from those fixtures with the UI in each language. Filenames gain a generation — `themes-ko-2.jpg`. GitHub caches README images by URL, so re-capturing under the same name can keep serving the old picture; the suffix is the only reliable bust. A test keeps all eight READMEs on one generation, so no reader ends up looking at a different run from everyone else. "Language is a judgement, not a detection" was the wrong claim, and the README went further and defended the 0.66 it measured as correct. It was not: the same question answered an English request with `zh` at 0.76. The section is rewritten as "Language is two questions, not one" with the measurements behind the split, and the ledger of times a low-confidence question turned out to be the wrong question goes from three entries to four. A new subsection covers the half that deciding correctly does not fix — the copy layer following its Chinese field schema instead of the instruction. The image rename, the README references and the test that ties them together have to land in one commit or the suite is red in between, which is why the docs and the artefacts are here together. CONTRIBUTING now documents `npm run record` and `npm run capture` as the path, including the generation bump and why ffmpeg is not an option for the animation.
The edit box's placeholder tells the reader, in their own language, what they can type. Demo mode is the default for anyone without keys, so a suggestion it cannot honour is the first thing they try and the first thing that does nothing. Feeding each locale's own `editPlaceholder` suggestions back through `replayEdit`: 8 of 16 dead. German, French, Spanish and Portuguese users were shown two suggestions each and neither worked. The word list behind the replay was written when there were four locales and nothing tied it to the placeholder, so the four added later slipped past. The same measurement is now a test, so the app cannot propose an edit it will not honour. Words for all eight languages, and matching folds diacritics first — people type "enjoue" and "precos" on keyboards where the accent is work. Two things the word list got wrong on its own. German inflects by dropping the e, so "dunkler" never matched "dunkel". And "sobre" is the natural French word for a plain look and a very common Spanish and Portuguese preposition, so it is replaced with unambiguous ones rather than left to misfire on "sobre os preços". Verified end to end in a browser: "Preise weglassen" and "enlève les tarifs" take the pricing block off the page, "deixe mais animado" switches the theme. The READMEs now say that demo-mode editing does not go through Jev, because every confidence it reports reads 1.00 and that is easy to mistake for Jev answering perfectly. The measurement that prompted this — edit intent itself is fine in all eight languages, 33 of 33 with a median confidence of 1.00, even though the criteria are written in Chinese — is why only the replay changed. Adding that note also went into six translations and skipped two without anything noticing, so the docs tests now compare block counts as well as headings. Heading parity catches a missing section; it says nothing about a paragraph dropped inside one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by running the system instead of reasoning about it: once against a dead endpoint, then against a real one with an English prompt. Both runs were wrong in ways nothing would have caught.
The language was one question that should have been two
POST /api/generatewith{"prompt": "a small ceramics studio in Kyoto that runs weekend workshops", "locale": "en"}returned a Chinese site.The planning round asked Jev a single Choice: which language should this site be in. Measured over ten requests it answered that English one with
zhat 0.76, was right but shaky at 0.63 on another, and came back under 0.85 on three of ten. Persistently low confidence is this project's own signal that the wrong party was asked — and the question turned out to be two stacked on top of each other:Split, the override question separated its two populations by 0.03 against 0.85 across eleven requests without misreading one, and the follow-up answered every override case at 1.00. Both ride in the existing planning round, so no extra round trip. Simplified versus traditional stays with Jev — that is about market and word choice, not script.
Deciding correctly turned out to be half of it. The copy layer's field schema is written in Chinese down to its
字counts, and with the language rule up in the preamble the model followed the schema rather than the instruction: a French request produced 476 Chinese characters, a German one 429. Moving the rule after the schema fixed French and not German. Repeating it in the user turn as well brought French, German, Korean and English to zero. Three field descriptions were separately demanding a Chinese site — a team member'snamespecified as "a Chinese name", prices as¥, figures in万.The rendered page carried the same assumption in three more places: testimonials wrapped every language in
「」, the exported HTML always claimedzh-CN, and so did the editor shell — which made:lang(zh) .quotedmatch an English site nested inside it. Quote styles now key off the element's own language.Eight real end-to-end runs afterwards, one per language:
Names and currencies follow too: Marie Dubois and
€, 김지은 and₩. An English page for a Kyoto studio still prices in yen, because currency follows the business's location rather than the copy language.Quote marks verified in a browser:
degets„…",fr« … »,zhandja「…」,enandkocurly doubles.A dead run looked like a slow one
fetch failednamed neither the service that was down nor the knob to turn. Both fetch sites now name the layer, the endpoint and the environment variable, with hints for the statuses that have an obvious cause.Seven skeletons kept pulsing after the run had died, so the page read as still loading, forever. A failure freezes them and dims the preview — verified by three screenshots with identical hashes and
animation-name: none, not by it looking stopped.The theme picker, export buttons and edit box stayed live with nothing behind them, and are hidden while a failure shows.
Fixtures and screenshots were generated artefacts nobody could regenerate
Three fixtures had been sitting in the repo showing Chinese copy on German, Spanish and French sites (212, 191 and 203 Chinese characters), and the screenshots taken from them showed it in three translated READMEs. For anyone without keys, that recording is the product.
npm run recordandnpm run capturemake both reproducible. The prompts live in the scripts, and the recorder refuses a run that errored, came back in the wrong language, or drifted into another script. The capture script reads theme keys off the page instead of hardcoding them — the list it would have hardcoded was already wrong — and clips each tile against the rendered site's own box, because the panel above it changes height with the locale.Image filenames gain a generation (
themes-ko-2.jpg), since GitHub caches README images by URL and re-capturing under the same name keeps serving the old picture. A test keeps all eight READMEs on one generation.Docs
"Language is a judgement, not a detection" was the wrong claim, and the README went further and defended the 0.66 it measured as correct. It was not. The section is rewritten as "Language is two questions, not one", in all eight languages, and the ledger of low-confidence questions that turned out to be the wrong question goes from three entries to four.
Also
Drops
npm run lint.next lintwas removed in Next 16, so it has been exiting 1 with "Invalid project directory". CI never ran it, which is why nobody noticed.Checks
tsc --noEmitclean, 149 tests passing,npm run buildclean,npm run audit:a11yreports no violations across all six themes. Every new test was mutation-verified — 23 deliberate breakages, 23 red, and the one that stayed green (a regex that let:lang(ja) .quotedstand in for the default quote pair) was tightened until it failed. All three commits build and pass on their own.