Conversation
A runner that could not start was answered like a failing test, so the interviewer asked what the failures had in common when there were none. A setup error now gets its own reaction: read the first error, say whether it stops loading, compiling or running, and name what to check before running again. A real failure asks for one failing case, its expected result and what the code produced, which is a question a candidate can answer rather than a pattern to guess at.
Once a candidate passed a problem it never came back, so nothing checked whether they could still solve it a week later. A passed problem now returns after a growing interval (1, 3, 7, 14, then 30 days) and the lobby says the review is due. Reports carry a verdict and a timestamp and nothing about confidence, so the schedule stays that simple rather than pretending to model retention. The picker reads each entry as it was saved, with only the timestamp normalized. Reading the sanitized report instead dropped verdicts from older contract bundles and turned ungraded sessions into failures.
A report's improvement plan used to end with the report. The lobby now carries its most recurring item forward as a practice focus, with its drill and what success looks like, and a candidate may choose to share it with the interviewer. Shared, it can shape at most one neutral follow-up, and the interviewer is told never to call it a weakness or a grading target. It is quoted whole, so it gets a bound of its own rather than the 80 characters a role or company gets, and it stays out of saved history and recordings. It reaches the interview through session storage, read once, and never through the URL, where a crafted link could put its own text into the interviewer's instructions without the lobby's consent.
A candidate shown the practice problem as published recognises it and recites an answer, which is not what an interview measures. Each problem is now posed as a scenario written around the same contract: a renamed entry point or class, and renamed parameters where the published ones identify it, a brief that leaves the limits to be asked for, clarifications, follow-ups and a private contract for the interviewer, and three hints that log_hint serves one rung at a time, holding the last until the candidate has an approach. The published title is shown once in small print so the problem can be found again afterwards, but never its number, statement or examples. The browser knows a problem by its page name; the published id lives in one map that is fetched for old links, old history and the opt-in lobby toggle. The generator refuses a variant whose text, examples, starters or judge cases still name the source, and every variant was solved from its brief alone against the judge. Solution notes move into problem-bank and reach only the report prompt, which may not name the published problem to the candidate. Bundle 5 replaces bundle 4, which stays scorable because the rubric and schema are unchanged. The disguise serves honest candidates and is not secrecy: the page map, each page's title and every judge are served to anyone.
A page could keep one published example beside one of the judge's own, and those were often the most memorable inputs in the bank: pwwkew, paper and title, anagram and nagaram. A published case is now never shown, and a case counts as published when any one argument is, so the published tree with another k is still the published tree. The scenarios that showed one use an unpublished case instead, and where the judge had none worth showing it gains one, 58 in all, each checked against the solutions written blind from the briefs.
With the examples gone from the page, nineteen scenarios still walked the candidate through one in a hint or a clarification: 3, 0, 6, 1, 5, the four windows of the published interval example, apple and app. Those now use values of their own, and the generator refuses prose, example output text included, that repeats a published example's input or any one of its arguments: in the published order, or in any order once it holds three distinct values. The hints, follow-ups and clarifications, which the interviewer may say aloud, are held to the source-title rule too; only the brief and contract were before.
A case label is printed beside OK or FAIL on every run, and many still spoke the published problem: a magazine too short, a cycle at the head, four queens, a larger staircase. They now use each scenario's own words. The generator carries a renamed parameter into the labels and checks them as they ship: none may spell the published entry point however it is spaced, call a case an example, or repeat another label in the same judge. No name a variant gives, parameters and terms included, may bring back the published title or entry point.
Four C starters return an array through returnSize without the note every other such starter carries, that the array must be allocated with malloc and the caller frees it. A candidate following one of them could return a stack array, which the harness then reads after it is gone.
A session ticked Coding with nothing typed, because the checklist is the interviewer's recorded evidence and the model recorded it from what the candidate said they would write. The server now refuses Coding, Test and Optimizations evidence until the editor holds code the candidate added beyond the starter: a few characters typed in order, so deleting the starter's comment neither counts nor cancels a one line answer, and a table bound keeps a huge buffer from stalling the room. The starters are the server's own, generated beside each variant and seeded with the hint ladder by RuntimeState::for_problem, so no packet can move the baseline, and a tab switch, a tab round trip or a paste into a cleared editor is measured against the real starter. A code packet without code changes nothing. The interviewer is told to read the editor before recording those phases, and skips are still accepted.
A request answered with the withheld last rung gave the candidate no clue, yet it counted as a hint and the report marks hints down. It is no longer counted. The key step also unlocked on any Algorithm evidence, which the interviewer can record from inference alone; it now needs Algorithm observed from what the candidate said, or Coding evidence, which already needs code they wrote. Bundle 5 records both rules and the written-code check.
History saved before problems had page names carries published ids, and the lobby translated them through the page map on every visit, for local and account history alike. The lobby now writes page names back to local history once. Account reports are stored under the published id whatever name they arrive with, since only the current page name resolves and a stored page name would name nothing after a rename, and the server reads them back under the page name of the day.
scripts/gen-problems.py had grown to 1,347 lines holding the LeetCode fetch, the bank reader, every variant rule and the writers, with one 227-line function doing all of the checking. It is now the command over a problem_bank package: bank, fetch, rules and emit, with validated_variant a sequence of named rule steps. Behaviour is unchanged; the generated files are byte for byte the same. The one test that patched a module global now hands in the bank's path, which the thread-pooled runner needs anyway.
rust_str escaped quotes, backslashes, newlines and non-ASCII, but wrote any other ASCII control character literally, so a bank string carrying one could stop the generated variant or guide table compiling. Every character outside printable ASCII is escaped now.
Three reports from the bank refresh tooling said less than they should. --plan-drift folded a repeated slug into a set and called the plan in sync, which the sync then refused. --scaffold did not say that a Node return needs an outputType or an in-place problem an output parameter, so a copied entry loaded and graded the wrong value. The fetch summary counted cache hits as network fetches.
The steps validated_variant runs were only exercised through whole variants, and the tuned numbers had nothing holding their boundaries: what counts as a recognisable published argument, when prose quotes one, and how much typing counts as written code. A step that quietly passed everything, or a threshold moved by one, would have shifted all 150 variants or every session unnoticed. Each rule step now has a test of its own on the coin-change fixture, the two unnamed literals in the quote check are named beside the others with why they sit where they do, and each threshold is pinned just below and at its boundary. validated_variants takes the parsed bank as an argument so the test does not read the real file.
The live prompt carried every follow-up from the first turn, several hundred characters paid on each turn of a session before they could be used, and a model holding them could raise one early. The prompt now says they are held back, and the evidence call that completes the coding round returns them once. A cold restart after that point hands them over again with each round given its own next step: a solved problem goes on to the follow-ups, a behavioral round stays with the candidate's answer, and a coding round in progress returns to the editor.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A candidate shown a practice problem as it is published recognises it and recites a remembered answer, which is not what an interview measures. This branch poses every problem as a workplace scenario written around the same judged contract: a renamed entry point or class, and renamed parameters where the published ones identify it, a brief that leaves the limits to be asked for, clarifications and follow-ups the interviewer can use, a private contract, and a three-rung hint ladder that
log_hintserves one rung at a time, withholding the last until the candidate has shown an approach. The interview page names the published title once in small print, so the problem can be found again afterwards, but never its number, statement or examples; the interviewer is told never to name it. The browser knows a problem by its page name, and the published id lives in one map that resolves old links and old history and backs the lobby's opt-in toggle. Solution notes reach only the report prompt, which may not name the published problem back to the candidate. The contract moves to bundle 5, and bundle 4 stays scorable because the rubric and schema did not change.The generator holds the disguise: it refuses a variant whose text, starters, constraints, parameter names or case labels still name the source, a page example that is a published example or reuses one of its arguments, and a hint or clarification that walks through one. Where a judge had no unpublished case worth showing, it gained one, each checked against solutions written blind from the briefs.
The REACTO checklist is ticked from the interviewer's recorded evidence, and a session once ticked Coding with nothing typed. The server now refuses Coding, Test and Optimizations evidence until the editor holds code the candidate added beyond the starter, counted in order so deleting the starter's comment neither counts nor cancels a one-line answer. The browser sends each language's starter beside the buffer, so a tab switch, a tab round trip or a paste into a cleared editor is measured correctly.
Four smaller changes come first, each standing on its own: a test setup error is told apart from a failing test, passed problems return for spaced review, a candidate can record an optional readiness check-in that never reaches the interviewer or the score, and a report's recurring improvement item can be shared as a practice focus that shapes at most one neutral follow-up.
Verified with
./scripts/test.shat the tip: exit 0, 481 browser tests and none skipped. The series was rebuilt with the full gate at the scenario and REACTO commits, and the drift checks, generator tests, Rust web and agent tests and the page-shape browser suites at each commit between them. Every variant was solved from its brief alone against its judge, 150 of 150, and every judge case added since was run against those solutions.scripts/interview-behavior-check.shpasses against live Gemini with the current prompts. CI'scheckandmutantslanes pass.Not run: the credentialed voice and parity checks, which need a live LiveKit room, and
scripts/visual-parity-check.sh, whose golden images were already stale on main.Summary by cubic
Poses each practice problem as a workplace scenario around the same judged contract, so a candidate who recognizes the source problem can't recite a remembered answer. Bundle 5 replaces bundle 4; bundle 4 stays scorable.
Main change
Smaller changes
problem_bankpackage and each rule step is tested at its threshold.Written for commit 3027bb8. Summary will update on new commits.