Skip to content

Pose each problem as an interview scenario - #38

Merged
jserv merged 16 commits into
mainfrom
insight
Sep 14, 2026
Merged

Pose each problem as an interview scenario#38
jserv merged 16 commits into
mainfrom
insight

Conversation

@jserv

@jserv jserv commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

A candidate shown a practice problem as it is published recognises it and recites a remembered answer, which is not what an interview measures. This branch poses every problem as a workplace scenario written around the same judged contract: a renamed entry point or class, and renamed parameters where the published ones identify it, a brief that leaves the limits to be asked for, clarifications and follow-ups the interviewer can use, a private contract, and a three-rung hint ladder that log_hint serves one rung at a time, withholding the last until the candidate has shown an approach. The interview page names the published title once in small print, so the problem can be found again afterwards, but never its number, statement or examples; the interviewer is told never to name it. The browser knows a problem by its page name, and the published id lives in one map that resolves old links and old history and backs the lobby's opt-in toggle. Solution notes reach only the report prompt, which may not name the published problem back to the candidate. The contract moves to bundle 5, and bundle 4 stays scorable because the rubric and schema did not change.

The generator holds the disguise: it refuses a variant whose text, starters, constraints, parameter names or case labels still name the source, a page example that is a published example or reuses one of its arguments, and a hint or clarification that walks through one. Where a judge had no unpublished case worth showing, it gained one, each checked against solutions written blind from the briefs.

The REACTO checklist is ticked from the interviewer's recorded evidence, and a session once ticked Coding with nothing typed. The server now refuses Coding, Test and Optimizations evidence until the editor holds code the candidate added beyond the starter, counted in order so deleting the starter's comment neither counts nor cancels a one-line answer. The browser sends each language's starter beside the buffer, so a tab switch, a tab round trip or a paste into a cleared editor is measured correctly.

Four smaller changes come first, each standing on its own: a test setup error is told apart from a failing test, passed problems return for spaced review, a candidate can record an optional readiness check-in that never reaches the interviewer or the score, and a report's recurring improvement item can be shared as a practice focus that shapes at most one neutral follow-up.

Verified with ./scripts/test.sh at the tip: exit 0, 481 browser tests and none skipped. The series was rebuilt with the full gate at the scenario and REACTO commits, and the drift checks, generator tests, Rust web and agent tests and the page-shape browser suites at each commit between them. Every variant was solved from its brief alone against its judge, 150 of 150, and every judge case added since was run against those solutions. scripts/interview-behavior-check.sh passes against live Gemini with the current prompts. CI's check and mutants lanes pass.

Not run: the credentialed voice and parity checks, which need a live LiveKit room, and scripts/visual-parity-check.sh, whose golden images were already stale on main.


Summary by cubic

Poses each practice problem as a workplace scenario around the same judged contract, so a candidate who recognizes the source problem can't recite a remembered answer. Bundle 5 replaces bundle 4; bundle 4 stays scorable.

Main change

  • Each scenario adds a renamed entry point or class, a brief, clarifications, and follow-ups returned when the coding round completes; a restarted interviewer after Test and Optimizations is told the problem is solved, not asked to return to coding. A three-rung hint ladder is served one rung per request, holding the last until the candidate shows an approach; a request with no clue isn't counted.
  • The browser and interview know only the scenario's page name; one map resolves old links and rewrites local history once. Account reports are stored under the stable published id and read back under today's page name, so a later scenario rename keeps old reports working. The page shows the published title once in small print, and the lobby can reveal the source on request.
  • The generator refuses variants that name the source in any candidate-visible text or starter, and blocks C starters that omit the returned-array malloc note.
  • Pages never show a published example; they show unpublished judge cases, adding new ones when the bank has none.
  • Solution notes reach only the report prompt, which never points the candidate back to the published problem.
  • Coding, Test, and Optimizations evidence requires code typed beyond the starter; skips still count.

Smaller changes

  • A test setup error is told apart from a failing test.
  • Passed problems return after 1, 3, 7, 14, then 30 days, and the lobby says when review is due.
  • A recurring report improvement can be shared as a practice focus that shapes at most one neutral follow-up, passed via session storage.
  • The generator is split into a problem_bank package and each rule step is tested at its threshold.

Written for commit 3027bb8. Summary will update on new commits.

Review in cubic

A runner that could not start was answered like a failing test, so
the interviewer asked what the failures had in common when there were
none. A setup error now gets its own reaction: read the first error,
say whether it stops loading, compiling or running, and name what to
check before running again. A real failure asks for one failing case,
its expected result and what the code produced, which is a question a
candidate can answer rather than a pattern to guess at.
Once a candidate passed a problem it never came back, so nothing
checked whether they could still solve it a week later. A passed
problem now returns after a growing interval (1, 3, 7, 14, then 30
days) and the lobby says the review is due. Reports carry a verdict and
a timestamp and nothing about confidence, so the schedule stays that
simple rather than pretending to model retention.

The picker reads each entry as it was saved, with only the timestamp
normalized. Reading the sanitized report instead dropped verdicts from
older contract bundles and turned ungraded sessions into failures.
cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

A report's improvement plan used to end with the report. The lobby
now carries its most recurring item forward as a practice focus, with
its drill and what success looks like, and a candidate may choose to
share it with the interviewer. Shared, it can shape at most one
neutral follow-up, and the interviewer is told never to call it a
weakness or a grading target. It is quoted whole, so it gets a bound of
its own rather than the 80 characters a role or company gets, and it
stays out of saved history and recordings.

It reaches the interview through session storage, read once, and never
through the URL, where a crafted link could put its own text into the
interviewer's instructions without the lobby's consent.
A candidate shown the practice problem as published recognises it and
recites an answer, which is not what an interview measures. Each problem
is now posed as a scenario written around the same contract: a renamed
entry point or class, and renamed parameters where the published ones
identify it, a brief that leaves the limits to be asked for,
clarifications, follow-ups and a private contract for the interviewer,
and three hints that log_hint serves one rung at a time, holding the
last until the candidate has an approach. The published title is shown
once in small print so the problem can be found again afterwards, but
never its number, statement or examples. The browser knows a problem by
its page name; the published id lives in one map that is fetched for
old links, old history and the opt-in lobby toggle. The generator
refuses a variant whose text, examples, starters or judge cases still
name the source, and every variant was solved from its brief alone
against the judge.

Solution notes move into problem-bank and reach only the report prompt,
which may not name the published problem to the candidate. Bundle 5
replaces bundle 4, which stays scorable because the rubric and schema
are unchanged.

The disguise serves honest candidates and is not secrecy: the page map,
each page's title and every judge are served to anyone.
A page could keep one published example beside one of the judge's own,
and those were often the most memorable inputs in the bank: pwwkew,
paper and title, anagram and nagaram. A published case is now never
shown, and a case counts as published when any one argument is, so
the published tree with another k is still the published tree. The
scenarios that showed one use an unpublished case instead, and where
the judge had none worth showing it gains one, 58 in all, each checked
against the solutions written blind from the briefs.
With the examples gone from the page, nineteen scenarios still walked
the candidate through one in a hint or a clarification: 3, 0, 6, 1, 5,
the four windows of the published interval example, apple and app.
Those now use values of their own, and the generator refuses prose,
example output text included, that repeats a published example's
input or any one of its arguments: in the published order, or in any
order once it holds three distinct values. The hints, follow-ups and
clarifications, which the interviewer may say aloud, are held to the
source-title rule too; only the brief and contract were before.
A case label is printed beside OK or FAIL on every run, and many still
spoke the published problem: a magazine too short, a cycle at the head,
four queens, a larger staircase. They now use each scenario's own
words. The generator carries a renamed parameter into the labels and
checks them as they ship: none may spell the published entry point
however it is spaced, call a case an example, or repeat another label
in the same judge. No name a variant gives, parameters and terms
included, may bring back the published title or entry point.
Four C starters return an array through returnSize without the note
every other such starter carries, that the array must be allocated
with malloc and the caller frees it. A candidate following one of them
could return a stack array, which the harness then reads after it is
gone.
A session ticked Coding with nothing typed, because the checklist is
the interviewer's recorded evidence and the model recorded it from what
the candidate said they would write. The server now refuses Coding,
Test and Optimizations evidence until the editor holds code the
candidate added beyond the starter: a few characters typed in order,
so deleting the starter's comment neither counts nor cancels a one line
answer, and a table bound keeps a huge buffer from stalling the room.

The starters are the server's own, generated beside each variant and
seeded with the hint ladder by RuntimeState::for_problem, so no packet
can move the baseline, and a tab switch, a tab round trip or a paste
into a cleared editor is measured against the real starter. A code
packet without code changes nothing. The interviewer is told to read
the editor before recording those phases, and skips are still accepted.
A request answered with the withheld last rung gave the candidate no
clue, yet it counted as a hint and the report marks hints down. It is
no longer counted. The key step also unlocked on any Algorithm
evidence, which the interviewer can record from inference alone; it
now needs Algorithm observed from what the candidate said, or Coding
evidence, which already needs code they wrote. Bundle 5 records both
rules and the written-code check.
History saved before problems had page names carries published ids,
and the lobby translated them through the page map on every visit, for
local and account history alike. The lobby now writes page names back
to local history once. Account reports are stored under the published
id whatever name they arrive with, since only the current page name
resolves and a stored page name would name nothing after a rename, and
the server reads them back under the page name of the day.
scripts/gen-problems.py had grown to 1,347 lines holding the LeetCode
fetch, the bank reader, every variant rule and the writers, with one
227-line function doing all of the checking. It is now the command
over a problem_bank package: bank, fetch, rules and emit, with
validated_variant a sequence of named rule steps. Behaviour is
unchanged; the generated files are byte for byte the same. The one test
that patched a module global now hands in the bank's path, which the
thread-pooled runner needs anyway.
rust_str escaped quotes, backslashes, newlines and non-ASCII, but wrote
any other ASCII control character literally, so a bank string carrying
one could stop the generated variant or guide table compiling. Every
character outside printable ASCII is escaped now.
Three reports from the bank refresh tooling said less than they should.
--plan-drift folded a repeated slug into a set and called the plan in
sync, which the sync then refused. --scaffold did not say that a Node
return needs an outputType or an in-place problem an output parameter,
so a copied entry loaded and graded the wrong value. The fetch summary
counted cache hits as network fetches.
The steps validated_variant runs were only exercised through whole
variants, and the tuned numbers had nothing holding their boundaries:
what counts as a recognisable published argument, when prose quotes
one, and how much typing counts as written code. A step that quietly
passed everything, or a threshold moved by one, would have shifted all
150 variants or every session unnoticed. Each rule step now has a test
of its own on the coin-change fixture, the two unnamed literals in the
quote check are named beside the others with why they sit where they
do, and each threshold is pinned just below and at its boundary.
validated_variants takes the parsed bank as an argument so the test
does not read the real file.
The live prompt carried every follow-up from the first turn, several
hundred characters paid on each turn of a session before they could be
used, and a model holding them could raise one early. The prompt now
says they are held back, and the evidence call that completes the
coding round returns them once. A cold restart after that point hands
them over again with each round given its own next step: a solved
problem goes on to the follow-ups, a behavioral round stays with the
candidate's answer, and a coding round in progress returns to the
editor.
@jserv
jserv merged commit 15dc24c into main Sep 14, 2026
9 checks passed
@jserv
jserv deleted the insight branch September 14, 2026 09:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant