Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
3 changes: 2 additions & 1 deletion .claude/skills/codetrial-verify/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,8 @@ what the generator would write now. A failure here is not a bug in your change;
it means the source moved and the output did not:

```sh
python3 scripts/gen-problems.py # web/problems, web/judges
python3 scripts/gen-problems.py # web/problems, web/judges, web/problem-pages.json,
# src/agent/problem_{topics,variants,guides}.rs
python3 scripts/gen-problem-cards.py # the problem cards in web/index.html
node scripts/gen-wire-fixtures.mjs # browser/agent wire fixtures
node scripts/gen-recording-fixtures.mjs
Expand Down
60 changes: 60 additions & 0 deletions docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,8 +153,68 @@ because the fetcher never asks LeetCode for problem prose, and each judge case
carries its input without an expected value, because `exampleTestcases` is
inputs only. Both are written by someone who has read the problem.

A ported problem also needs its entry in `problem-bank/variants.json`, and the
generator refuses to run without one. The page shows a scenario written around
the same contract instead of the published problem, and the interviewer holds
the rest. Each variant has a `title` and a `brief`, which the page shows; a
private `contract` the interviewer judges against; one or two `examples`, each
naming a judge case by index and not all of them published ones;
`clarifications` answered only when asked, with the constraints and edge-case
policies the brief leaves out; `followUps` for after a tested solution; and
three `hints`, a nudge, a direction and the key step.

A function problem also declares a new `entry` and a class problem a new
`className`, since the published name is as recognisable as the title. Either
may declare `parameters` where a parameter name gives it away, and `terms` for
any other word a starter snippet carries, such as a comment naming the
published node type. The bank keeps LeetCode's names; the generator puts the
variant's in place in the judge and starter code it writes, and refuses a
variant whose title, brief, contract, examples, constraints, starter code or
judge cases still name the published problem.

The browser knows a problem only by its page name, made from the scenario
title: the interview URL, the page and judge files, the token request and saved
history all use it, and the server resolves it to the problem. Each page names
its published title once, as `source`, which the interview shows in small print
beside the scenario so the problem can be found again afterwards. The published
id is in one file, `web/problem-pages.json`, which the browser fetches only to
open a link that still carries a published id, to read history saved before
pages had names, or when the candidate asks the lobby to show the titles on the
cards. The title reaches the live interview page only: saved history, the
downloaded report and the replay keep the scenario's.

None of this is secrecy. The page map, each page's `source` and every judge
are served to anyone who asks, and a candidate with developer tools can map a
scenario back to its published problem and read its test cases. The disguise
exists so an honest candidate meets the problem the way an interview poses it;
nothing that must hold against a determined one, integrity checks included,
may rely on it.
Keep a variant's title stable once it ships: a retired page name opens the
default exercise, with a note saying so.

`problem-bank/guides.json` holds optional solution notes for the report
reviewer, with the license they are used under. The generator refuses a note
for a problem the bank does not have, or one still carrying page furniture from
the import, and writes `src/agent/problem_guides.rs`. Only the report prompt
reads them.

## Checks outside the gate

Whether the interviewer actually follows the live prompt, rather than whether
the prompt says the right things, needs a Gemini key. The check scripts a
candidate through three problems against a text model given the same
instructions, greeting and tools, and fails on a named source, a volunteered
limit, an unanswered size question, or a hint that goes past the rung it was
served:

```bash
scripts/interview-behavior-check.sh
BEHAVIOR_PROBLEMS=3sum,lru-cache scripts/interview-behavior-check.sh
```

A free key allows fifteen requests a minute, so the check waits out rate
limits; three problems take about two minutes.

The end-to-end browser check additionally needs Playwright and Chromium:

```bash
Expand Down
7 changes: 6 additions & 1 deletion docs/interview-contract-versions.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,11 @@ can select it.

## The active bundle

Bundle 4: live prompt 1, report prompt 4, rubric 1, report schema 1.
Bundle 5: live prompt 2, report prompt 5, rubric 1, report schema 1.

| Bundle | Introduced |
|---|---|
| 5 | Each problem posed as an interview scenario rather than the published problem: the live prompt holds the scenario, its private contract and the clarifications to answer when asked, the follow-ups arrive with the evidence that completes the coding round, and the prompt never holds the source title, the hint ladder or a solution walkthrough; `log_hint` serves the authored hints one rung per request and holds the last until the candidate has stated an approach, meaning Algorithm evidence observed from what they said or Coding evidence, which needs code they wrote; a request answered with a withheld rung gives no clue and is not counted as a hint; Coding, Test and Optimizations evidence is refused until the editor holds code the candidate wrote beyond the starter; the report prompt gives the reviewer both the published problem and the scenario, with the reference notes, and forbids naming the published problem in anything written to the candidate |
| 4 | The observable-delivery policy, made explicit in the report prompt and the server validator, with no change to the rubric or the public shape |
| 3 | Framework phase scores kept explicitly formative, and prohibited from mechanical use in a hiring decision while calibration remains incomplete |
| 2 | Provider-enforced structured report output and strict validation, with no change to rubric semantics or the public schema |
Expand All @@ -28,6 +29,10 @@ move together. A released bundle number is never reused for different behavior.
- Reports without `interviewContract` predate this contract. They stay readable
and are labeled `legacy/unversioned`; they are never assigned the current
rubric.
- The browser scores the active bundle and bundle 4, which shares its rubric
and report schema and differs only in the prompts that wrote the report
(`SCORABLE_CONTRACTS` in `web/lib.js`). A report keeps the bundle it claims.
A bump that changes the rubric or the schema does not join that list.
- The browser renders the active report schema normally. An older renderer may
ignore additive fields only after the bundle and schema migration explicitly
permits it.
Expand Down
Loading