Skip to content

Extend practice feedback and problem bank checks - #54

Open
jserv wants to merge 22 commits into
mainfrom
dev
Open

jserv wants to merge 22 commits into
mainfrom
dev

Conversation

@jserv

@jserv jserv commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

This branch widens what a practice session gives back and tightens what the problem bank has to prove. Candidates can write their own test cases and see what their code returns, practice without a camera when the interview is not recorded, and get a post-interview debrief that states the hint rung they reached and the level they practiced for, while the report is refused if it names the published problem. The lobby keeps a longer history whose reports can be reopened, keeps due reviews across level changes, resets a failed review to one day, and shows progress by topic. The bank now accepts original exercises, every judge needs five cases including a boundary one, and a C starter that returns an array has to say who frees it. On the server, a provider whose credentials are refused leaves rotation, and report-model reviews during quiet pauses are capped per interview. CI gains a check that the release image matches the manifest's Rust version and a weekly run that requires the pinned avatar model, and the legacy migration paths for the git hooks and vendored files are removed. The interview contract moves to bundle 8, live prompt 3 and report prompt 8, with each step recorded in docs/interview-contract-versions.md.

Verified with ./scripts/test.sh at each of the 18 commits, not only the tip, and all pass; the tip runs 672 Rust tests and 523 browser tests with nothing skipped. The guarantees the commits claim were checked by breaking the guarded line and confirming a test failed: a prompt edit without a version bump, the interim review cap and its limit of 72, 401 and 403 provider refusal, the published-name check, judge case coverage, the C ownership note, and the version parity check. The candidate case fixes were reproduced before and after in Chromium. One upstream test this branch does not touch, setup_page_accepts_credentials_and_writes_the_primary_config_file in tests/cli.rs, failed once under load and passed on every rerun.

Deliberately left for follow-up: a report for an original exercise can still mention LeetCode, because the published-name check keys off a source title those exercises do not have; a sixth candidate case blocks the test run instead of being refused on its own; a null expected value breaks the palindrome checker; a mistyped class-case method breaks the whole C++ or Java compile; only four of the five candidate cases reach the reviewer; the parity check passes when neither version can be read; and the failed-review test cannot fail as written. A checkout that still holds web/vendor/avatar/jim.vrm from before the model moved to the browser cache now fails verify-vendor.sh until the file is deleted.


Summary by cubic

Gives practice sessions richer feedback and tightens what the problem bank has to prove. Candidates can add their own typed test cases, skip the camera when the interview isn't recorded, and get a debrief with the hint rung, volunteered hints, and practiced level; reports that name the published problem are repaired rather than delivered.

Practice and reports

  • Browser-observed candidate cases run beside judge cases and reach both the interviewer and the report prompt; class judges read return types from the contract so a case without an expected value still builds.
  • The lobby keeps a compact history of 500 attempts, reopens reports past the full-report cap, keeps due reviews across level changes, resets a failed review to one day only when it is the newest review outcome, and shows progress by topic.
  • Valid review history and candidate-case checker failures stay visible when neighboring entries are malformed.
  • Quiet-pause interim reviews are capped per interview (CODETRIAL_MAX_INTERIM_REVIEWS, default 12, max 72).
  • The interview contract moves to bundle 8, live prompt 3, report prompt 8, and report schema 2; schema 1 reports remain scored.

Problem bank and tooling

  • The bank accepts original problems, adding the original fixed-capacity-ring-buffer; every judge needs five cases including a boundary case, and C starters that return arrays must say who frees them.
  • Tree contracts now admit empty trees, and malformed bank input is refused consistently.
  • Provider projects rejected with 401 or 403 leave rotation until their credential is replaced.
  • CI checks that the release Rust image matches Cargo.toml's rust-version and runs a weekly avatar-model check.
  • Legacy git-hook and vendored-avatar migration paths are removed; a stale web/vendor/avatar/jim.vrm must be deleted before verify-vendor.sh passes.

Written for commit ebf1d89. Summary will update on new commits.

Review in cubic

A prompt edit used to invalidate stored scores, so every change had
to be added to a browser-side list that nothing forced anyone to
update. Compare the recorded rubric and schema versions instead: a
prompt-only bump leaves scores comparable and the list goes away. The
golden digest now covers every prompt builder, so an edit that skips
its version bump fails the gate rather than a review.
A report could name the published exercise or the practice site it came
from, which hands back the source the interview scenario deliberately
withholds. Validate against the selected problem at the exact paths
a candidate reads, and let Gemini repair a violation so one stray
title does not cost the whole report.
A verdict arrived with nothing to learn from it. Carry debrief metadata
on generated and fallback reports alike, under a second report schema
so the addition is recorded rather than inferred. Schema 1 reports
stay scored, because existing history is still the candidate's.
Spaced-repetition reviews were filtered by the level chosen for new
problems, so changing level silently hid work already due. Choose
due reviews before that filter applies. A failed review drops to a
one-day interval, which is what being due is for.
Full reports are capped, so an older attempt disappeared from the
lobby altogether rather than ageing out of detail. Keep a compact
index past the cap holding enough to list and reopen an attempt,
and stop asking the page map about entries the cap has already retired.
A candidate could only run the judge's cases, so the cases they
thought to try left no trace in the evidence. Saved typed cases run
beside the judge's without touching its totals, and what the browser
observed reaches both the interviewer and the report prompt, where
testing instinct is worth reading.
Every bank entry had to be an imported exercise carrying a published
title and examples, which tied the bank to one study plan and left
no way to write a problem for this tree. An entry may now declare
itself original and carry no published fields: its title comes from
the variant, validation stops demanding a source, and the plan checks
pass it over. Rubrics become generated output, because the title an
entry shows is no longer a constant the bank can state.
A hint count alone cannot separate a candidate who asked once for
a deep hint from one the interviewer kept prompting unasked. Pass
the rung reached and how many hints were volunteered, and say in the
prompt that both are context: a volunteered hint is weaker evidence
than a requested one and neither is a numeric deduction.
A missing or declined camera turned a candidate away from an interview
that never needed video. Require one only when recording is enabled,
since that is what the recording notice describes. The reason the
camera went unused is signed into the integrity trail so the report
reads a neutral condition rather than absence of evidence.
Quiet-pause reviews and the final report spend the same provider quota,
and the existing rate limits still allow seventy-two reviews in a long
interview, ahead of the report that matters. Bound them per interview,
fixed when the interview starts so a config reload cannot raise the
ceiling underneath a session already running.
The pool probe removed a project only for an explicit 429, so a project
whose credential had been revoked stayed in rotation and failed later
at token minting, where the candidate sees it. Read 401 and 403 as
refusals too. Everything else still counts as available, because
refusing an interview over a brief network fault is the worse error.
A candidate practising for a senior role got the same verdict as one
practising for a junior role, with nothing on the page saying which
bar had been applied. Pass the selected level as coaching context
and show it beside the verdict. The decision stays against the fixed
mid-level bar, which the prompt now states twice because it is what
a reviewer is most likely to let drift.
Progress listed recurring weaknesses but never which topics had
actually been practised, so a topic failed twice and a topic never
attempted looked alike. Group attempts by the topics their problems
carry, with passes and the date each was last seen.
A judge could ship with two cases and none of them near a boundary,
which makes a passing run weak evidence of anything. Require five cases
and one the declared argument types admit as small at their own domain,
rather than treating a single JSON shape as a boundary everywhere. The
exception file carries its own count, so a line left behind after
its cases are authored fails instead of excusing that judge forever.
A C starter that returns an array through returnSize says nothing
about who frees it, and a candidate reading it has no way to find
out. Require the starter comment to state that the caller frees the
returned storage.
The release workflow pinned a toolchain that nothing compared against
what the crate claims to build with, so the two could drift until
a release failed. Declare the minimum in the manifest and fail the
gate when the Linux image no longer matches it.
The avatar browser check accepts an unavailable avatar, so that a
developer without the model can still run the flow. That also means
nothing would notice the pinned model going away. Run the flow on a
schedule with the fallback disallowed, which checks the pin somewhere
without making the local run depend on it.
The hook installer and the vendor fetch carried branches for trees laid
out the way this one no longer is, and nothing left in the repository
can reach them. Require the current behavior outright. The docs go
with them: the Java lane needs a working toolchain rather than a
version nobody here tested, and the connection notes drop a claim
about which assertions moved that was never checked.
cubic-dev-ai[bot]

This comment was marked as resolved.

Keep cases, reports, and judge bounds aligned in browser and agent.
cubic-dev-ai[bot]

This comment was marked as resolved.

Apply failure resets only to newer review outcomes.
Keep valid history and checker failures visible under hostile inputs.
cubic-dev-ai[bot]

This comment was marked as resolved.

Keep empty-tree contracts and malformed bank input consistent.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 4 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="problem-bank/problems.json">

<violation number="1">
P2: The new bound contradicts the scenario this problem ships with. The constraint now claims an empty tree is valid (0 nodes), but problem-bank/variants.json for this problem states the opposite: the contract reads "receives a non-empty tree of 1 to 10^4 nodes" and clarification 3 says "Between 1 and 10^4 stations, so there is always at least the gateway." The judge in problem-bank/judges.json also has no empty-tree case. Because src/agent/prompts.rs joins variant.constraints into the interviewer's prompt alongside the contract, the interviewer now reads conflicting input domains, and a candidate may be penalized either for failing to handle `root = []` or for assuming it can occur. Either keep the published-style "1 <= number of nodes" bound, or actually support empty trees: update the variant contract and clarification and add a judge case for `root = []` (expected "[]").</violation>
</file>

<file name="src/agent/problem_variants.rs">

<violation number="1" location="src/agent/problem_variants.rs:903">
P2: The constraints line for Hop Depth Signal Means now allows an empty tree (0 nodes), but the contract still says the tree is non-empty (1 to 10^4 nodes) and the clarification says there is always at least the gateway. That contradiction tells candidates an empty input may be tested while the contract rules it out. Align them: either keep 1 as the lower bound, or update the contract and clarification (and define the empty return) if allowing an empty tree is intended.</violation>
</file>

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

brief: &["A field sensor network relays data through a tree of stations, each forwarding to at most two stations below it. Every station reports a signal reading, and the dashboard shows the mean reading for all stations that are the same number of hops from the gateway.", "Implement meanReadingPerHop(root), where root is the TreeNode for the gateway and each node's value is that station's reading, and return the mean reading for each hop distance, starting with the gateway."],
contract: "meanReadingPerHop(root) receives a non-empty tree of 1 to 10^4 nodes with 32-bit signed values and returns a list of the unrounded arithmetic mean of node values at each depth, root depth first. Means are compared as exact numbers, so integer-valued means like 3 and 3.0 both match.",
constraints: &["1 <= number of nodes <= 10^4", "-2^31 <= Node.val <= 2^31 - 1"],
constraints: &["0 <= number of nodes <= 10^4", "-2^31 <= Node.val <= 2^31 - 1"],

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The constraints line for Hop Depth Signal Means now allows an empty tree (0 nodes), but the contract still says the tree is non-empty (1 to 10^4 nodes) and the clarification says there is always at least the gateway. That contradiction tells candidates an empty input may be tested while the contract rules it out. Align them: either keep 1 as the lower bound, or update the contract and clarification (and define the empty return) if allowing an empty tree is intended.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/agent/problem_variants.rs, line 903:

<comment>The constraints line for Hop Depth Signal Means now allows an empty tree (0 nodes), but the contract still says the tree is non-empty (1 to 10^4 nodes) and the clarification says there is always at least the gateway. That contradiction tells candidates an empty input may be tested while the contract rules it out. Align them: either keep 1 as the lower bound, or update the contract and clarification (and define the empty return) if allowing an empty tree is intended.</comment>

<file context>
@@ -900,7 +900,7 @@ pub const PROBLEM_VARIANTS: &[(&str, ProblemVariant)] = &[
         brief: &["A field sensor network relays data through a tree of stations, each forwarding to at most two stations below it. Every station reports a signal reading, and the dashboard shows the mean reading for all stations that are the same number of hops from the gateway.", "Implement meanReadingPerHop(root), where root is the TreeNode for the gateway and each node's value is that station's reading, and return the mean reading for each hop distance, starting with the gateway."],
         contract: "meanReadingPerHop(root) receives a non-empty tree of 1 to 10^4 nodes with 32-bit signed values and returns a list of the unrounded arithmetic mean of node values at each depth, root depth first. Means are compared as exact numbers, so integer-valued means like 3 and 3.0 both match.",
-        constraints: &["1 <= number of nodes <= 10^4", "-2^31 <= Node.val <= 2^31 - 1"],
+        constraints: &["0 <= number of nodes <= 10^4", "-2^31 <= Node.val <= 2^31 - 1"],
         clarifications: &[("Should the means be rounded or truncated?", "No. Return each mean as a floating-point value."), ("How large can readings be?", "Each reading is any 32-bit signed integer, from -2^31 to 2^31 - 1."), ("How many stations are there?", "Between 1 and 10^4 stations, so there is always at least the gateway."), ("What order should the means be in?", "From the gateway outward: the gateway's own reading first, then one hop away, and so on.")],
         follow_ups: &["The dashboard now wants the median reading per hop distance instead of the mean. What changes?", "Stations report continuously and readings update. How would you keep the per-hop means current?", "The tree is too wide to hold a full hop level in memory. What would you do?"],
</file context>
Suggested change
constraints: &["0 <= number of nodes <= 10^4", "-2^31 <= Node.val <= 2^31 - 1"],
constraints: &["1 <= number of nodes <= 10^4", "-2^31 <= Node.val <= 2^31 - 1"],

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant