Skip to content

AK6-AK7: raea ask learns from the person: verdicts, notes and approvals, calibrate and try - #212

Merged
shinyobjectz merged 1 commit into
mainfrom
ask-calibrate
Nov 21, 2026
Merged

shinyobjectz merged 1 commit into
mainfrom
ask-calibrate

Conversation

@shinyobjectz

Copy link
Copy Markdown
Owner

Stacked on #211 (base ask). Merge #211 first.

This is AK6–AK7 in docs/ask.feature. A question in raea ask now learns from the person's own eye: it stays advice until their labels show it separates the problem, and from then on it flags at their threshold.

Labels (apps/ask/src/labels.rs)

  • A label belongs to a question and to the clips a call sent (by hash), not to one answer. New words asked of the same clips therefore share the same labels.
  • The person's labels come from two places:
    • raea ask verdict ANSWER WORD: the person's own answer to the question, in its terms. A question flagged on "no" takes "no" as the problem, and the lower half of a grade counts as a problem. A verdict outranks a note, and a note outranks an approval.
    • The feed:
      • a note of the person's (author kind person) on the posted render counts as a yes when it's pinned inside the window (1.5 s before to 0.5 s after the pin) or not pinned, and holds one of the question's label words;
      • a post they marked done (the feed's approval) with no such note counts as a no.
  • Each call now records its file's identity (size plus a hash of the first and last MB), and notes are matched against the post's copy in the feed. A later render to the same path is never labelled by an earlier render's note.
  • An agent's own look (--eye agent) is kept as eye:agent and never counted.

Calibration (apps/ask/src/calibrate.rs)

  • raea ask calibrate [ID] reads the feed's labels again and holds each labelled version to them, using the fingerprint's own auc, quantile and gates (16 labels of each kind, 3 projects, AUC 0.8).
    • Clef isn't trained on the labels, so its AUC on them is already held out. The only thing fitted is the threshold, which flags 10% of the clean labels.
    • A version that passes is kept in ~/.raea/ask/calibration.json and goes out with the question from then on. The runner's plan applies it, so every caller gets it, and raea ask list shows it. This replaces raea's shipped calibration.
    • A version that stops passing loses its calibration.
    • One under AUC 0.6 after 32 labels "doesn't separate" and lists the labels it missed.
  • raea ask try "WORDS" --like ID sends the new words, with each labelled call's kept clips and state, through the same runner. It prints both AUCs on the same labels, and the new answers are recorded as their own version.

Also

  • The record writes each row in a single write, so concurrent runs can't split each other's rows. The tests caught this.
  • The manual page covers labels, calibrate and try, and describes the --eye and --like flags.
  • docs/ask.feature marks AK6 and AK7 done.
  • AK8's double check is now planned as its own crate, raea-app-double. raea-app-ask is at 1,841 of its 2,000 lines, and L1's tasks check counts planned lines, so keeping AK8 inside it would put the crate over the limit.

Checked

  • raea-app-ask: 9 tests pass, five times in a row. The new ones, which never spend:

    • a person's note on the posted render labels it, and an approved post labels it clean;
    • an agent's note isn't read, and another render with the same file name isn't matched;
    • verdicts are read in the question's terms, and another eye's verdict isn't counted;
    • 18 labels of each kind from 3 projects calibrate a question at AUC 1.0, and the runner then sends it with that calibration;
    • flat answers don't separate and list their misses;
    • new words on the same 36 kept clips go from AUC 0.5 to 1.0.
  • raea ask calibrate on this machine's real record runs and reports that it has no labels yet.

  • The ask code adds no dupes groups.

  • L1 (tools/agents/check --land --lane ask-calibrate) passes:

    • cargo check, licenses, unused deps;
    • dupes (294 exact, 58 near);
    • one-version, windows paths, layers, names, tasks, sdk coverage, sdk props, taxonomy, context examples.

    The same five steps fail as on AK0-AK4: raea ask, questions about a piece's picture and sound, paid from the person's plan #211, none on this branch's files:

    • vendor (bevy_hanabi);
    • no-void and size (skills/video-decompiler);
    • cli surface and manual reference: the shared binary carries other lanes' verb changes, and their pages are out of date.

…ls label it, calibrate and try

A label belongs to a question and the clips a call sent, so new words asked of the same clips share its labels.
raea ask verdict ANSWER WORD is the person's answer in the question's own terms; a note of theirs on the posted
render, inside the window and holding one of the question's label words, is a yes, and a post they approved without
one is a no. Each call now keeps its file's identity, so the note is matched to the render it was left on, never to a
later render at the same path. An agent's own look (--eye agent) is kept and never counted.

raea ask calibrate holds each labelled version to the person's labels through the fingerprint's own AUC, quantile
and gates (16 of each kind, 3 projects, 0.8); a version that passes flags where 10% of their clean work would, and
goes out with the question from then on, in place of raea's shipped calibration. One under 0.6 after 32 labels lists
what it missed. raea ask try sends new words with the kept clips through the same runner and prints both AUCs.

The record now writes each row in one write, since several runs append to the same files.
@shinyobjectz
shinyobjectz changed the base branch from ask to main November 21, 2026 00:39
@shinyobjectz
shinyobjectz merged commit 5a57a5e into main Nov 21, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant