Repository navigation
AK6-AK7: raea ask learns from the person: verdicts, notes and approvals, calibrate and try - #212
Merged
Merged
Conversation
…ls label it, calibrate and try A label belongs to a question and the clips a call sent, so new words asked of the same clips share its labels. raea ask verdict ANSWER WORD is the person's answer in the question's own terms; a note of theirs on the posted render, inside the window and holding one of the question's label words, is a yes, and a post they approved without one is a no. Each call now keeps its file's identity, so the note is matched to the render it was left on, never to a later render at the same path. An agent's own look (--eye agent) is kept and never counted. raea ask calibrate holds each labelled version to the person's labels through the fingerprint's own AUC, quantile and gates (16 of each kind, 3 projects, 0.8); a version that passes flags where 10% of their clean work would, and goes out with the question from then on, in place of raea's shipped calibration. One under 0.6 after 32 labels lists what it missed. raea ask try sends new words with the kept clips through the same runner and prints both AUCs. The record now writes each row in one write, since several runs append to the same files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #211 (base
ask). Merge #211 first.This is AK6–AK7 in docs/ask.feature. A question in
raea asknow learns from the person's own eye: it stays advice until their labels show it separates the problem, and from then on it flags at their threshold.Labels (apps/ask/src/labels.rs)
raea ask verdict ANSWER WORD: the person's own answer to the question, in its terms. A question flagged on "no" takes "no" as the problem, and the lower half of a grade counts as a problem. A verdict outranks a note, and a note outranks an approval.person) on the posted render counts as a yes when it's pinned inside the window (1.5 s before to 0.5 s after the pin) or not pinned, and holds one of the question's label words;--eye agent) is kept aseye:agentand never counted.Calibration (apps/ask/src/calibrate.rs)
raea ask calibrate [ID]reads the feed's labels again and holds each labelled version to them, using the fingerprint's ownauc,quantileand gates (16 labels of each kind, 3 projects, AUC 0.8).~/.raea/ask/calibration.jsonand goes out with the question from then on. The runner's plan applies it, so every caller gets it, andraea ask listshows it. This replaces raea's shipped calibration.raea ask try "WORDS" --like IDsends the new words, with each labelled call's kept clips and state, through the same runner. It prints both AUCs on the same labels, and the new answers are recorded as their own version.Also
calibrateandtry, and describes the--eyeand--likeflags.raea-app-double. raea-app-ask is at 1,841 of its 2,000 lines, and L1's tasks check counts planned lines, so keeping AK8 inside it would put the crate over the limit.Checked
raea-app-ask: 9 tests pass, five times in a row. The new ones, which never spend:
raea ask calibrateon this machine's real record runs and reports that it has no labels yet.The ask code adds no dupes groups.
L1 (
tools/agents/check --land --lane ask-calibrate) passes:The same five steps fail as on AK0-AK4: raea ask, questions about a piece's picture and sound, paid from the person's plan #211, none on this branch's files: