Skip to content

AK0-AK4: raea ask, questions about a piece's picture and sound, paid from the person's plan - #211

Merged
shinyobjectz merged 6 commits into
mainfrom
ask
Nov 21, 2026
Merged

shinyobjectz merged 6 commits into
mainfrom
ask

Conversation

@shinyobjectz

Copy link
Copy Markdown
Owner

raea ask asks questions about a piece's picture and sound. clef-omni (the decision model behind raea's checks) answers each one in words with a probability. The call is paid from the person's raea plan, and every answer is kept so a question can be calibrated over time. This PR is AK0 through the raea side of AK4 in docs/ask.feature.

What's in it

  • raea-core-ask (core/ask) defines:

    • the question: yes/no, choose, grade or rate, plus what it looks at and listens to, what it uses, when it applies, its window and its version (a hash of its wording);
    • how an answer reads, with a 0–1 problem score so one calibration fits every type;
    • clef's limits, which used to live only in playtest.ts;
    • the standard sets:
      • rubric, read from sdk/guides/craft/rubric.md (one source, not a copy);
      • av, for picture against sound;
      • defects, the fingerprint axes that have no labels yet;
      • platform;
      • decide.

    The types are exported to sdk/raea/src/ask.gen.ts.

  • raea-io-clef (io/clef) is a single clef client, which raea judge now uses too. It handles:

    • media cutting: video at 480 px wide, at most 60 s; sound as mono MP3; stills as JPEG;
    • a content-addressed cache;
    • the routes, tried in order:
      1. the raea app's /host/ask, on the person's plan;
      2. RAEA_AGENT_KEY with RAEA_API, for a pipeline or CI;
      3. the person's own Cloudflare keys;
      4. otherwise a plain refusal.

    When the pool is spent, it stops (unless the person has their own keys), and the questions not yet asked wait instead of being dropped.

  • raea ask (apps/ask) reads the piece before it spends anything:

    • It reads the file, the nearest brief, the words file and whether anyone speaks.
    • A question that doesn't apply isn't asked, and its line says why. A piece where no one speaks gets sound-silent instead of the rubric's sound. When it's unknown whether anyone speaks, a voice question rides in the same call and settles which variant counts.
    • Each clip is cut once.
    • Every question about one clip goes in one call (up to 64 questions).
    • Calls run in parallel (--jobs).
    • Answers stream back as JSON lines (--json).
    • --batch takes many clips (a JSON list, JSON lines, or - for stdin).
    • --dry shows exactly what would be sent and its estimated cost, and spends nothing.
    • --gate / --gate any give exit codes a pipeline can use.
    • raea ask list lists the standard questions.
    • Every call, answer and cut clip is kept in ~/.raea/ask; the clips are capped at 2 GB.
  • --brief none is for a caller that writes the whole state itself, as the app's critique does.

  • raea/ask in sdk/raea provides ask() and askMany() over raea ask --batch - --json.

  • The manual pages are docs/manual/reference/cli/ask.md and sdk/ask.md, and docs/settings.md lists the new settings.

  • The ask verb is dispatched through apps/solid, because raea-cli is at its 2,000-line limit.

Related PRs

The pool route waits on raea-api's deploy. Until then, raea ask uses the person's own keys.

Checked

  • raea-app-ask has 6 tests, which never spend:

    • when and swap;
    • 65 questions make 2 calls, and calls split by window and media;
    • a pick or comparison isn't asked without --with;
    • one cut per clip on a real FFmpeg clip;
    • the voice answer sets the other variant aside;
    • a pool spent on call 3 keeps calls 1–2 and the rest waits;
    • own-words ids.
  • raea-io-clef's tests cover routing, refusals, the spent pool and replay. raea-core-ask's tests cover parsing, versions and the library.

  • raea-cli's help tests pass.

  • One real call on own keys cost $0.0003. A test pattern scored 0.88 on the defect question, "nobody speaking" scored 0.00, and "the sound fits" scored 0.06. The answer was cached and recorded.

  • L1 (tools/agents/check --land --lane ask, on origin/main plus this branch) passes:

    • cargo check, every bevy plugin, licenses, unused deps;
    • dupes (294 exact, 58 near: an earlier run found six snippets read as copies, and they were reshaped in the last commit);
    • one-version, windows paths, layers, names, tasks, sdk coverage, sdk props, taxonomy, context examples.

    Five steps fail, none on this branch's files:

    • vendor (bevy_hanabi's shader_contract_tests.rs);
    • no-void and size (skills/video-decompiler's font licence and scene.tsx);
    • cli surface and manual reference: the shared binary carries other lanes' verb changes (raea controller, show's labels), and their pages are out of date. ask's own pages are current.

…om the person's plan

docs/ask.feature sets the model and plans AK1-AK9: one question format and one clef client for every caller (the CLI,
pipelines and scripts, raea check, the fingerprint's double check, the app's agents, critique and game checks, raea
judge); ask understands a question before it spends; the person's pool first, then an agent key, then their own
Cloudflare keys; one parallel runner that cuts each clip once; standard sets (rubric, picture with sound, the
fingerprint's unlabelled axes, the check packs' questions, decisions); and answers kept and labelled so each question
earns its threshold, better words can be measured, and a calibrated question can later be learned locally.
…sets and clef's limits

A question is clef's own shape (yes, choose, grade, rate), what it needs to be asked (`when`, and the question asked in
its place with `swap`, so sound to sound-silent is data), what it looks at and listens to, its window, label words and
calibration. Every answer reads as a problem score from 0 to 1 whatever its type, so one calibration fits all; a
question flags past its calibrated threshold, or past even odds as advice. A wording is versioned by a stable hash.

The standard sets: the rubric, read from its guide (one copy); av, picture with sound; defects, the fingerprint's
seven axes with no labels, in its own words; platform, the check packs' 17 model questions with their calibration (3
pass the 0.8 gate); decide, picks and before/after. The limits are raea-api's where stricter, so a call that passes
them passes on every route. sdk/raea/src/ask.gen.ts is generated from the types and checked by a test.
…t; raea judge moved onto it

One call: the state, the questions by id, and stills (JPEG), videos (MP4) and sounds (MP3) cut with raea's FFmpeg to
clef's limits (480 px H.264 at a small bitrate, mono 16 kHz MP3, JPEG at most 1,600 px; ffprobe says first what a file
holds). It goes the first way that answers: the raea app's host (/host/ask, which adds its sign-in), an agent key
(RAEA_AGENT_KEY with RAEA_API, for pipelines and servers), the person's own Cloudflare account; with none it says how
to get one. No plan or no ask scope hands on to the next way; a spent pool stops unless the person has their own keys.
Unchanged questions on unchanged media come from a cache by request hash; RAEA_CLEF_REPLAY gives recorded answers to
tests.

raea judge asks through it (Question::from_wire reads its rubric's clef-native questions), so apps/judge's own
clef.rs is gone and the judge pays from the plan when the app is signed in. docs/settings.md lists the new settings.
… asked in parallel and kept

raea-app-ask. prepare.rs reads what raea knows of the piece before it spends: what the file holds, the nearest brief.ts
(through raea spec) and a words file. A question that doesn't apply is dropped or swapped with why (a silent piece
asks sound-silent in place of sound); not knowing whether anyone speaks, both variants go in one call with the voice
question, and the answer sets the other aside. runner.rs plans a batch into calls (by file, window, files beside it
and what the questions look at, 64 a call), cuts each clip once, calls in parallel (--jobs), streams the answers, and
when the pool is spent partway keeps what came and reports the rest as waiting. record.rs keeps every call and
answer under ~/.raea/ask with the clips it cut, by hash, at most 2 GB. verbs.rs is raea ask: own words (--choose,
--grade, --rate, --flag), the standard sets (--set, --std, ask list), windows (--from, --to, --at), --with, --batch
(a JSON list or lines, - for stdin), --dry, --json and --gate for pipelines.

sdk/raea's raea/ask gives scripts ask and askMany over raea ask --batch --json. The clef client's failures are
typed so a spent pool waits rather than fails; decide.changed needs files beside it. The manual's CLI and SDK pages
are written with it; AK1, AK1b, AK2 and AK3 are done in docs/ask.feature.
…itself

The app's critique and game check now ask through raea ask (raea-feed) with their own state: --brief none (or
"brief": "none" in a batch) keeps raea ask from adding the nearest brief's facts to it, and with no brief and the
caller's state, the caller says what the piece is. docs/ask.feature AK4 says what's built and in review: the pool
route's sounds, criteria, pool left and ask scope on raea-api, and the app's /host/ask, ask tool and critique.
L1's dupes step counted 300 exact groups against 294 allowed, and each of the six new groups was ask code beside an
existing twin. Recorded answers are read in one step, the cache folder names its PathBuf, the media prune reads each
entry's metadata in one chain, a temp clip is removed with .ok(), the client encodes base64 with the engine itself,
and a question id is checked over its bytes.
@shinyobjectz
shinyobjectz merged commit 328aba8 into main Nov 21, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant