You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(aorlta): starter plugins for three kinds of task (ISA fuzzer, Q&A expert, sanitizer triage) - #614
Three starter plugins to copy when you build a task of your own, under aorlta/examples/verticals/:
isa-fuzzer/: the model writes a short test program in a declared toy format; the placeholder check is the format's grammar, length and required instructions, with an optional external tool after that.
qa-expert/: a question in your domain, answered on an Answer: line; the placeholder check is an exact match after normalising.
sanitizer-triage/: the model reads a tool's report and answers with a JSON verdict (a category from a closed list and a rationale); the placeholder check reads only the verdict's form.
Each is framework only: a toy prompt pool (marked PLACEHOLDER, not real tasks or reports), a working placeholder check and reward with TODOs where the real one goes, a check profile, a score and a train config for one GPU, and a README under 30 lines. Each is its own plugin package, so a directory can be copied alone. There's no bug-finding content.
tests/examples/test_verticals.py loads every config, tests each placeholder check and reward (the service's reward rules: range, parts, unsettled never clean), and scores and trains each starter on CPU through the real wiring. The ISA tool gets the same treatment as the Q&A judge in #613: its identity is part of the check's version, and a timeout or close() kills its whole process group.
…&A expert and sanitizer triage
Each is a plugin package on its own under examples/verticals/: its module (task, parse with a
repair turn, feedback, check adapter, reward function), its check profile, a toy prompt pool split
into train and held-out, and a score and a train config beside it. The checks and rewards are
placeholders, marked as such, with a TODO that states the rules a real reward must keep.
tests/examples/test_verticals.py tests each placeholder, including the unscored paths, and scores
and trains each vertical on CPU through the real wiring.
…ntity and dies with it
The same two points #613's review raised for the Q&A judge: the tool's command, timeout and script identity are now
in the check's version (cached legs are keyed by it), and the tool runs through ProcessRunner, so a timeout or the
worker's close() kills its whole process group.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Unresolved correctness issues can reject valid JSON verdicts and toy programs, while changes to PATH-resolved checkers can cause stale cached results to be reused.
Adds three standalone starter plugins for aorlta’s task-extension framework: ISA fuzzing, domain Q&A, and sanitizer triage. These complement the worked examples from #613 with copyable templates using clearly marked toy data and placeholder checks.
Changes:
Adds task prompts, parsing, checks, rewards, and optional external-tool support for the ISA starter.
Supplies separate training and held-out pools, single-GPU configurations, and adaptation instructions.
Adds tests for plugin loading, reward invariants, tool identity and cleanup, and CPU scoring/training integration.
… the ISA prompt's immediates, a tool on PATH
- sanitizer-triage found its JSON verdict with a non-greedy brace regex, which stopped at a } inside a string; the
JSON decoder now finds each object's end.
- isa-fuzzer's check limits an immediate to six digits, which its prompt did not say; the prompt now states it.
- isa-fuzzer's tool, given as a bare command, is fingerprinted as PATH finds it, so replacing it is a new version.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The answer parser’s whitespace matching crosses line boundaries, causing correct final answers to be rejected and empty final answer markers to be incorrectly accepted.
The answer pattern's \s* crossed line ends, so an empty marker took the next line as its answer and an empty last
marker fell back to an earlier one. Each line is now matched alone, and an empty last marker is no answer.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Answer normalization leaves closing formatting delimiters when followed by a full stop, causing otherwise correct answers to receive a wrong-answer reward.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Strip trailing punctuation after removing answer formatting
A full stop outside the answer's formatting leaves a closing delimiter in the normalised value: Answer: "5". normalises to 5", and Answer: **5**. to 5**. Both therefore receive wrong_answer for the first training question, although the documented normalisation removes surrounding formatting and a trailing full stop. Strip the trailing delimiters and full stop together so their order does not change the reward.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Answer normalization incorrectly rejects valid quoted or bold aliases, and the sanitizer prompt omits the rationale-length limit enforced by validation.
SYSTEM = "You triage reports from a checking tool."
USER = ("Report:\n{report}\n\nReply with one JSON object with exactly the keys \"category\" and \"rationale\": the "
"category is one of " + ", ".join(CATEGORIES) + ", and the rationale is one or two sentences.")
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three starter plugins to copy when you build a task of your own, under
aorlta/examples/verticals/:isa-fuzzer/: the model writes a short test program in a declared toy format; the placeholder check is the format's grammar, length and required instructions, with an optional external tool after that.qa-expert/: a question in your domain, answered on anAnswer:line; the placeholder check is an exact match after normalising.sanitizer-triage/: the model reads a tool's report and answers with a JSON verdict (a category from a closed list and a rationale); the placeholder check reads only the verdict's form.Each is framework only: a toy prompt pool (marked PLACEHOLDER, not real tasks or reports), a working placeholder check and reward with TODOs where the real one goes, a check profile, a score and a train config for one GPU, and a README under 30 lines. Each is its own plugin package, so a directory can be copied alone. There's no bug-finding content.
tests/examples/test_verticals.pyloads every config, tests each placeholder check and reward (the service's reward rules: range, parts, unsettled never clean), and scores and trains each starter on CPU through the real wiring. The ISA tool gets the same treatment as the Q&A judge in #613: its identity is part of the check's version, and a timeout orclose()kills its whole process group.Stacked on #613. Draft.