Repository navigation
Add gpt-5.6-luna to benchmark; bump pi-ai to 0.80; reviewerReasoning setting - #32
Merged
Merged
Conversation
- Bump @earendil-works/pi-ai + pi-coding-agent + pi-tui to ^0.80.6 so gpt-5.6-luna resolves in the reviewer model registry. - Update reviewer + digest imports to @earendil-works/pi-ai/compat per the 0.80 release notes (compat is a strict superset of the removed root exports; extension loader aliases root->compat at runtime, but scripts running via tsx bypass that alias). - Add PI_AUTO_REVIEWER_REASONING env override. Motivated by luna returning empty responses at reasoning='minimal' (its thinkingLevelMap only exposes off/xhigh/max; OpenAI's endpoint accepts the request and returns 0 tokens instead of 4xx-ing). - benchmark-models.ts: add gpt-5.6-luna rows at low + off, plumb the reasoning field on ModelSpec via PI_AUTO_REVIEWER_REASONING, fix the TOTAL-line parser (was matching the short digest-summarizer TOTAL instead of the full reviewer-scenarios one, showing 0 avg_ms / $0 cost), and strip ANSI escapes from vitest output so pass counts parse correctly when running under a TTY. - HISTORY: document the env override, the empty-at-minimal finding, and the 1x 3-way comparison (mini @ minimal vs luna @ low vs luna @ off). Luna @ off matches mini on accuracy at 1x but is ~12x more expensive; rejected as reviewer default.
Files were sitting untracked in the working tree. .nvmrc pins node 22 (matches the pi-coding-agent 0.80 engine requirement of >=22.19.0). pnpm-workspace.yaml + pnpm-lock.yaml round out the pnpm setup that the npm-based package-lock.json already exists alongside.
The previous commit added PI_AUTO_REVIEWER_REASONING as an ad-hoc
process.env read in reviewer.ts. settings-store.ts warns against that:
env-var overrides must be registered in the canonical ENV_VAR_OVERRIDES
table so the layered loader tracks provenance and no other module reads
env directly.
- types.ts: add ReviewerReasoningLevel union
("auto"|off|minimal|low|medium|high|xhigh|max) and reviewerReasoning
field on PiAutoSettings.
- pi-auto.ts: default reviewerReasoning to "auto", which preserves the
historical hardcoded behavior ("low" for codex-auto-review, "minimal"
otherwise). Any non-"auto" value overrides both paths.
- settings-store.ts: register PI_AUTO_REVIEWER_REASONING in
ENV_VAR_OVERRIDES with a whitelist. Unknown values are ignored (lower
layers win), matching the existing PI_AUTO_USE_CODEX_POLICY pattern.
- settings-ui.ts: enum entry so /pi-auto-settings shows and manages it.
- reviewer.ts: read from settings.reviewerReasoning, cast "off" to
pi-ai's ThinkingLevel type (pi-ai's runtime handles "off" fine in
openai-responses.js:131 but its type excludes it).
- tests/settings-store.test.ts: update env-var registry assertion (this
is the test the settings-store comment tells you to update), plus two
new tests exercising the override + unrecognized-value paths.
- HISTORY.md: reword the sub-section to reflect the promotion.
At 1x the accuracy comparison was noisy; the recorded 91/93 (luna) vs 89/93 (mini) inverted at 5x. Real numbers: gpt-5-mini @ minimal: 467/483 (96.7%), 1136 ms avg, $0.141 gpt-5.6-luna @ off: 459/483 (95.0%), 860 ms avg, $0.472 Small but consistent ~1.7pp accuracy regression for luna, ~24% faster, ~3.35x more expensive (down from ~12x at 1x because prompt-cache reads dominate at 5x and cancel mini's 4x input-price advantage). Verdict unchanged: rejected as reviewer default. Setting remains available.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
gpt-5.6-lunato the model benchmark, upgrades pi-ai to0.80, and promotes reviewer reasoning-effort to a real setting.
Contents
1. Bump
@earendil-works/pi-ai+pi-coding-agent+pi-tuito^0.80.6.gpt-5.6-lunaisn't in pi-ai 0.74.2's registry. Extensions no longerneed to know: pi's extension loader aliases the pi-ai root to
/compat(a strict superset of the removed root exports) per the0.80 release notes. This repo's
scripts (
scripts/benchmark-models.ts) run undertsxand bypass thatalias, so
extensions/reviewer.tsandextensions/digest.tsimportfrom
@earendil-works/pi-ai/compatexplicitly.2. New
reviewerReasoningsetting.ReviewerReasoningLevel = "auto" | "off" | "minimal" | "low" | "medium" | "high" | "xhigh" | "max". Default"auto"resolves to"low"forcodex-auto-review(per its fine-tuning) and"minimal"otherwise,preserving prior behavior.
PI_AUTO_REVIEWER_REASONINGis registeredin the canonical
ENV_VAR_OVERRIDEStable insettings-store.ts(notread ad-hoc in
reviewer.ts).Motivating case:
gpt-5.6-lunareturns empty responses atreasoning: "minimal". ItsthinkingLevelMapin pi-ai only exposesoff/xhigh/max. OpenAI's endpoint accepts the request and returns 0tokens instead of 4xx-ing, so without the setting the reviewer would
be silently broken for luna.
3. Benchmark additions and parser fixes.
gpt-5.6-luna (low)andgpt-5.6-luna (off)rows inscripts/benchmark-models.ts, plus areasoningfield onModelSpec.TOTAL instead of the reviewer-scenarios one, showing 0 avg_ms / $0
cost).
counts survive TTY-detected color output.
4. HISTORY entry documenting the empty-at-
minimalfinding, the1× 3-way comparison table, and the "rejected on cost" verdict for
luna as reviewer default.
5. Untracked cleanup.
.nvmrc(Node 22, matchespi-coding-agent@0.80engine requirement),pnpm-lock.yaml,pnpm-workspace.yaml.Measured (5× live suite;
luna @ lowdropped after 1× showed it strictly worse thanoff)gpt-5-mini@minimal(baseline)gpt-5.6-luna@offAt 5× the accuracy delta is small but consistent — ~1.7 pp regression for
luna, not "within noise". Luna is ~24% faster and ~3.35× more expensive
per suite. Cost ratio shrank vs 1× (~12×) because prompt-cache reads
dominate at 5× (both models: cacheRead = 10% of input), largely
cancelling mini's 4× input-price advantage; remaining cost is driven by
luna's 3× output price. Rejected as reviewer default on the combo of
measurable accuracy regression + higher cost. Setting remains available
for users who want the latency win.