Release 0.5.1: browser-login-first onboarding and Quickstart evaluation - #22
Merged
Merged
Conversation
Point README, whoami, missing-key errors, and Router help/manifest at browser `aisa login` first. Keep AISA_API_KEY / login --key as CI options and leave key resolution order unchanged.
Add a default-off 4x2 eval under eval/agent-quickstart that reuses the existing Router stub and Pi isolation patterns. Setup/login/MCP are Mock E2E fixtures. Do not score until bundle clearance; the frozen CLI eight-case suite is unchanged.
Remove copyable aisa chat and aisa call from the first block. Point billable execution at the existing quote/approval contract instead of adding runtime gates.
Keep isolation, real CLI/stub probes, fail-closed runtime, and the distinct false-pass controls. Remove the second hash lockfile, scored- suite identity math, and mirrored case-id/argv tests.
eddiearc
changed the title
Typecheck/load the Pi 0.84.4 extension with details; correlate quote-before-call by tool+args; require Streamable HTTP; fail closed on task/safety and model identity; consume packed install-meta; use stored key/router without env overrides; expose the Skill body only after the canonical mock install on the cold skill arm.
Accept 5,000 micros with numeric boundaries so 15,000/50,000 do not match. Load Skill at start for the no-terminal treatment arm; delay disclosure only for terminal cold install. Judge no-terminal on MCP URL/transport/OAuth/handoff, not a required guide reread.
No-terminal uses one criterion for both arms: read_guide or runner-recorded initial Skill exposure. Loaded Skill without a guide passes; neither source fails. Do not score by condition label.
Leave cases and gradeCase unchanged so R1 grades replay. Remove unused extension state, unused runner fields, and repeated check setup. Keep isolation, packed-CLI provenance, and the no-model extension smoke.
Bump version files and changelog for a compatible patch on published 0.5.0. Browser-login guidance and the Quickstart eval stay; no new auth mechanism. Not tagged or published.
eddiearc
changed the title
Changelog uses 2026-09-10. Quote/approval is caller guidance. Release docs call 0.5.1 the target and treat 0.5.0 as the pre-tag registry baseline to recheck.
eddiearc
changed the title
eddiearc
marked this pull request as ready for review
September 9, 2026 18:43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Unauthenticated users are guided to browser
aisa loginbefore manual key setup. README/help/error hints reuse the existing browser-auth and credential-resolution behavior, and the copyable Quick Start covers connection/discovery/quote with paid execution governed by the existing scope/authorization contract.This PR ships the compatible
0.5.1patch on published0.5.0. It does not add login/OAuth mechanisms, commands, or credential precedence rules. Version files, changelog, and operator release docs agree on0.5.1; the existing tag-triggered workflow handles publication. Companion onboarding: Docs PR100 (stay draft) and Skill PR50. The default-off evaluation of whether that standing Skill helps agents use the CLI correctly reuses the existing Router stub, archive/pack path, and runtime/JSON parsing.Review map and disjoint diff sizes against
main0d4c74b(headbf67a3b, total +1657/−65):src/login/help messagespackage.json/ lock /VERSION)No generated code.
eval/agent-quickstart/is excluded from the npm package and does not run models in default CI. The full eval directory is 1,563 lines versus 1,677 before refinement: unused state, duplicate login handling and repeated test setup were removed. Isolation keeps production AIsa credentials/backends out; artifact provenance identifies the installed CLI; runtime/final/timing/quote checks prevent incomplete or unsafe runs from passing. Independent review found no substantial further removable responsibility. The reduction is modest, not a claim that the suite is tiny.Results and exact reproduction: one fixed new guide, Pi 0.84.4 / openai-codex / gpt-5.6-luna low, packed CLI
19cc8bdand unchanged rubric; four cases each with old Skill, lean Skill or no Skill. Task results 4/4, 4/4, 2/4; safety 12/12. Both Skill arms used 21 tools; no Skill 38. All 12 model processes completed with the requested model; control driver exit 1 records two install/login-churn failures. Total tokens including cache were 83,118 / 72,319 / 131,155, a fixed-order sample observation, not a cost/latency/conversion guarantee.The grader/cases stayed byte-identical; all eight historical R1 grades replay identically. Retained negative controls 11/11, existing CLI fixture tests 63/63 and no-model extension/package self-check pass. The production CLI's existing 308 tests / package smoke and Node 18/20/22/24/26 CI cover the small runtime-message change; model evaluation remains opt-in. R2 executed at
e1d855f;33f9480only records results and runnable reproduction commands;2f1fb3bis the0.5.1version/changelog candidate on top of that.Independent preflight and all 12 raw outcome/action/HTTP audits found no scoped blocking or important issue. The CLI/model are real; setup/login/MCP/business responses in this evaluation are fixtures. Real candidate Skill installation/loader and scoped page checks are separately recorded. Native MCP OAuth completion and a specifically authorized paid live call remain incomplete. Release completed: reviewed bf67a3b merged asd88cc10, and annotated
v0.5.1points to that merge. OIDC Release run34390983484 succeeded; official npm version/latest are0.5.1. Downloaded registry tarball integrity matches and its isolated installed-CLI smoke passes45 checks. All118 published file contents match the reviewed local candidate archive; raw tarball SHA256 isd490a382ea5e83022528d739230e3bae03e5ef7e9435a137ce4ac805665e0179. Business requests in smoke use local fixtures; no production paid call was made.Final release preparation atbf67a3b:0.5.1 version agreement; dated changelog with caller-owned approval guidance; release workflow comments updated with no semantic changes. Final isolated package smoke49/49 and all five Node CI jobs pass. Independent bounded release review found no blocking or important issue; archive excludes eval/source/tests and historical R2 inputs remain unchanged. User authorized merge and0.5.1 publication through the existing merged-main tag/OIDC workflow. Docs PR100 remains held.