test: make the smoke run's narrowing check able to fail - #35
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
This PR updates the live login smoke run so its scope-narrowing verification can actually disagree with a misbehaving deployment, by performing two brokered acquisitions: one “broad” (prove access derivation works at all) and one “narrowed” (strict subset, so narrowing can meaningfully fail). It also updates the smoke-run documentation and the implementation plan to reflect the new evidence and definition-of-done completion.
Changes:
- Update
TestLoginSmoketo acquire twice (broad + narrowed) using separateauth.Brokerinstances, and add anacquirehelper to standardize smoke output. - Expand
test/smoke/RUNNING.mdto explain why the second acquisition is the meaningful narrowing measurement and how to interpret refusals. - Mark the definition-of-done item as completed in
docs/plans/login-first-slice.md, with a short summary of the new empirical evidence.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
test/smoke/RUNNING.md |
Documents the new two-acquisition smoke run and clarifies how to interpret narrowing-related outcomes. |
test/smoke/login_smoke_test.go |
Implements the two-step acquisition (broad + narrowed) so the narrowing check can fail when a backend ignores narrowing. |
docs/plans/login-first-slice.md |
Updates plan status/definition-of-done text to reflect the now-non-vacuous live narrowing proof. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The live run asked the broker for config.Scopes — every permission the session already carries — so internal/auth/narrowing.go compared the issued scopes against an identical request. That comparison holds however the deployment behaved. A backend that disregarded the narrowing request entirely would still have been reported as granted, and the definition of done asks for a backend *proven* to satisfy the scope policy. The audience half of the same verification was real and had caught a live misconfiguration. The scope half could not fail. config.NarrowTarget() had existed for this since the package was written and nothing called it. So the run now acquires twice. The first request is the broad one and proves the broker can derive access at all, which is what a first run against a new deployment needs to know. The second asks for one permission out of the several the session holds — the request a module actually makes, and the only form the verification can disagree with. A refusal on the broad request stops the run rather than repeating one finding, because a deployment that will not issue against this audience at all says nothing about narrowing. Reusing one broker for both refused with auth.already_granted: a module gets one acquisition per command. Two invocations is not a way around that rule but the accurate model of it — two commands, in turn, against one session. Against a rotating deployment the second acquisition now also proves the first persisted its replacement, since it can only reach the token endpoint with what the first one stored. Measured against wso2/wso2is:7.3.0: 1261 characters carrying both permissions, then 1230 carrying one. Two tokens the previous single-acquisition form could not have told apart. The module half of the same definition-of-done clause needs nothing further. A module never learns which issuer minted its token, so running it against a live deployment proves nothing TestLoginThenTheModuleReceivesIssuerMintedNarrowedAccess does not already prove against the fake issuer, where the token is introspected and rotation is pinned across three runs. Claude-Session: https://claude.ai/code/session_01RojiAgW9hi3b9f9G6ZXBVp
0959a52 to
e8aa8e4
Compare
Review found the smoke run's refusal example truncated. The run prints the shell's own message on an indented second line, and the example stopped before it — while the paragraph beside it said that message is what distinguishes one cause from another. A reader following that advice found nothing to follow it with. The example now carries the indented line, and the four other messages are tabulated beside it. The summary sentence above them is deliberately identical for all five, because auth.narrowing_unavailable covers five distinct causes and a summary naming one would be wrong four times out of five. Only the indented line is specific, which is precisely why it had to be in the example. Two corrections in section 8 of the walkthrough, which the new cross-reference sends readers to and which contradicted what this branch's parent measured. The audience row still said the value to configure is the API resource identifier. That is right on Identity Server and wrong on Asgardeo, where the only value ever placed in `aud` is the client ID — so a reader hitting an audience refusal on Asgardeo was being told to set the one value that cannot work. It now names both products and points at the sections that measured them. The paragraph under the table still said whether Asgardeo narrows would be recorded "once measured against a live tenant". It was measured on 2026-08-06, on both products, and both narrow. Claude-Session: https://claude.ai/code/session_01RojiAgW9hi3b9f9G6ZXBVp
Follow-up to #34, stacked on it. Closes the last open definition-of-done box in
docs/plans/login-first-slice.md.The problem
The live smoke run's narrowing check could not fail.
login_smoke_test.goacquired once, asking forconfig.Scopes— every permission the session already carries.internal/auth/narrowing.goverifies by comparing the issued scopes against the requested ones, so when the request is the whole session, that comparison is a set against itself. It holds however the deployment behaved. Point the run at a backend that disregards a narrowing request entirely and it would still printgranted.The audience half of that verification was real and did catch things — an Asgardeo run refused on it. The scope half was theatre. Which matters, because the definition of done reads "against a backend proven to satisfy the scope/audience policy", and half of that proof was vacuous.
config.NarrowTarget()had existed for exactly this since the smoke package was written, and nothing called it.What this does
Acquires twice, because the two acquisitions fail for different reasons:
The first proves the broker can derive access at all, which is what a first run against a new deployment needs to know. The second is a strict subset — the request a module actually makes — and is the only form the verification can disagree with.
A refusal on the broad request stops the run rather than repeating one finding: a deployment that will not issue against this audience at all says nothing about narrowing. A refusal on the narrowed request is a real statement about the deployment, and still passes, because the shell declining to hand a module more authority than it asked for is the designed outcome.
One thing the change surfaced
The first attempt reused a single
auth.Brokerand failed:The shell allows a module one acquisition per command. So the two acquisitions run as two invocations — which is not a way around the rule but the accurate model of it: two commands, in turn, against one session, the way the shell is actually used. Against a rotating deployment the second acquisition now also proves the first persisted its replacement refresh token, since it can only reach the token endpoint with what the first one stored.
Not changed, and why
The same definition-of-done clause says the reference module receives the token. That half needs nothing further.
TestLoginThenTheModuleReceivesIssuerMintedNarrowedAccessalready launches the real module subprocess, has the issuer introspect what it presented, and pins refresh rotation across three runs. A module never learns which issuer minted its token — that is what the source seam is for — so running it against a live deployment would add a subprocess and a status service to a test whose value is about the deployment, and prove nothing the deterministic chain does not.Verification
Live run against
wso2/wso2is:7.3.0, the deployment #34 registered. The two tokens differ in length and in scope set, which is the evidence the previous single-acquisition form could not produce.docs/plans/login-first-slice.mdis now fully ticked, definition of done included.Relates to #17.
https://claude.ai/code/session_01RojiAgW9hi3b9f9G6ZXBVp