Skip to content

test(enterprise): cover the model availability round trip (#1669) - #1670

Merged
rafaelgiln merged 1 commit into
mainfrom
claude/model-availability-roundtrip
Sep 1, 2026
Merged

test(enterprise): cover the model availability round trip (#1669)#1670
rafaelgiln merged 1 commit into
mainfrom
claude/model-availability-roundtrip

Conversation

@rafaelgiln

Copy link
Copy Markdown
Collaborator

Closes #1669. The write half of the models screen, which #1660 excluded on a premise that turned
out to be my own measurement error.

1. Edit models works — #1659 was three errors, not a defect

It is a toggle: aria-pressed goes falsetrue, and each of the 44 rows swaps its state
text for a control.

view mode:   - text: Disabled
edit mode:   - switch "Show gpt-5.6-sol in model pickers"

with Enable all models, Hide all models, Cancel and Save models appearing alongside.

Models list items buttons panel text
before 1 44 6 1645 chars
after 1 44 9 1295 chars

The three errors, recorded in the doc because the same three would catch anyone probing this
screen:

  1. Looking for a dialogrole=dialog, role=menu, a URL change, all correctly zero, because
    none of them is how the control works.
  2. Reading the shrinking panel as a collapse — 1645 → 1295 is not the list disappearing; the
    item count is 44 in both states. It is 44 rows each losing the word Disabled to a switch.
  3. Counting the wrong role, twicerole=switch before entering edit mode (0, correctly, and
    the probe's own guard then meant it never clicked at all), then role=button with the toggle's
    name where the control is a role=switch.

That is the third time in this area a partial read of a screen produced a confident wrong
conclusion — after the delete button's aria-label and the provider card that "vanished" while
sitting in the next region of the same screen. #1659 is closed invalid with the same write-up.

2. Covered tests

# Test What it validates
1 entering edit mode offers one control per model and writes nothing yet One switch per model, each naming its model; enabled_model_keys untouched. A screen that wrote on entering an edit mode would be surprising in the worst way.
2 enabling a model and saving lands exactly that model in the policy Flipping stages with no request; Save models issues PUT /api/v1/model-availability-policy200; the policy holds the key of the model the switch named, not a position.
3 enabling a model does not make it visible while the provider has no credentials The row stays Pending / Hidden.
4 cancelling an edit discards it No write leaves the browser, asserted on the request.

3. Test 3 replaced the test I was about to write

The natural assertion — "the row stops reading Hidden" — is false. Measured: after saving one
enabled model the row still reads Pending / Hidden, and that is correct. Builder visibility
follows the provider's credentials, not the availability policy
: one enabled model on a provider
that is still Pending is not something a builder can pick.

Asserting the natural version would have manufactured a defect. The two are different axes and the
screen does not conflate them — the same lie #1660's read tests catch, arriving from the other
direction.

4. Force-fail

Mutation Result
edit mode never entered red (4 of them)
the saved key asserted as a model the test did not pick red
the row asserted Visible after enabling red
Cancel replaced by Save models red

5. Validation

langflow-enterprise:latest (2026-08-27), RBAC variant.

  • Spec alone: 8 passed (4 read + 4 write), ~29 s.
  • Whole admin-console/: 29 passed, twice, ~1.3 min.
  • Instance left clean: enabled_model_keys: [], no governance records.

One advisory, noted and deliberately not filed. 503 /api/v1/session"Authorization service
unavailable"
— appeared once across the two directory runs. Not reproducible: 76 polls of that
endpoint during four consecutive policy writes all answered 200, and the same shape has been
observed once before on this image without reproducing. The fixture logs it as advisory and does not
fail, which is the right handling for a single unreproducible transient — filing it would send
someone after a phantom, and hiding it would be worse.

6. Notes for the reviewer

🤖 Generated with Claude Code

The models screen's WRITE half, excluded from #1660 on the strength of #1659 —
which reported `Edit models` as an enabled control that opened nothing. That
report was three measurement errors and is closed invalid.

`Edit models` is a TOGGLE: `aria-pressed` goes false -> true, and each of the 44
rows swaps its state text for a `role="switch"` named after its model, alongside
`Enable all models`, `Hide all models`, `Cancel` and `Save models`. The three
errors, recorded in the doc because the same three would catch anyone probing
this screen: looking for a dialog; reading a shrinking panel (1645 -> 1295 chars)
as a collapse when the item count is 44 in both states; and counting the wrong
role twice — `switch` BEFORE entering edit mode, then `button` where the control
is a `switch`.

Four tests:

- Entering edit mode offers one switch per model and writes NOTHING yet. A screen
  that wrote on entering an edit mode would be surprising in the worst way and
  nothing else here would notice.
- Enabling one and saving lands exactly that model's key —
  `PUT /api/v1/model-availability-policy` -> 200, `enabled_model_keys:
  ["openai::llm::<model>"]` — keyed on the model the switch NAMED rather than on
  a position.
- Enabling a model does NOT make it visible while the provider has no
  credentials: the row stays `Pending` / `Hidden`.
- `Cancel` discards, asserted on the request rather than on a state read taken
  immediately after.

The third test replaced the one I was about to write. The natural assertion —
"the row stops reading Hidden" — is FALSE, and asserting it would have
manufactured a defect. Builder visibility follows the provider's CREDENTIALS, not
the availability policy: one enabled model on a provider that is still `Pending`
is not something a builder can pick. The two are different axes and the screen
does not conflate them, which is the same lie #1660's read tests catch, arriving
from the other direction.

Four mutations: never entering edit mode, asserting a model the test did not
pick, asserting the row reads `Visible`, and replacing `Cancel` with `Save
models` all go red.

29 passed across `admin-console/` twice; the instance is left with
`enabled_model_keys: []` and no governance records.

One advisory noted and not filed: `503 /api/v1/session` "Authorization service
unavailable" appeared once across the two runs. Not reproducible — 76 polls of
that endpoint during four policy writes all answered 200 — and the same shape has
been seen once before on this image without reproducing. The fixture logs it as
advisory and does not fail, which is the right handling for a single
unreproducible transient.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@rafaelgiln
rafaelgiln merged commit ef085be into main Sep 1, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

enterprise models screen: the availability round trip is reachable and untested

1 participant