test: gate live-cluster tests behind --live and announce resolved credentials

[nilsmechtel]

Running the test suite with a Hypha credential in scope silently deploys to and calls the live bioimage-io workspace on hypha.aicell.io, with nothing in the output saying so. This makes that impossible by accident and visible when it is deliberate.

tests/conftest.py calls load_dotenv() unconditionally, and find_dotenv() searches upward from tests/, so a credential arrives without anyone asking for it. The documented exclusion list (--ignore=tests/end_to_end …) does not exclude what that enables: 29 tests in the standard scope act on production — 25 in tests/apps/model-runner/, which call the live model-runner service, and 4 in tests/test_artifact_version.py, which call upload_app and delete artifacts on bioimage-io/bioengine-worker. Tree-wide the live set is 80. The only visible symptom is that the suite takes ten times longer, so a flake hunt that loops it forty times becomes forty rounds of calling production.

What changes

Live tests are deselected unless --live is passed. A test counts as live if it requests hypha_client, hypha_token or model_runner, or carries the new @pytest.mark.live. Detection is by fixture closure, so no existing test needed editing and the rule holds for tests added later against the same fixtures.

Every resolved credential is announced before the first test, with the server, the workspace each token is scoped to, and the number of live-reaching tests it enables:

====================== live cluster credentials resolved ======================
25 collected test(s) can reach the live cluster at https://hypha.aicell.io, and a credential for it resolved:
  HYPHA_TOKEN -> workspace bioimage-io
Deselected 25 test(s); pass --live to run them.
==============================================================================

The count is printed with or without --live, and restated next to the wall clock when the run ends. The flag only prevents the accidental case; once someone has opted in deliberately, the count and the runtime in the log are the only things that make an after-the-fact audit possible. Runtime is the corroborating signal — the same scope takes ~30s offline and several minutes against a cluster.

The workspace comes from the wid: scope in the Hypha JWT, decoded locally without verification; the token itself is never printed. load_dotenv() is untouched — other things depend on it, and it is no longer what decides which tests run.

Five things the first pass got wrong

The banner never fired on the default invocation. pytest.ini's addopts carries --numprocesses=1, so collection runs inside an xdist worker whose terminalreporter output is discarded and whose pytest_deselected stats never reach the controller. Measured on the standard scope with a credential resolved: zero banner lines, no deselected count anywhere. Every sub-run in the first set of tests passed -o addopts=, which is precisely why none of them caught it. Probing the channels showed a worker's print and terminalreporter are swallowed but its stderr is inherited by the controller, so the banner goes there; two tests now run under pytest.ini as shipped, skipped only where pytest-xdist is absent. Independently swept at -n 0, 1, 2, 4 and auto: exactly one banner copy at every width. Deselection itself always worked under xdist, so this was visibility, not safety — but visibility was the entire point of the announcement.

Two live files were ungated. tests/apps/cellpose/test_ui_e2e.py drives a browser at the deployed app and injects HYPHA_TOKEN into its localStorage, but uses only the page fixture. It fails at import where playwright is absent, which is why the standard command already ignores it and why nobody noticed; on a machine with playwright it ran live. tests/test_artifact_version.py is the most destructive file in the suite and its fixture asserts on the token rather than skipping, so it errors rather than skips without one — which is why a skip-reason census missed it. The fixture-name rule does catch it, but only because the file happens to name its own fixture hypha_client; rename that and the gate silently stops applying. Both now carry an explicit module-level marker, and the test asserts on the marker rather than on the deselection, so removing it fails.

A malformed token aborted the session. The claims.get("scope") lookup sat outside the try guarding the base64/JSON decode, so a payload decoding to a JSON scalar raised AttributeError out of a collection hook — which pytest turns into INTERNALERROR. It failed closed and leaked nothing, but it now returns an unreadable-token marker instead.

-m live resolved a wrong subset. Registering a live marker in pytest.ini implies -m live works, but most live tests are detected from their fixture closure rather than from a marker anyone wrote, so the hook has to attach it to them. Until it did, -m live --live returned 4 of 29 on a mixed scope — a wrong answer that looks like a complete one. The fix is one line, item.add_marker(...), and it initially shipped with no test: deleting it left the entire gate suite green. There is now a regression test, and it compares the count -m live --live selects against the count the baseline deselects rather than asserting a literal, so it survives live tests being added. A second test pins that -m live without the flag selects nothing, so naming the marker cannot become a way past the gate.

Two documented commands became silent no-ops. pytest tests/end_to_end/ -v in the maintainer skill and pytest tests/apps/model-runner/ -v -o "addopts=" in that directory's conftest docstring both exit 5 with nothing collected. Both now carry --live, and the skill says why. They are in this PR rather than pushed separately on purpose: landing them apart would put a window on main where the documentation tells agents to pass a flag that does not parse yet, or the reverse.

On "run it from a worktree"

The interim guidance in circulation — and the first version of this PR's README text — said a git worktree has no .env and is therefore safe. That is false for a worktree under <repo>/.claude/worktrees/: find_dotenv() walks up and reaches the repo root's .env. Verified directly; it resolves to /data/nmechtel/bioengine/.env. A worktree outside the repo tree does stop the walk. tests/test_artifact_version.py loads the repo-root .env by explicit path regardless of where you are. And source .env or an exported token defeats all of it.

So the README now states all three routes and leads with the only clause true in every case: no directory choice protects you from a credential you export yourself. Running inside the worker image is safe for a different reason — only the repo is bind-mounted, so the upward walk cannot escape it.

Verification

Standard suite at a2196b4 before: 525 passed, 25 skipped, 29.8s. After: 541 passed, 2 skipped, 25 deselected, 49.2s. Additive — every previously passing test still passes, the 18 new ones are the gate's own, and the 2 skips are the xdist-dependent tests under a runner that installs a narrower package set than requirements-test.txt. The 25 deselected is that runner's scope, which excludes tests/test_artifact_version.py; in the full standard scope it is 29, and tree-wide 80.

Sixteen mutations, each caught by a named test:

mutation	outcome
never deselect	3 tests fail
deselect even with --live	test_live_tests_are_selected_with_the_flag
banner never printed	4 tests fail
banner printed with no credential	test_nothing_is_announced_when_no_credential_resolves
no fixture counts as live	4 tests fail
workspace never decoded	4 tests fail
BIOIMAGE_IO_TOKEN not watched	2 tests fail
xdist worker not special-cased	test_the_banner_survives_the_default_addopts
wall-clock summary suppressed	test_the_wall_clock_summary_states_the_exposure_even_with_the_flag
scope lookup back outside the try	test_a_malformed_token_does_not_abort_the_session
playwright marker removed	test_the_modules_a_marker_alone_can_gate_carry_one[…test_ui_e2e.py]
artifact-version marker removed	test_the_modules_a_marker_alone_can_gate_carry_one[…test_artifact_version.py]
marker attachment removed	test_m_live_selects_exactly_what_the_gate_deselects
tryfirst → trylast	test_m_live_selects_exactly_what_the_gate_deselects
-m live made a way past the gate	test_m_live_without_the_flag_selects_nothing

Two mutations survived and both are recorded rather than papered over. Removing tryfirst entirely changes nothing on this machine — a conftest hookimpl already runs before pytest's mark plugin under pluggy's reverse-registration order — so the decorator is a guarantee against a future plugin registering an earlier modifyitems, not the fix; trylast does break the invariant, which is why the ordering is worth stating. Removing the gw0 de-duplication in the xdist emitter also survives; it is cosmetic, and the -n sweep above covers it.

Two earlier mutations survived during development and were what prompted the tests above: the artifact-version marker (caught by the fixture-name rule anyway, so the test was asserting the outcome rather than its cause) and the marker attachment.

Sub-runs use synthetic JWTs and either deselect everything or pass --collect-only, so nothing here contacts a cluster.

🤖 Generated with Claude Code