Skip to content

fix(guacamole): answer 503, not a denial, when authorization is unavailable - #2991

Merged
Brad-Edwards merged 2 commits into
devfrom
fix/guac-renewal-transient-authz
Nov 19, 2026
Merged

Brad-Edwards merged 2 commits into
devfrom
fix/guac-renewal-transient-authz

Conversation

@Brad-Edwards

Copy link
Copy Markdown
Collaborator

Problem

On the live event tenant on 2026-11-19 at 19:57:28 UTC, three OpenFGA checks timed out:

OpenFGA check failed closed: exc_type=ReadTimeoutError

The provider returned DecisionKind.EVALUATOR_ERROR. The fail-closed range and device layers converted that outage into a plain denial. The portal logged reauthorization denied ... exc_type=PermissionError twice and exc_type=CMSError once.

The Guacamole authority renewal endpoint then answered {"allowed": false}. The gateway extension denies the lease on false, so it closed three live participant sessions. One minute later, the same three bootstraps returned allowed=True.

The extension already treats a non-200 answer as "unavailable". It keeps the 90-second lease deadline and retries every 15 seconds. The portal did not use that answer.

Change

  • Add AuthorizationUnavailable in shared/authorization/port.py. It is a PermissionError subclass, so every existing caller still denies access.
  • engine/services/_range_authority.py:
    • require_native_range_action raises it only for AuthorizationProviderBindingError.
    • authorized_instance_range passes it through.
    • Every other failure keeps its bounded PermissionError.
  • engine/services/_device_authority.py: admit_native_device_access raises it only for AuthorizationProviderBindingError.
  • cms/services/_scoped_devices.py: admit_device_remote_authority marks an evaluator outage with the existing {"code": "authority-unavailable"} CMS code. Every other failure keeps the bounded CMSError.
  • The CMS range layer already uses that code (require_range_action), so it does not change.
  • mission_control/guacamole_authority.py:
    • The renewal path raises AuthorizationUnavailable for an outage.
    • Revalidation skips the session until the next cycle.
    • authorize_bootstrap keeps its contract and returns False for an outage, so session creation, URL delivery and the status poll still refuse access.
  • mission_control/api/guacamole_control.py: the authority view answers HTTP 503 with {"error": "gateway.authority_unavailable"} for an outage.
  • docs/technical/shifter_platform/content-range-authority.md: document the outage behavior.

The lease still ends a session if the outage continues past the lease deadline.

Live status

This change is not deployed on the live event tenant.

Verification

  • New tests in tests/mission_control/test_guacamole_authority_outage.py:
    • An evaluator outage during renewal gives HTTP 503 from the authority view.
    • A policy denial still gives {"allowed": false}.
    • Revalidation does not close a native session during an outage.
    • Session creation refuses during an outage, and authorize_bootstrap still denies.
    • Each changed layer carries only the evaluator outage. A policy denial or another failure keeps its previous bounded denial.
  • On PostgreSQL 16: the new file passes (12 tests). With related existing suites (test_principal_guacamole.py, test_guacamole_control.py, test_guacamole_revocation_events.py, test_principal_devices.py, test_principal_range_authority.py, test_temporary_range_access.py), 115 tests passed.
  • Without the source change, 6 of the new tests fail.
  • ruff, ruff format, mypy on the changed files, and lint-imports (8 contracts kept) pass.

…ilable

When an OpenFGA check timed out, the fail-closed range and device layers
converted the evaluator outage into a plain denial. The Guacamole authority
renewal endpoint then answered {"allowed": false}, and the gateway closed
live sessions at once. On 2026-11-19 at 19:57:28 UTC this closed three
participant sessions. One minute later the same sessions were allowed.

Add AuthorizationUnavailable, a PermissionError subclass for an evaluator
outage. The engine range and device layers raise it only for
AuthorizationProviderBindingError. Every other failure keeps its bounded
denial. The CMS device layer marks the outage with the existing
"authority-unavailable" code.

The renewal endpoint now answers HTTP 503 for an outage. The gateway keeps
its lease deadline and retries, so the lease still ends a session if the
outage continues. Revalidation skips a session during an outage. Session
creation, URL delivery and the other authorize_bootstrap callers still
refuse access.
@Brad-Edwards
Brad-Edwards merged commit 423e723 into dev Nov 19, 2026
104 checks passed
@Brad-Edwards
Brad-Edwards deleted the fix/guac-renewal-transient-authz branch November 19, 2026 21:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant