Repository navigation
fix(guacamole): answer 503, not a denial, when authorization is unavailable - #2991
Merged
Merged
Conversation
…ilable
When an OpenFGA check timed out, the fail-closed range and device layers
converted the evaluator outage into a plain denial. The Guacamole authority
renewal endpoint then answered {"allowed": false}, and the gateway closed
live sessions at once. On 2026-11-19 at 19:57:28 UTC this closed three
participant sessions. One minute later the same sessions were allowed.
Add AuthorizationUnavailable, a PermissionError subclass for an evaluator
outage. The engine range and device layers raise it only for
AuthorizationProviderBindingError. Every other failure keeps its bounded
denial. The CMS device layer marks the outage with the existing
"authority-unavailable" code.
The renewal endpoint now answers HTTP 503 for an outage. The gateway keeps
its lease deadline and retries, so the lease still ends a session if the
outage continues. Revalidation skips a session during an outage. Session
creation, URL delivery and the other authorize_bootstrap callers still
refuse access.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On the live event tenant on 2026-11-19 at 19:57:28 UTC, three OpenFGA checks timed out:
The provider returned
DecisionKind.EVALUATOR_ERROR. The fail-closed range and device layers converted that outage into a plain denial. The portal loggedreauthorization denied ... exc_type=PermissionErrortwice andexc_type=CMSErroronce.The Guacamole authority renewal endpoint then answered
{"allowed": false}. The gateway extension denies the lease onfalse, so it closed three live participant sessions. One minute later, the same three bootstraps returnedallowed=True.The extension already treats a non-200 answer as "unavailable". It keeps the 90-second lease deadline and retries every 15 seconds. The portal did not use that answer.
Change
AuthorizationUnavailableinshared/authorization/port.py. It is aPermissionErrorsubclass, so every existing caller still denies access.engine/services/_range_authority.py:require_native_range_actionraises it only forAuthorizationProviderBindingError.authorized_instance_rangepasses it through.PermissionError.engine/services/_device_authority.py:admit_native_device_accessraises it only forAuthorizationProviderBindingError.cms/services/_scoped_devices.py:admit_device_remote_authoritymarks an evaluator outage with the existing{"code": "authority-unavailable"}CMS code. Every other failure keeps the boundedCMSError.require_range_action), so it does not change.mission_control/guacamole_authority.py:AuthorizationUnavailablefor an outage.authorize_bootstrapkeeps its contract and returnsFalsefor an outage, so session creation, URL delivery and the status poll still refuse access.mission_control/api/guacamole_control.py: the authority view answers HTTP 503 with{"error": "gateway.authority_unavailable"}for an outage.docs/technical/shifter_platform/content-range-authority.md: document the outage behavior.The lease still ends a session if the outage continues past the lease deadline.
Live status
This change is not deployed on the live event tenant.
Verification
tests/mission_control/test_guacamole_authority_outage.py:{"allowed": false}.authorize_bootstrapstill denies.test_principal_guacamole.py,test_guacamole_control.py,test_guacamole_revocation_events.py,test_principal_devices.py,test_principal_range_authority.py,test_temporary_range_access.py), 115 tests passed.lint-imports(8 contracts kept) pass.