Skip to content

chore(gcp): match the repo to the live event settings - #2988

Merged
Brad-Edwards merged 2 commits into
devfrom
chore/live-event-settings
Nov 19, 2026
Merged

Brad-Edwards merged 2 commits into
devfrom
chore/live-event-settings

Conversation

@Brad-Edwards

@Brad-Edwards Brad-Edwards commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

These values run on the live event tenant since 2026-11-19. They kept portal logins and Guacamole sessions available under event load. This PR puts the same values in the repo.

Live setting Value File
Portal liveness probe TCP check on port http (initial delay 60 s, period 30 s, timeout 5 s, 3 failures) platform/charts/shifter/templates/web-deployment.yaml
Portal readiness probe /health/, timeout 5 s (was 1 s) platform/charts/shifter/templates/web-deployment.yaml
Portal BackendConfig health check unhealthyThreshold 5 (was 2) platform/charts/shifter/values.yaml
Guacamole API_SESSION_TIMEOUT 15 minutes (was 480) platform/charts/shifter/templates/guacamole-client-deployment.yaml
p300 portal autoscaling min 68, max 100 (was 38, 76) shifter/installation/capacity_profiles_gcp.py
p300 portal web workers 6 (was 4) shifter/installation/capacity_profiles_gcp.py
p300 guacd autoscaling min 48, max 72 (was 24, 48) shifter/installation/capacity_profiles_gcp.py
RDP graphics pipeline "disable-gfx": "true" shifter/shifter_platform/mission_control/_guacamole_connection_params.py
p300 OpenFGA runtime replicas (live 20:03 UTC) 4 (was 2) shifter/installation/capacity_profiles_gcp.py
p300 portal OPENFGA_TIMEOUT_MS (live 21:00 UTC) 5000 (was the default 1500) shifter/installation/capacity_profiles_gcp.py, shifter/installation/runtime_inventory_gcp.py

Notes

  • The p300 portal and guacd replicas values also change to 68 and 48. The profile invariant requires replicas to equal the autoscaling minimum.
  • The SQL and Redis connection budgets still cover the new p300 portal contexts. No budget changed.
  • The access-node bounds for p300 did not change. The CPU-limit headroom note in capacity_profiles_gcp.py does not apply to p300 now, and a comment says so.
  • The GCP render baselines in test_chart_contract.py are regenerated. The render diff contains only the probe, BackendConfig and API_SESSION_TIMEOUT changes.
  • docs/technical/architecture.md now states the 15-minute Guacamole session limit.
  • OpenFGA: checks slower than 1.5 s failed closed, and the Guacamole authority renewal ended live sessions at 19:57 and 20:42 UTC. The OpenFGA pods used about 0.24 CPU. The capacity profile has a new openfga section. p300 gets 4 replicas and 5000 ms. Other tiers keep 2 replicas and 1500 ms.
  • The profile emits openfga.replicas. The bootstrap merges it into the Helm values with _merge_capacity_values, which keeps the identity keys from gcp_native_identity.py. The timeout goes through runtimeEnv into the platform-runtime ConfigMap. OPENFGA_TIMEOUT_MS is now a capacity runtime key, and the published contract is regenerated.

Verification

  • shifter/installation: test_capacity_profiles_gcp.py, test_runtime_inventory.py and test_contract.py pass.
  • platform/charts/shifter/tests/test_chart_contract.py and test_openfga_rollout.py pass. helm lint passes.
  • tests/mission_control/test_guacamole_identity.py passes, with a new test for disable-gfx.
  • ruff check and ruff format --check pass on the changed Python files.
  • bigrae-config contract check passes. test_gcp_bundle.py passes, with the new tests for the p300 OpenFGA values and the defaults of the other tiers.

On 2026-11-19 these values were set on the live event tenant to keep portal
logins and Guacamole sessions available under event load. This change puts
the same values in the chart, the GCP capacity profile and the RDP settings.

- Portal liveness uses a TCP check. Readiness keeps /health/ with a 5 s
  timeout.
- The portal BackendConfig needs 5 failed health checks in a row, not 2.
- Guacamole frees a session with no open connection after 15 minutes.
- p300 portal: 68 to 100 replicas and 6 web workers.
- p300 guacd: 48 to 72 replicas.
- RDP sessions do not use the graphics pipeline, so they use 16-bit color.
On 2026-11-19 at p300 load, OpenFGA checks that took longer than 1.5 s
failed closed. The Guacamole authority renewal then answered "denied" and
ended live sessions at 19:57 and 20:42 UTC. The OpenFGA pods used about
0.24 CPU, so load was not the cause.

These values were set on the live event tenant:

- OpenFGA runtime server replicas: 4 (was 2), at 20:03 UTC.
- Portal OPENFGA_TIMEOUT_MS: 5000 (was the default 1500), at 21:00 UTC.

Add an openfga section to the GCP capacity profile. p300 gets 4 replicas
and 5000 ms. Other tiers keep 2 replicas and 1500 ms. The profile emits the
replicas as openfga.replicas, which the bootstrap merges into the Helm
values without replacing the identity keys. It emits the timeout through
runtimeEnv into the platform-runtime ConfigMap. desired_state() lists both.
Add OPENFGA_TIMEOUT_MS to the capacity runtime keys and regenerate the
published contract.
@Brad-Edwards
Brad-Edwards merged commit eeccd10 into dev Nov 19, 2026
99 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant