fix(garm): make application removal safe during relation teardown

[yanksyoon]

Bug description

Removing the GARM application can race with Juju relation teardown. Relation-departed/broken hooks re-enter the normal reconcile path after relation data has already disappeared, causing the removal workflow to fail before or during application teardown.

Evidence from production removal test

During a targeted removal of github-runner-garm:

• ingress-relation-departed failed with the relation snapshot warning "app" expected but not received.
• garm-configurator-relation-broken failed repeatedly with the same snapshot warning and secret_get errors while reconstructing charm state.
• postgresql-relation-broken and debug-ssh-relation-broken also observed missing relation snapshot data.
• The unit remained in error/awaiting error resolution until failed hooks were marked resolved without retry.
• A remaining cross-model debug-ssh relation had to be force-removed before the dying application disappeared.

Relevant Juju bug reference emitted by the relation libraries: https://bugs.launchpad.net/juju/+bug/1960934

Expected behavior

The remove path should not run normal relation-driven reconciliation after teardown has started. Relation-departed/broken handlers should tolerate missing snapshot data during application removal, and the dedicated on.remove cleanup should run while GARM and its database/API are still available.

Suggested direction

• Gate or short-circuit relation handlers when the application is dying/removing.
• Ensure cleanup ordering does not depend on relation data that Juju may remove first.
• Add an integration test for application removal with all current relations, including cross-model relations.
• Preserve the GARM encryption secret or otherwise define database/secret lifecycle for application recreation.

Environment

• PS7 production Kubernetes model: k8s-pfe-ps7-prod-github-runner-prod
• GARM revision: 104
• Test ticket: ISD-6082

[yanksyoon]

Additional production test findings

The removal behavior was tested against the PS7 production Kubernetes model:

• Model: k8s-pfe-ps7-prod-github-runner-prod
• GARM revision: 104
• One GARM leader unit
• GARM, configurator, and PostgreSQL were healthy before the test
• GARM inventory contained one scaleset, amdnm-edge, with zero runner instances

No active GitHub runner was interrupted by this test.

Observed removal sequence

The targeted command was:

juju remove-application github-runner-garm --no-prompt

The application entered dying, and Juju began relation teardown before the application was fully removed. Relation-departed/broken hooks ran while their relation snapshots were incomplete. The unit entered error and awaited error resolution.

The affected hooks included:

• ingress-relation-departed
• ingress-relation-broken
• garm-configurator-relation-broken
• debug-ssh-relation-departed
• other relation teardown hooks

The repeated errors included:

'app' expected but not received
'app_name' expected in snapshot but not found

The traceback showed the normal reconcile path calling CharmState.from_charm(), which attempted to read relation-backed data and secrets after Juju had already removed the relation data.

The failed hooks had to be marked resolved with juju resolved --no-retry. The cross-model debug-ssh relation remained after the unit terminated and required a targeted forced relation removal before the dying application disappeared.

The observed lifecycle was effectively:

relation teardown -> failed relation hooks -> stop -> remove

The remove hook therefore did not reliably execute while the GARM service and API were available. Because the test scaleset was already empty, this test does not prove that a populated scaleset would be drained during real application removal.

Secret and database lifecycle failure during restoration

Terraform recreated the GARM application after removal. Juju generated a new application-owned garm-secrets secret, while PostgreSQL retained the previous GARM database. GARM then failed to decrypt its existing data:

sio: decryption failed: authentication failed

The database had been encrypted using the previous GARM secret. Restoring the application therefore changed the encryption key while retaining the encrypted database.

Recovery required:

1. Resetting only the GARM database.
2. Recreating the database.
3. Granting schema privileges to the effective GARM database owner role.
4. Retrying the GARM hook.

The first database recreation also failed with:

permission denied for schema public

The effective login role was mapped to charmed_github-runner-garm_owner, so granting schema access only to the login role was insufficient. Granting USAGE and CREATE on schema public to the effective owner role allowed GARM migrations to complete.

Operational impact

• GARM was unavailable during removal and restoration.
• Multiple relations had to be recreated.
• OTEL temporarily entered a blocked state while relations settled.
• The GARM database had to be reset because the original encryption secret was lost.
• Existing GARM metadata was discarded during the reset.

Follow-up direction

The removal design should:

• avoid normal relation-driven reconciliation once application teardown has started;
• tolerate missing relation snapshots and relation-backed secrets in teardown hooks;
• provide a cleanup mechanism that runs while the GARM workload/API is still available;
• define the lifecycle of the GARM encryption secret together with the PostgreSQL database;
• add an integration test with one configured scaleset and one spawned runner, then verify runner and scaleset deletion during application removal.

[yanksyoon]

We need to think about:

• What to do when relation departed of critical dependency (e.g. postgresql) happens
• How to differentiate relation departs/teardowns from application removal

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/ISD-6441.

This message was autogenerated