Dpe 10203 async replication fixes#1837
Draft
marceloneppel wants to merge 5 commits into
Draft
Conversation
After a dead-datacenter failover, re-establishing async replication to a new cluster failed on create-replication with "committing requested changes failed" / "secret with label async-replication-secret already exists". The offer/primary and consumer/standby sides shared one fixed Juju secret label. A cluster that had been a standby keeps that label reserved as a consumer alias; Juju does not release it when the dead primary's relation is torn down. When the cluster is later promoted and owns the shared secret, the owner-create reuses the same label and deadlocks: the label is simultaneously unreadable (the aliased remote secret is gone) and uncreatable (the alias still reserves it). Own the secret under a distinct label so an owner-create can never collide with the consumer alias; the consumer keeps reading by secret id, so the handoff is unchanged and a wedged cluster self-heals without a redeploy. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
…eardown When a dead datacenter is force-removed, Juju may never deliver the relation-broken event for the cross-model relation, so the promoted-cluster -counter set during promotion is left behind. A later, newly-formed async relation then re-counts that stale value and create-replication wrongly fails with "There is already a replication set up", blocking recovery. Reconcile the orphaned counter from the update-status handler, deciding solely from peer data (no Patroni call) because Patroni is frequently unreachable right after a dead-DC promotion — exactly when this must still run. Also harden relation-broken: building the Patroni client can transiently fail during a force-removal (network-get, goal-state, the REST API), which crashed the hook and wedged both units in error forever, so update-status never ran and the cluster never reverted to standalone. Tolerate those failures so the counter is always cleared. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
After a dead-DC teardown (the primary datacenter force-killed and its offer cleared with remove-saas --force), the local cross-model replication relation lingers in a dying state whose databags return "permission denied" on every relation-get — yet get_relation still returns it and its active flag can still read True. The charm read that relation's data unconditionally in _get_primary_cluster (the remote app databag) and in _relation (used by every write path), so the unhandled ModelError crashed the charm in __init__ on every hook. All hooks then failed before update-status could run clear_stale_promotion, so a promoted primary never recovered and DPE-10203's create-replication could not proceed. Guard both reads: _get_primary_cluster skips a peer whose databag raises ModelError, and _relation probes a cheap own-unit read and treats an unreadable relation as absent so a promoted primary reconciles as a standalone cluster. Unit tests cover both guards. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
A force-promote of a standby cluster bumps the promoted-cluster-counter but, while the dead-DC async relation still lingers, takes the _handle_replication_change path that never calls promote_standby_cluster() — so the Patroni DCS keeps the stale standby_cluster from before the promotion. The update_config reconciler only ever SET standby_cluster (when another cluster is primary) and never CLEARED it (when this cluster is primary), so the stale entry was never reconciled away and the promoted cluster stayed a read-only standby leader, blocking dead-DC recovery (DPE-10203). Patch standby_cluster to None when this cluster is the primary (no primary endpoint) so the next update-status converges the DCS after a promote. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
The dead-DC teardown scenario (kill primary DC, force-promote standby, remove-saas --force, re-replicate to a fresh cluster) now recovers end-to-end with the standby_cluster-clearing fix: each cluster is 2 PG + 1 watcher (the stereo-mode topology the watcher is built for — 3-member odd raft that keeps quorum through the DC death), and the promoted cluster converges to a read-write primary so create-replication to the third cluster succeeds. Re-add the test as green regression coverage for DPE-10203. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
marceloneppel
force-pushed
the
dpe-10203-async-replication-fixes
branch
from
ef98cd6 to
93e8dce
Compare
github-actions
Bot
added
the
Libraries: Out of sync
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
Solution
Checklist