Monitor cross-dc-relation (Feature?)

[marcusboden]

When you have cross-site replication, you only notice juju issues when you try to switch the sites.
I had the situation where the cross-site-replication was fine but juju had issues with the cross-site relation. It only showed up when I tried to to the switchover. Which is usually a bit too late.

Steps to reproduce

1. Deploy two postgresql
2. setup cross site replication
3. have a juju issue (e.g. user accidentally disabled)

Expected behavior

The charm realizes that it cannot reach the other juju controller anymore and warns the operator about the issue.

Actual behavior

Nothing happens, but if you try to switchover the sites, it will not work and the consumer site may go into error states. Once that happens, you also see the CMR being in failed state.

Versions

Operating system: Ubuntu 24.04.4

Juju CLI: 3.5.7

Juju agent: 3.6.12

Charm revision: 1158

LXD: 5.0.6-7fc3b36

Log output

controller-0: 08:12:58 INFO juju.worker.remoterelations cmr start "hdev-swarm-a-postgresql-replication-offer"                                                                                                                                                  
controller-0: 08:12:58 ERROR juju.worker.remoterelations cmr error in remote application worker for hdev-swarm-a-postgresql-replication-offer: watching status for offer: cannot get discharge from "https://10.33.236.139:17070/offeraccess": third party refu
sed discharge: cannot discharge: permission denied                                                                                                                                                                                                             
controller-0: 08:12:58 INFO juju.worker.remoterelations cmr stopped "hdev-swarm-a-postgresql-replication-offer", err: watching status for offer: cannot get discharge from "https://10.33.236.139:17070/offeraccess": third party refused discharge: cannot dis
charge: permission denied                                                                                                                                                                                                                                      
controller-0: 08:12:58 INFO juju.worker.remoterelations cmr non-fatal error "hdev-swarm-a-postgresql-replication-offer": watching status for offer: cannot get discharge from "https://10.33.236.139:17070/offeraccess": third party refused discharge: cannot 
discharge: permission denied                                                                                                                                                                                                                                   
controller-0: 08:12:58 ERROR juju.worker.remoterelations cmr exited "hdev-swarm-a-postgresql-replication-offer": watching status for offer: cannot get discharge from "https://10.33.236.139:17070/offeraccess": third party refused discharge: cannot discharg
e: permission denied                                                                                                                                                                                                                                           
controller-0: 08:12:58 INFO juju.worker.remoterelations cmr restarting "hdev-swarm-a-postgresql-replication-offer" in 15s                                                                                                                                      

Digging in the controller logs reveals:

controller-1: 07:52:34 ERROR juju.worker.remoterelations cmr error in remote application worker for hdev-swarm-b-postgresql-replication-offer: watching status for offer: cannot get discharge from "https://10.33.236.76:17070/offeraccess": third party refused discharge: cannot discharge: permission denied

Additional context

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/DPE-10480.

This message was autogenerated

[taurus-forever]

Hi @marcusboden , I agree, we should have better UI and UX here.
Can you please clearly describe better have a juju issue (e.g. user accidentally disabled).
Do you have steps to reproduce which I can copy&paste to automate tests for your case? Tnx!

I have converted it from bug to enhancement (to implement requested monitoring).
We have discussed it internally, it requires the proper planning to complete it in full.
Please bring it to @7annaba3l attention officially.

[marcusboden]

My issue was that the user used to set up cross-dc-relation had been accidentally disabled. To reproduce:

1. create juju user
2. grant access to the postgress model on site A
3. register controller with that user on site B
4. set up replication between A and B
5. disable user on site A

Now you everything looks fine according to juju and the postgres charm. But if you try to promote-to-primary scope=cluster on site B, it will not work and the relation will go into error.

Other issue I could see happening in day 2 ops: Firewall changes restrict access on the oam network used by juju, while database network still works. Would likely end up in a similar scenario.