Skip to content

Patroni and health checks proxy handling #1714

Description

@jpnorenam

Despite Patroni being healthy:

root@juju-78f33d-0:/var/lib/juju/agents/unit-postgresql-0/charm# curl -sk https://10.32.69.4:8008/cluster | jq .
{
  "members": [
    {
      "name": "postgresql-0",
      "role": "leader",
      "state": "running",
      "api_url": "https://10.32.69.4:8008/patroni",
      "host": "10.32.69.4",
      "port": 5432,
      "timeline": 1
    }
  ],
  "scope": "postgresql"
}


root@juju-78f33d-0:/var/lib/juju/agents/unit-postgresql-0/charm# curl -sk https://10.32.69.4:8008/switchover | jq .
{
  "state": "running",
  "postmaster_start_time": "2026-05-26 10:11:01.122042+00:00",
  "role": "master",
  "server_version": 160013,
  "xlog": {
    "location": 24840288
  },
  "timeline": 1,
  "dcs_last_seen": 1779791222,
  "database_system_identifier": "7644140955326675877",
  "patroni": {
    "version": "3.3.8",
    "scope": "postgresql",
    "name": "postgresql-0"
  }
}

It seems httpx.AsyncClient (v0.28+) respects HTTPS_PROXY environment variables by default and routes the Patroni API request through the HTTP proxy, while requests (used for the health check) also respect them but handle no_proxy CIDR notation differently. Resulting in the leader unit awaiting start of the primary:

May 26 10:10:55 juju-78f33d-0 systemd[1]: Started snap.charmed-postgresql.patroni.service - Service for snap application charmed-postgresql.patroni.
░░ Subject: A start job for unit snap.charmed-postgresql.patroni.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit snap.charmed-postgresql.patroni.service has finished successfully.
░░ 
░░ The job identifier is 3046.
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: The files belonging to this database system will be owned by user "_daemon_".
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: This user must also own the server process.
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: The database cluster will be initialized with locale "en_US.UTF-8".
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: The default text search configuration will be set to "english".
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: Data page checksums are enabled.
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: fixing permissions on existing directory /var/snap/charmed-postgresql/common/var/lib/postgresql ... ok
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: fixing permissions on existing directory /var/snap/charmed-postgresql/common/data/logs ... ok
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: creating subdirectories ... ok
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: selecting dynamic shared memory implementation ... posix
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: selecting default max_connections ... 100
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: selecting default shared_buffers ... 128MB
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: selecting default time zone ... Etc/UTC
May 26 10:10:58 juju-78f33d-0 charmed-postgresql.patroni[7069]: creating configuration files ... ok
May 26 10:10:59 juju-78f33d-0 charmed-postgresql.patroni[7069]: running bootstrap script ... ok
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7069]: performing post-bootstrap initialization ... ok
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7069]: syncing data to disk ... ok
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7069]: initdb: warning: enabling "trust" authentication for local connections
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7069]: initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7069]: Success. You can now start the database server using:
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7069]:     /snap/charmed-postgresql/current/usr/lib/postgresql/16/bin/pg_ctl -D /var/snap/charmed-postgresql/common/var/lib/postgresql -l logfile start
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7087]: 10.32.69.4:5432 - no response
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7086]: 2026-05-26 10:11:00 UTC [7086]: user=,db=,app=,client=,line=1 LOG:  00000: pgaudit extension initialized
May 26 10:11:00 juju-78f33d-0 charmed-postgresql.patroni[7086]: 2026-05-26 10:11:00 UTC [7086]: user=,db=,app=,client=,line=2 LOCATION:  _PG_init, pgaudit.c:2263
May 26 10:11:01 juju-78f33d-0 charmed-postgresql.patroni[7086]: 2026-05-26 10:11:01 UTC [7086]: user=,db=,app=,client=,line=3 LOG:  00000: redirecting log output to logging collector process
May 26 10:11:01 juju-78f33d-0 charmed-postgresql.patroni[7086]: 2026-05-26 10:11:01 UTC [7086]: user=,db=,app=,client=,line=4 HINT:  Future log output will appear in directory "/var/snap/charmed-postgresql/common/var/log/postgresql".
May 26 10:11:01 juju-78f33d-0 charmed-postgresql.patroni[7086]: 2026-05-26 10:11:01 UTC [7086]: user=,db=,app=,client=,line=5 LOCATION:  SysLogger_Start, syslogger.c:712
May 26 10:11:01 juju-78f33d-0 charmed-postgresql.patroni[7098]: 10.32.69.4:5432 - accepting connections
May 26 10:11:01 juju-78f33d-0 charmed-postgresql.patroni[7100]: 10.32.69.4:5432 - accepting connections
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]: Exception in thread Thread-43 (process_request_thread):
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]: Traceback (most recent call last):
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]:   File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]:     self.run()
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]:   File "/usr/lib/python3.12/threading.py", line 1010, in run
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]:     self._target(*self._args, **self._kwargs)
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]:   File "/usr/lib/python3/dist-packages/patroni/api.py", line 1685, in process_request_thread
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]:     request.do_handshake()
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]:   File "/usr/lib/python3.12/ssl.py", line 1320, in do_handshake
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]:     self._sslobj.do_handshake()
May 26 10:23:13 juju-78f33d-0 charmed-postgresql.patroni[7033]: ssl.SSLError: [SSL: HTTP_REQUEST] http request (_ssl.c:1000)

Steps to reproduce:

$ juju add model database
$ juju model-config http-proxy=http://squid.internal:3128
$ juju model-config https-proxy=http://squid.internal:3128
$ juju model-config no-proxy="localhost,127.0.0.1,10.0.0.0/8,0,.1,.2,.3,.4,.5,.6,.7,.8,.9,.10,.11,.12,.13,.14,.15,.16,.17,.18,.19,.20,.21,.22,.23,.24,.25,.26,.27,.28,.29,.30,.31,.32,.33,.34,.35,.36,.37,.38,.39,.40,.41,.42,.43,.44,.45,.46,.47,.48,.49,.50,.51,.52,.53,.54,.55,.56,.57,.58,.59,.60,.61,.62,.63,.64,.65,.66,.67,.68,.69,.70,.71,.72,.73,.74,.75,.76,.77,.78,.79,.80,.81,.82,.83,.84,.85,.86,.87,.88,.89,.90,.91,.92,.93,.94,.95,.96,.97,.98,.99,.100,.101,.102,.103,.104,.105,.106,.107,.108,.109,.110,.111,.112,.113,.114,.115,.116,.117,.118,.119,.120,.121,.122,.123,.124,.125,.126,.127,.128,.129,.130,.131,.132,.133,.134,.135,.136,.137,.138,.139,.140,.141,.142,.143,.144,.145,.146,.147,.148,.149,.150,.151,.152,.153,.154,.155,.156,.157,.158,.159,.160,.161,.162,.163,.164,.165,.166,.167,.168,.169,.170,.171,.172,.173,.174,.175,.176,.177,.178,.179,.180,.181,.182,.183,.184,.185,.186,.187,.188,.189,.190,.191,.192,.193,.194,.195,.196,.197,.198,.199,.200,.201,.202,.203,.204,.205,.206,.207,.208,.209,.210,.211,.212,.213,.214,.215,.216,.217,.218,.219,.220,.221,.222,.223,.224,.225,.226,.227,.228,.229,.230,.231,.232,.233,.234,.235,.236,.237,.238,.239,.240,.241,.242,.243,.244,.245,.246,.247,.248,.249,.250,.251,.252,.253,.254,.255"

$ juju model-config cloudinit-userdata='''#cloud-config
write_files:
- path: /etc/environment
  permissions: '0644'
  owner: root:root
  content: |
    LANG=en_US.UTF-8
    LC_ALL=en_US.UTF-8
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
    http_proxy=http://10.22.2.1:3128
    https_proxy=http://10.22.2.1:3128
    no_proxy="localhost,127.0.0.1,10.0.0.0/8,0,.1,.2,.3,.4,.5,.6,.7,.8,.9,.10,.11,.12,.13,.14,.15,.16,.17,.18,.19,.20,.21,.22,.23,.24,.25,.26,.27,.28,.29,.30,.31,.32,.33,.34,.35,.36,.37,.38,.39,.40,.41,.42,.43,.44,.45,.46,.47,.48,.49,.50,.51,.52,.53,.54,.55,.56,.57,.58,.59,.60,.61,.62,.63,.64,.65,.66,.67,.68,.69,.70,.71,.72,.73,.74,.75,.76,.77,.78,.79,.80,.81,.82,.83,.84,.85,.86,.87,.88,.89,.90,.91,.92,.93,.94,.95,.96,.97,.98,.99,.100,.101,.102,.103,.104,.105,.106,.107,.108,.109,.110,.111,.112,.113,.114,.115,.116,.117,.118,.119,.120,.121,.122,.123,.124,.125,.126,.127,.128,.129,.130,.131,.132,.133,.134,.135,.136,.137,.138,.139,.140,.141,.142,.143,.144,.145,.146,.147,.148,.149,.150,.151,.152,.153,.154,.155,.156,.157,.158,.159,.160,.161,.162,.163,.164,.165,.166,.167,.168,.169,.170,.171,.172,.173,.174,.175,.176,.177,.178,.179,.180,.181,.182,.183,.184,.185,.186,.187,.188,.189,.190,.191,.192,.193,.194,.195,.196,.197,.198,.199,.200,.201,.202,.203,.204,.205,.206,.207,.208,.209,.210,.211,.212,.213,.214,.215,.216,.217,.218,.219,.220,.221,.222,.223,.224,.225,.226,.227,.228,.229,.230,.231,.232,.233,.234,.235,.236,.237,.238,.239,.240,.241,.242,.243,.244,.245,.246,.247,.248,.249,.250,.251,.252,.253,.254,.255"
    HTTP_PROXY=http://10.22.2.1:3128
    HTTPS_PROXY=http://10.22.2.1:3128
    NO_PROXY="localhost,127.0.0.1,10.0.0.0/8,0,.1,.2,.3,.4,.5,.6,.7,.8,.9,.10,.11,.12,.13,.14,.15,.16,.17,.18,.19,.20,.21,.22,.23,.24,.25,.26,.27,.28,.29,.30,.31,.32,.33,.34,.35,.36,.37,.38,.39,.40,.41,.42,.43,.44,.45,.46,.47,.48,.49,.50,.51,.52,.53,.54,.55,.56,.57,.58,.59,.60,.61,.62,.63,.64,.65,.66,.67,.68,.69,.70,.71,.72,.73,.74,.75,.76,.77,.78,.79,.80,.81,.82,.83,.84,.85,.86,.87,.88,.89,.90,.91,.92,.93,.94,.95,.96,.97,.98,.99,.100,.101,.102,.103,.104,.105,.106,.107,.108,.109,.110,.111,.112,.113,.114,.115,.116,.117,.118,.119,.120,.121,.122,.123,.124,.125,.126,.127,.128,.129,.130,.131,.132,.133,.134,.135,.136,.137,.138,.139,.140,.141,.142,.143,.144,.145,.146,.147,.148,.149,.150,.151,.152,.153,.154,.155,.156,.157,.158,.159,.160,.161,.162,.163,.164,.165,.166,.167,.168,.169,.170,.171,.172,.173,.174,.175,.176,.177,.178,.179,.180,.181,.182,.183,.184,.185,.186,.187,.188,.189,.190,.191,.192,.193,.194,.195,.196,.197,.198,.199,.200,.201,.202,.203,.204,.205,.206,.207,.208,.209,.210,.211,.212,.213,.214,.215,.216,.217,.218,.219,.220,.221,.222,.223,.224,.225,.226,.227,.228,.229,.230,.231,.232,.233,.234,.235,.236,.237,.238,.239,.240,.241,.242,.243,.244,.245,.246,.247,.248,.249,.250,.251,.252,.253,.254,.255"
'''
$ juju deploy postgresql -n 3

$ juju status
App         Version  Status  Scale  Charm       Channel     Rev  Exposed  Message
postgresql  16.13    active      3  postgresql  16/stable  1089  no       

Unit           Workload  Agent  Machine  Public address  Ports     Message
postgresql/0*  waiting   idle   0        10.32.69.4      5432/tcp  awaiting start of the primary
postgresql/1   waiting   idle   2        10.32.69.6      5432/tcp  awaiting for cluster to start
postgresql/2   waiting   idle   4        10.32.69.10     5432/tcp  awaiting for cluster to start

Machine  State    Address      Inst id        Base          AZ          Message
0        started  10.32.69.4   juju-78f33d-0  ubuntu@24.04  pc8b-infra  Running
2        started  10.32.69.6   juju-78f33d-2  ubuntu@24.04  pc8b-infra  Running
4        started  10.32.69.10  juju-78f33d-4  ubuntu@24.04  pc8b-infra  Running

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working as expected

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions