charmed-cloudflared: tunnel never starts on Ubuntu Core — /etc layout masks resolv.conf, DNS falls back to localhost

[jmgiaever]

Summary

On Ubuntu Core the snap cannot resolve DNS, so cloudflared exits immediately and systemd restart-loops the service until it gives up. Host DNS is healthy throughout — only the snap is affected.

Environment

• charmed-cloudflared 2026.5.0 (rev 77), latest/stable, arm64
• Ubuntu Core 26 (base core26), snapd 2.76, Raspberry Pi 5
• Host resolution works: resolvectl query snapcraft.io succeeds; systemd-resolved stub on 127.0.0.53

Symptom

ERR edge discovery: error looking up Cloudflare edge IPs: the DNS query failed
  error="lookup _v2-origintunneld._tcp.argotunnel.com on [::1]:53: read udp [::1]:45135->[::1]:53: read: connection refused"
ERR Initiating shutdown error="Could not lookup srv records on _v2-origintunneld._tcp.argotunnel.com: ..."
subprocess.CalledProcessError: Command '['/snap/charmed-cloudflared/77/usr/bin/cloudflared', 'tunnel', '--no-autoupdate', '--metrics', '127.0.0.1:39000', '--protocol', 'http2', 'run']' returned non-zero exit status 1.
systemd[1]: snap.charmed-cloudflared.cloudflared.service: Start request repeated too quickly.

Cause

charmed-cloudflared-snap/snap/snapcraft.yaml declares (still present at HEAD):

layout:
  /etc:
    bind: $SNAP_DATA/etc

This replaces the whole of /etc inside the snap's mount namespace with $SNAP_DATA/etc, which is empty on a fresh install. The base's /etc/resolv.conf (a symlink to ../run/systemd/resolve/stub-resolv.conf) is therefore masked:

$ sudo snap run --shell charmed-cloudflared.cloudflared -c 'cat /etc/resolv.conf'
cat: /etc/resolv.conf: No such file or directory

$ sudo snap run --shell charmed-cloudflared.cloudflared -c 'stat -c %n /run/systemd/resolve/stub-resolv.conf'
/run/systemd/resolve/stub-resolv.conf

The symlink target is reachable inside the namespace; only /etc/resolv.conf itself is missing.

cloudflared is a Go binary, and with no /etc/resolv.conf Go's resolver falls back to localhost:53. On Ubuntu Core nothing listens there — systemd-resolved binds 127.0.0.53 and 127.0.0.54 — so every lookup fails with connection refused, including the SRV lookup for edge discovery.

Nothing in the snap creates the file: the only hook is configure, and the sole reference to resolv anywhere in the snap is inside the cloudflared binary itself.

$ ls -A /var/snap/charmed-cloudflared/current/etc/      # empty on a fresh install
$ ls /snap/charmed-cloudflared/current/meta/hooks/
configure

For comparison, a snap on a core22 base without an /etc layout (e.g. node-red) sees the symlink normally and resolves fine on the same host, which is what narrowed this down to the layout rather than to confinement or to Ubuntu Core itself.

Workaround

Create the file the layout leaves missing:

printf 'nameserver 127.0.0.53\nnameserver 1.1.1.1\nnameserver 1.0.0.1\n' \
  | sudo tee /var/snap/charmed-cloudflared/current/etc/resolv.conf
sudo snap restart charmed-cloudflared

The tunnel then registers all four connections normally and survives reboots.

Suggested fix

Any of:

1. Seed a default resolv.conf into $SNAP_DATA/etc from an install hook — e.g. a symlink to /run/systemd/resolve/stub-resolv.conf, or a small file with the stub plus a public fallback.
2. Narrow the layout to the specific paths the snap needs under /etc rather than binding all of /etc.
3. If the current behaviour is intended, document that $SNAP_DATA/etc/resolv.conf must be provided by the operator.

Happy to test a fix on Ubuntu Core.

Still current

Checked against main: charmed-cloudflared-snap/snap/snapcraft.yaml still carries the /etc layout, and charmed-cloudflared-snap/snap/hooks/configure only normalises snap options and restarts the service — nothing creates $SNAP_DATA/etc/resolv.conf.

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/ISD-6491.

This message was autogenerated