Skip to content

fix(networkpolicy): admit the entry points by port number - #61

Open
gurkangul wants to merge 1 commit into
appwrite:mainfrom
gurkangul:fix/edge-numeric-ports
Open

gurkangul wants to merge 1 commit into
appwrite:mainfrom
gurkangul:fix/edge-numeric-ports

Conversation

@gurkangul

Copy link
Copy Markdown
Contributor

What

With networkPolicy.enabled, <fullname>-edge selects the API, realtime and console pods and admits the named port http. That name is 8080 on the API and realtime and 3000 on the console. This is valid Kubernetes, but it seems to trip a race in kube-router, the NetworkPolicy controller k3s ships: for a named port with two numbers, new connections can be rejected for a moment during a policy sync (cloudnativelabs/kube-router#2226).

This admits the same ports by number. A policy has one podSelector, so it takes two policies:

  • <fullname>-edge: API and realtime, TCP 8080
  • <fullname>-edge-console (new): console, TCP 3000

Each keeps both ingress rules of the old policy: networkPolicy.ingress.from on the port (any source when empty), and pods of the release on any port. So this works around kube-router#2226, with no behaviour change on other CNIs: the same traffic is admitted.

  • The numbers are the containerPorts the three Deployments hard-code. A unit test now fails if a Deployment's port changes without its policy.
  • The console's policy is rendered first, so helm upgrade creates it before it narrows edge (Helm applies a release's resources in manifest order).
  • No values change. builds-to-api, orchestrator, mqtt and object-storage are untouched: each of their named ports is one number.

What I can and can't show:

  • The race is a finding from reading kube-router's source. I have not captured it live (no iptables-save or ipset trace), and #2226 has no maintainer reply yet.
  • On our single-node k3s cluster, clients logged 106 Bad Gateway responses from Traefik in front of Appwrite between 2026-11-16 07:00 and 2026-11-17 22:02 UTC. A wrapper chart has added the same numeric-port rules next to edge since about 23:15 UTC on 2026-11-17. From 23:30 UTC to 2026-11-18 06:54 UTC: none in the client logs, and no 502 in Traefik's access log.
  • That is 7.5 hours, which is too short to call it settled: the last 502 came an hour before the change, and we have had a 10-hour quiet stretch with no fix before. I'll report the longer run on edge NetworkPolicy: named port http (8080 on api/realtime, 3000 on console) trips a kube-router race on k3s #60.

Fixes #60

Testing

  • make test passes locally with Helm 4.3.2: ct lint, helm lint --strict for every ci/ scenario, make unittest (261 passed), make kubeconform, make examples, make check-generated (no generated file changed) and make check-changes.
  • helm unittest --strict charts/appwrite also passes with Helm 3.18.6.
  • Rendered every ci/ scenario before and after. Apart from the generated secrets, which differ on every render, the only difference is edge (selector without the console, port 8080/TCP) and the new edge-console.
  • Changing a port in the template or in a Deployment makes the unit tests fail, so the new assertions do check it.
  • Not run: make kind-up install helm-test smoke. Our cluster runs the equivalent rules through a wrapper chart, not this branch.

Checklist

  • make test passes locally
  • make generate was run and the generated files (values.schema.json, the chart README.md, the artifacthub.io/images annotation) are committed (nothing changed)
  • Unit tests added or updated
  • Guides in docs/ and files in examples/ updated for changes to how the chart is configured (docs/security.md; no example names the policy)
  • An entry under [Unreleased] in CHANGELOG.md for user-facing changes (or the skip-changelog label). No version bump or artifacthub.io/changes edit: those happen when a release is cut, see RELEASING.md
  • Renamed or removed values go through Deprecating a value (no value is renamed or removed)

🤖 Generated with Claude Code

The edge policy admitted the named port "http" for the API and realtime
(8080) and the console (3000). One policy admitting a named port with two
numbers trips a race reported against kube-router, the NetworkPolicy
controller k3s ships, that can briefly reject new connections
(cloudnativelabs/kube-router#2226; found in its source, not yet captured
live).

Admit the ports by number instead, one policy per number: "edge" for the
API and realtime on 8080, and the new "edge-console" for the console on
3000. Each keeps both ingress rules of the old policy, so the same traffic
is admitted. A unit test ties the numbers to the containerPorts of the
three Deployments.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@appwrite

appwrite Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

🟢 Tier S · Ready to merge

The rendered policy split preserves the prior ingress behavior, and the changed paths are covered by tests.

The edge NetworkPolicy now renders separate numeric TCP-port policies for API/realtime and console, preserving the configured-source and same-release ingress rules. Tests, the security guide, and the changelog were updated for the split and its port mappings.

Verdict New comments Fixed Still open
💬 Commented 0 0 0

Note

@gurkangul does not have write access to this repository, so Hansi does not approve automatically. A maintainer can review and approve.

📂 Walkthrough · 6
File Change
CHANGELOG.md Documents the numeric-port edge policy split.
charts/appwrite/templates/networkpolicy/edge.yaml Renders console and API/realtime policies in that order with numeric TCP ports.
charts/appwrite/tests/builds_test.yaml Checks build traffic is not admitted by either edge policy.
charts/appwrite/tests/labels_test.yaml Checks API, realtime, and console Deployment ports match the policy ports.
charts/appwrite/tests/networkpolicy_test.yaml Asserts the two policies, selectors, ports, and same-release ingress rule.
docs/security.md Documents the split policies and their admitted traffic.

Reviewed 71d5a61 · Details · Comment @appwrite review to re-run, or mention @appwrite with a question.

@appwrite appwrite Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Tier S · Looks good to merge. Summary

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

edge NetworkPolicy: named port http (8080 on api/realtime, 3000 on console) trips a kube-router race on k3s

1 participant