feat(stack): engine-managed WARP IPv6 pool by default

docs/free-tier-gates.md

@@ -236,28 +236,32 @@ A caller can satisfy all four gates perfectly, send a textbook request, and
 still get a `429`. That is not a gate failure and nothing about the fingerprint
 will fix it.
 
-**What is measured here.** Six nodes in one colo yielded five distinct source
-addresses (WARP hands out roughly seven for six nodes, and nodes collide by
-default). Five nodes shared one address, so five nodes drew on one budget.
-Widening the pool to distinct addresses removed those 429s, with the
-identity and request shape held constant across the change; which rules out
-the fingerprint and points at the address as the variable. The measured
-numbers and the dedupe logic behind them are documented in
-[`stack/dedupe/dedupe.py`](../stack/dedupe/dedupe.py) and
-[`stack/generate-compose.py`](../stack/generate-compose.py).
-
-**What is inferred, not measured.** That upstream's counter actually keys on
-the source address. No controlled A/B was run here: two addresses, identical
-request streams, watching each start returning 429. The observation is
-consistent with per-address bucketing and hard to explain otherwise, since the
-pool widened throughput with nothing else changed — but it is a strong
-inference from one system, not a measurement of upstream's accounting.
-
-**How to falsify it.** Send a fixed request stream through two nodes and record
-when each begins returning 429. Per-address bucketing predicts the two curves
-starting at different times. A global or per-key cap predicts them starting
-together. That experiment is cheap; it simply has not been done, so treat the
-claim accordingly when you rely on it.
+**What is measured here (2026-10-27).** The counter keys on the **egress
+block**, not the individual address: a `/64` on IPv6 and a `/32` on IPv4. A
+residential IPv4 address and Cloudflare's shared WARP IPv4 NAT both returned
+`429` with a `Retry-After` that resolved to **00:00:00 UTC**, i.e. a daily
+window. A fresh WARP IPv6 exit (`2a09:bac5:...::/64`) returned `200` "42" while
+the IPv4 path was still refused, and three parallel `beam-free` requests through
+two distinct `/64` exits all returned `200`. That is the per-block model with a
+daily reset, not the per-address model this document previously inferred.
+
+**What that changes.** Cloudflare's WARP IPv4 egress is one shared NAT address,
+so rotating it does not add headroom: everyone draws on the same exhausted
+bucket. A routed IPv6 `/64` lets each exit bind its own outer source and obtain
+its own egress `/64`, which is why `warp.stack: ipv6` plus `source_ipv6` is the
+recommended path. The engine (`opencode2api` v1.4.4+) owns this pool and retires
+an exit on `429`/`403` instead of retrying the same block.
+
+**Still inferred.** The exact window boundary is observed, not documented:
+every refusal we measured pointed at the next 00:00 UTC. Whether a block's
+window is calendar-daily or rolling toward midnight is not distinguishable from
+the outside. Treat `Retry-After` as authoritative and do not model it yourself.
+
+**How to check it again.** Send the same request through two exits and compare
+their `Retry-After`. Same value at the same second means one shared block or a
+shared reset boundary; different values mean separate counters.
+`scripts/gate-probe.py --proxy ...` still distinguishes a gate from the quota
+axis, and exits `3` on quota.
 
 This axis is what the proxy pool exists for. It is also why the pool refuses to
 pretend: `ofu-dedupe` measures real egress so that six nodes are not quietly

scripts/smoke.sh

@@ -10,7 +10,7 @@ set -uo pipefail
 
 BASE_URL="${1:-http://127.0.0.1:8080}"
 API_KEY="${2:-local-test-key}"
-MODEL="${3:-beam-free}"
+MODEL="${3:-deepseek-v4.1-flash-free}"
 WEBUI="${WEBUI:-http://127.0.0.1:8081}"
 
 fail=0
@@ -39,11 +39,35 @@ rc=1
 if [ "${count:-0}" -gt 0 ]; then rc=0; fi
 check $rc "v1/models returned $count models"
 
-step "3/6  proxy pool reachable and de-duplicated (each proxy must forward to an ip echo)"
+step "3/6  egress pool"
+QUOTA_EXHAUSTED=0
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+MODE=$(python3 -c "
+import json
+cfg = json.load(open('${SCRIPT_DIR}/../stack/config.json'))
+print('warp' if cfg.get('warp', {}).get('enabled') else 'external')
+" 2>/dev/null)
+
+if [ "$MODE" = "warp" ]; then
+  # The engine owns registration and rotation. Ready means at least one
+  # active exit; cooling and standby are normal states, not faults.
+  health=$(curl -s -m 10 "$BASE_URL/healthz")
+  status=$(printf '%s' "$health" | python3 -c 'import json,sys; print(json.load(sys.stdin).get("warp",{}).get("status",""))' 2>/dev/null)
+  active=$(printf '%s' "$health" | python3 -c 'import json,sys; print(json.load(sys.stdin).get("warp",{}).get("active",0))' 2>/dev/null)
+  echo "  warp.status=${status:-unknown} active=${active:-0}"
+  cooling=$(printf '%s' "$health" | python3 -c 'import json,sys; print(json.load(sys.stdin).get("warp",{}).get("cooling",0))' 2>/dev/null)
+  if [ "$status" = "ready" ] && [ "${active:-0}" -ge 1 ]; then
+    check 0 "engine-managed WARP pool has an active exit"
+  elif [ "${cooling:-0}" -gt 0 ]; then
+    QUOTA_EXHAUSTED=1
+    echo "SKIP: every WARP exit is cooling down (upstream daily quota). Retry after 00:00 UTC."
+  else
+    check 1 "engine-managed WARP pool has an active exit (status=${status:-unknown})"
+  fi
+else
 # Proxy ports are read from the engine config so this stays correct after a
 # resize, paired with their node id so a drained node can be told apart from a
 # broken one.
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 mapfile -t cfg_rows < <(python3 -c "
 import json
 cfg = json.load(open('${SCRIPT_DIR}/../stack/config.json'))
@@ -112,6 +136,7 @@ else
   if [ "$ok" -gt 0 ] && [ "$distinct" -ge "$ok" ]; then rc=0; fi
   check $rc "pool is de-duplicated ($distinct/$ok serving distinct, $drained drained, $total configured)"
 fi
+fi
 
 # A single free-tier completion. Retries once: when the engine is shared with
 # other clients, a single request can lose the race and hit a 60s timeout.
@@ -121,7 +146,10 @@ ask_chat() { # $1 = extra json fragment (e.g. '"stream":false,')
   for attempt in 1 2; do
     body=$(curl -s -m 150 -X POST "$BASE_URL/v1/chat/completions" \
       -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
-      -d "{\"model\":\"$MODEL\",${extra}\"messages\":[{\"role\":\"user\",\"content\":\"Reply with exactly: OK\"}],\"max_tokens\":120}")
+      -d "{\"model\":\"$MODEL\",${extra}\"messages\":[{\"role\":\"user\",\"content\":\"Reply with exactly: OK\"}],\"max_tokens\":200}")
+    if printf '%s' "$body" | grep -q 'FreeUsageLimitError\|Free usage limit reached'; then
+      return 2
+    fi
     if printf '%s' "$body" | python3 -c 'import json,sys
 d = json.load(sys.stdin)
 if d.get("error"):
@@ -137,15 +165,31 @@ sys.exit(0 if "OK" in text else 1)' 2>/dev/null; then
 }
 
 step "4/6  free-tier chat completion ($MODEL)"
-ask_chat ""
-check $? "chat completion returned content"
+if [ "$QUOTA_EXHAUSTED" = "1" ]; then
+  echo "SKIP: no exit with headroom; the completion checks need a successful baseline."
+else
+  ask_chat ""
+  rc=$?
+  if [ "$rc" -eq 0 ]; then check 0 "chat completion returned content"
+  elif [ "$rc" -eq 2 ]; then echo "SKIP: upstream daily quota exhausted (FreeUsageLimitError). Retry after 00:00 UTC."
+  else check 1 "chat completion returned content"
+  fi
+fi
 
 # Gates 3 and 4 of the free-tier fingerprint: no tools at all, and stream:false.
 # A gateway that only mints valid session headers returns 403 for this exact
 # shape. See docs/free-tier-gates.md -- this is the cheapest canary we have.
 step "5/6  fingerprint gate: zero tools + stream:false"
-ask_chat '"stream":false,'
-check $? "zero-tool non-stream request accepted (gates 3+4 intact)"
+if [ "$QUOTA_EXHAUSTED" = "1" ]; then
+  echo "SKIP: no exit with headroom; the gate canary needs a successful baseline."
+else
+  ask_chat '"stream":false,'
+  rc=$?
+  if [ "$rc" -eq 0 ]; then check 0 "zero-tool non-stream request accepted (gates 3+4 intact)"
+  elif [ "$rc" -eq 2 ]; then echo "SKIP: upstream daily quota exhausted (FreeUsageLimitError). Retry after 00:00 UTC."
+  else check 1 "zero-tool non-stream request accepted (gates 3+4 intact)"
+  fi
+fi
 
 step "6/6  webui reachable on $WEBUI"
 code=$(curl -s -o /dev/null -w '%{http_code}' -m 10 "$WEBUI/" 2>/dev/null)

stack/.env.example

@@ -16,4 +16,15 @@ WARP_ACCEPT_TOS=0
 
 # Local API key the engine accepts. Must match `server_keys` in config.json.
 # Change it if the engine is reachable from anything other than localhost.
-# OPENCODE2API_SERVER_KEYS=sk-change-me
+# OPENCODE2API_SERVER_KEYS=sk-change-me
+
+# Engine-managed WARP pool (warp.enabled: true in config.json).
+#
+# A routed IPv6 /64 the engine may bind individual source addresses from. Each
+# WARP exit then gets its own egress /64, which is the block the free-tier quota
+# is counted in. Without it, WARP falls back to the host's default source and
+# the pool can share one exhausted bucket.
+#
+# Find yours with: ip -6 addr show scope global
+# Example: WARP_SOURCE_IPV6=2001:db8:1234:5678::/64
+WARP_SOURCE_IPV6=

stack/config.json

@@ -6,10 +6,14 @@
   "zen_keys": [],
   "go_keys": [],
   "anonymous": true,
+  "warp": {
+    "enabled": true,
+    "pool_size": 4,
+    "stack": "ipv6",
+    "source_ipv6": ""
+  },
   "proxies": [
-    "http://127.0.0.1:18090",
-    "http://127.0.0.1:18091",
-    "http://127.0.0.1:18092"
+    "direct"
   ],
   "proxyfile": "",
   "upstream": {
@@ -18,7 +22,7 @@
     "anonymous": "https://opencode.ai/inference"
   },
   "retry": {
-    "max_attempts": 3,
+    "max_attempts": 6,
     "timeout_seconds": 180
   },
   "models": {

stack/docker-compose.yml

@@ -1,155 +1,39 @@
 # GENERATED BY stack/generate-compose.py - do not edit by hand.
-# Each node = one WARP network namespace + one CONNECT proxy on its own port.
-
-x-warp: &warp
-  build:
-    context: ./warp
-    dockerfile: Dockerfile
-  image: ofu-warp-rotator:local
-  pull_policy: never
-  privileged: true
-  cap_add:
-    - NET_ADMIN
-  devices:
-    - /dev/net/tun:/dev/net/tun
-  restart: unless-stopped
-  healthcheck:
-    test: ["CMD", "curl", "-sf", "http://localhost:8001/health"]
-    interval: 30s
-    timeout: 5s
-    retries: 3
-    start_period: 90s
-
-x-tinyproxy: &tinyproxy
-  build:
-    context: ./tinyproxy
-  image: ofu-tinyproxy:local
-  pull_policy: never
-  restart: unless-stopped
-  healthcheck:
-    test: ["CMD-SHELL", "nc -z localhost 8888 || exit 1"]
-    interval: 15s
-    timeout: 3s
-    retries: 3
-    start_period: 10s
+# Free-model traffic leaves through the engine's built-in WARP pool.
+# No external WARP namespaces, tinyproxy nodes, or dedupe service.
 
 services:
   # Seeds the engine's config into its named volume (bind-mounting that
   # path breaks startup, so it is copied instead).
   config-init:
     image: docker.io/library/busybox:1.36
-    command: ["sh", "-c", "install -o 1000 -g 1000 -m 0644 /seed.json /vol/config.json"]
-    volumes:
-      - ./config.json:/seed.json:ro
-      - ofu-engine:/vol
-    restart: "no"
-
-  alztrk-warp-a:
-    <<: *warp
-    container_name: ofu-warp-a
-    environment:
-      - CORS_ALLOW_ORIGINS=*
-      - WARP_ROTATION_INTERVAL=86400
-      - WARP_ACCEPT_TOS=${WARP_ACCEPT_TOS:-0}
-    ports:
-      - "18090:8888"
-    volumes:
-      - ofu-warp-a:/app/data
-
-  tinyproxy-a:
-    <<: *tinyproxy
-    container_name: ofu-proxy-a
-    network_mode: service:alztrk-warp-a
-    depends_on:
-      alztrk-warp-a:
-        condition: service_healthy
     environment:
-      # Names this node's drain marker, /markers/<NODE_ID>.
-      - NODE_ID=a
-    volumes:
-      - ofu-dedupe:/markers
-
-  alztrk-warp-b:
-    <<: *warp
-    container_name: ofu-warp-b
-    environment:
-      - CORS_ALLOW_ORIGINS=*
-      - WARP_ROTATION_INTERVAL=86425
       - WARP_ACCEPT_TOS=${WARP_ACCEPT_TOS:-0}
-    ports:
-      - "18091:8888"
-    volumes:
-      - ofu-warp-b:/app/data
-
-  tinyproxy-b:
-    <<: *tinyproxy
-    container_name: ofu-proxy-b
-    network_mode: service:alztrk-warp-b
-    depends_on:
-      alztrk-warp-b:
-        condition: service_healthy
-    environment:
-      # Names this node's drain marker, /markers/<NODE_ID>.
-      - NODE_ID=b
-    volumes:
-      - ofu-dedupe:/markers
-
-  alztrk-warp-c:
-    <<: *warp
-    container_name: ofu-warp-c
-    environment:
-      - CORS_ALLOW_ORIGINS=*
-      - WARP_ROTATION_INTERVAL=86450
-      - WARP_ACCEPT_TOS=${WARP_ACCEPT_TOS:-0}
-    ports:
-      - "18092:8888"
-    volumes:
-      - ofu-warp-c:/app/data
-
-  tinyproxy-c:
-    <<: *tinyproxy
-    container_name: ofu-proxy-c
-    network_mode: service:alztrk-warp-c
-    depends_on:
-      alztrk-warp-c:
-        condition: service_healthy
-    environment:
-      # Names this node's drain marker, /markers/<NODE_ID>.
-      - NODE_ID=c
-    volumes:
-      - ofu-dedupe:/markers
-
-  # Keeps egress addresses distinct across the pool. WARP draws from a
-  # small per-region pool, so nodes collide by default; this one polls
-  # every node, re-rolls the collisions, and drains nodes that cannot be
-  # given their own address. Best effort: if it dies, the pool still runs.
-  ofu-dedupe:
-    build:
-      context: ./dedupe
-    image: ofu-dedupe:local
-    pull_policy: never
-    container_name: ofu-dedupe
+      - WARP_SOURCE_IPV6=${WARP_SOURCE_IPV6:-}
     command:
-      - "--interval=${DEDUPE_INTERVAL:-30}"
-    environment:
-      - POOL_NODES=alztrk-warp-a,alztrk-warp-b,alztrk-warp-c
+      - sh
+      - -c
+      - |
+        if [ "$$WARP_ACCEPT_TOS" != "1" ]; then
+          echo 'Cloudflare WARP terms were not accepted.' >&2
+          echo 'Read https://www.cloudflare.com/application/terms/ and set WARP_ACCEPT_TOS=1 in stack/.env.' >&2
+          exit 1
+        fi
+        if [ -n "$$WARP_SOURCE_IPV6" ]; then
+          sed "s|\"source_ipv6\": \"\"|\"source_ipv6\": \"$$WARP_SOURCE_IPV6\"|" /seed.json > /tmp/config.json
+        else
+          cp /seed.json /tmp/config.json
+        fi
+        install -o 1000 -g 1000 -m 0644 /tmp/config.json /vol/config.json
     volumes:
-      - ofu-dedupe:/markers
-    restart: unless-stopped
-    depends_on:
-      alztrk-warp-a:
-        condition: service_healthy
-    healthcheck:
-      test: ["CMD", "python3", "-c", "import sys; sys.exit(0 if 'dedupe' in open('/proc/1/cmdline').read() else 1)"]
-      interval: 30s
-      timeout: 5s
-      retries: 3
-      start_period: 30s
+      - ./config.json:/seed.json:ro
+      - ofu-engine:/vol
+    restart: "no"
 
   opencode-engine:
-    image: ghcr.io/jasonxu114514/opencode2api:v1.4.1
+    image: ghcr.io/jasonxu114514/opencode2api:v1.4.5
     container_name: ofu-engine
-    network_mode: host   # so 127.0.0.1:<proxy port> resolves the pool
+    network_mode: host   # built-in WARP needs outbound UDP and source-address binding
     volumes:
       - ofu-engine:/var/lib/opencode2api
     restart: unless-stopped
@@ -159,7 +43,3 @@ services:
 
 volumes:
   ofu-engine:
-  ofu-dedupe:      # per-node drain markers, shared with the proxies
-  ofu-warp-a:
-  ofu-warp-b:
-  ofu-warp-c: