diff --git a/CLAUDE.md b/CLAUDE.md index 040c223..8455141 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -35,6 +35,14 @@ an empty op means unaudited and unauthorized (validate only). - `presence.go` is the IPN bus view. `tsping` name resolution uses it and falls back to the 30s `Status()` cache only while that view is unseeded. +- `collector.local_only` runs the collector with no tailscaled (the ProxyGroup + sidecar placement): `config.go` defaults a network-less target to `local` + and rejects `network: tailnet`, `tsping` and the UI; `main.go` then skips + the LocalAPI preflight, presence view, resolver binding and dependency + learner, and takes the identity from `os.Hostname()`, so payloads carry + `collector.name` and omit `collector.tailscale_ip`. The classifier already + accepts a nil presence and classifies `network: local` failures from device + evidence alone. - `alert.go` is one mutex-guarded state machine. Grace and flap timers hand work to a goroutine from `AfterFunc`: `tstest.Clock.Advance` holds the clock lock while callbacks fire, so they cannot call back into the clock. @@ -151,3 +159,33 @@ - `stageRelease` lays files out like an unpacked GoReleaser archive, which keeps the `examples/` prefix; keep the staging, the docs and `.goreleaser.yaml` in step. - `github.com/pkg/sftp` and `golang.org/x/crypto/ssh` are direct deps only for this test. + +## kind ProxyGroup test + +- `TestKindProxyGroup` (`kind_proxygroup_test.go`) is gated on `TAIL_WATCH_KIND=1`, + skips when `kind`, `kubectl`, `helm` or `docker` is missing, and runs the fenced bash + blocks under each method heading in `docs/how-to/kubernetes-proxygroup.md`, so editing + those blocks changes the test. The sidecar contract the blocks produce: pods with both + `tailscale.com/parent-resource-type: proxygroup` and `tail-watch/inject: "true"` get a + local-only `tail-watch` container reading ConfigMap `tail-watch-` (key + `tail-watch.yaml`, optional); sink secrets come from a same-named Secret via `envFrom`. + Every method builds that name from the pod's `tailscale.com/parent-resource` label; + never hard-code `ingress`. +- `examples/kubernetes/sidecar.yaml` is the reference container. `TestSidecarCopiesAgree` + (not gated) fails when the Kyverno, MutatingAdmissionPolicy, freeze-and-patch or + how-to webhook copy drifts from it, so edit all five together. +- `scripts/kind-stack.sh up|down` is the harness (cluster config + `cmd/tail-watch/testdata/kind-proxygroup.yaml`, fixtures in + `cmd/tail-watch/testdata/kind/`); the test runs `up` itself when the cluster or the + `ingress` ProxyGroup is missing. The operator chart is pinned to the newest chart at or + below the `tailscale.com` version in `go.mod`; the OAuth client comes from the + `tailscale-phase-2/tailscale` pulumi env as `tailscale:operatorClientId/Secret` and + reaches Helm through a mode-0600 values file, never the command line or a log. +- The test's `sidecarConfig` is the documented ConfigMap with tighter timing; keep its + mechanics (local-only, `tcp:` check, webhook sink) in step with + `examples/kubernetes/configmap.yaml`. The receiver is `alert-receiver` in `testdata/kind/`: + it stores POST bodies and returns them on `GET /events`, and the test asserts new events + by offset because the receiver accumulates. +- The test leaves the harness up. Teardown is `scripts/kind-stack.sh down`: it deletes the + cluster, the Tailscale Service `svc:tw-pg-web`, and every tailnet device with the `tw-pg` + hostname prefix. diff --git a/README.md b/README.md index 7de3659..ad4d85f 100644 --- a/README.md +++ b/README.md @@ -40,7 +40,8 @@ The [documentation index](docs/README.md) lists every page. The main ones: systemd](docs/how-to/deploy.md#run-under-systemd)), [write the ACLs](docs/how-to/acls.md), [send alerts to Slack](docs/how-to/slack-sink.md), - [manage monitors in the UI](docs/how-to/monitor-ui.md), and + [manage monitors in the UI](docs/how-to/monitor-ui.md), + [watch a Kubernetes ProxyGroup](docs/how-to/kubernetes-proxygroup.md), and [troubleshoot](docs/how-to/troubleshoot.md) - Look up: [configuration](docs/reference/config.md), [command line](docs/reference/cli.md), [metrics](docs/reference/metrics.md), and diff --git a/cmd/tail-watch/config.go b/cmd/tail-watch/config.go index 0e83e33..6ea671a 100644 --- a/cmd/tail-watch/config.go +++ b/cmd/tail-watch/config.go @@ -63,6 +63,10 @@ type CollectorConfig struct { Listen string StateDir string UI bool + // LocalOnly runs the collector without tailscaled: targets default to + // network local, and tailnet targets, tsping checks and the UI are + // rejected. + LocalOnly bool } // Target is one monitored device with its checks and resolved timings. @@ -143,9 +147,10 @@ type rawConfig struct { } type rawCollector struct { - Listen string `yaml:"listen"` - StateDir string `yaml:"state_dir"` - UI *bool `yaml:"ui"` + Listen string `yaml:"listen"` + StateDir string `yaml:"state_dir"` + UI *bool `yaml:"ui"` + LocalOnly *bool `yaml:"local_only"` } type rawDefaults struct { @@ -245,16 +250,30 @@ func builtinTiming() timing { } } +// errUILocalOnly is returned for collector.ui and for --ui in local-only mode. +var errUILocalOnly = errors.New("the UI is unavailable in local-only mode") + +// targetRules is what every target in one config resolves against. +type targetRules struct { + base timing + localOnly bool +} + func resolveConfig(raw rawConfig) (*Config, []error) { var errs []error + localOnly := raw.Collector.LocalOnly != nil && *raw.Collector.LocalOnly + if localOnly && raw.Collector.UI != nil && *raw.Collector.UI { + errs = append(errs, fmt.Errorf("collector.ui: %w", errUILocalOnly)) + } base := resolveDefaults(&errs, raw.Defaults) - targets, targetErrs := resolveTargets(raw.Targets, base) + targets, targetErrs := resolveTargets(raw.Targets, targetRules{base: base, localOnly: localOnly}) errs = append(errs, targetErrs...) cfg := &Config{ Collector: CollectorConfig{ - Listen: raw.Collector.Listen, - StateDir: raw.Collector.StateDir, - UI: raw.Collector.UI != nil && *raw.Collector.UI, + Listen: raw.Collector.Listen, + StateDir: raw.Collector.StateDir, + UI: raw.Collector.UI != nil && *raw.Collector.UI, + LocalOnly: localOnly, }, Targets: targets, Sinks: nil, @@ -280,13 +299,13 @@ func resolveDefaults(errs *[]error, d rawDefaults) timing { // resolveTargets validates a target list as a whole: each target, then the // parent references and parent cycles across the list. -func resolveTargets(raw []rawTarget, base timing) ([]Target, []error) { +func resolveTargets(raw []rawTarget, rules targetRules) ([]Target, []error) { var errs []error targets := make([]Target, len(raw)) seen := map[string]int{} checksOK := make([]bool, len(raw)) for i, rt := range raw { - t, ok := resolveTarget(&errs, i, rt, base, seen) + t, ok := resolveTarget(&errs, i, rt, rules, seen) targets[i] = t checksOK[i] = ok } @@ -332,19 +351,19 @@ func resolveTargets(raw []rawTarget, base timing) ([]Target, []error) { return targets, errs } -func resolveTarget(errs *[]error, i int, rt rawTarget, base timing, seen map[string]int) (Target, bool) { +func resolveTarget(errs *[]error, i int, rt rawTarget, rules targetRules, seen map[string]int) (Target, bool) { t := Target{ Name: rt.Name, Addr: rt.Addr, Site: rt.Site, Parent: rt.Parent, InsecureTLS: rt.InsecureTLS, - Interval: base.Interval, - Timeout: base.Timeout, - FailThreshold: base.FailThreshold, - RecoverThreshold: base.RecoverThreshold, - FlapLimit: base.FlapLimit, - FlapWindow: base.FlapWindow, + Interval: rules.base.Interval, + Timeout: rules.base.Timeout, + FailThreshold: rules.base.FailThreshold, + RecoverThreshold: rules.base.RecoverThreshold, + FlapLimit: rules.base.FlapLimit, + FlapWindow: rules.base.FlapWindow, Severity: SeverityPage, Network: NetworkTailnet, @@ -378,17 +397,7 @@ func resolveTarget(errs *[]error, i int, rt rawTarget, base timing, seen map[str fmt.Errorf("%s.severity: target %s: severity %q must be page or warn", loc, targetLabel(t), rt.Severity), ) } - switch rt.Network { - case "", string(NetworkTailnet): - t.Network = NetworkTailnet - case string(NetworkLocal): - t.Network = NetworkLocal - default: - *errs = append( - *errs, - fmt.Errorf("%s.network: target %s: network %q must be tailnet or local", loc, targetLabel(t), rt.Network), - ) - } + resolveNetwork(errs, loc, rt.Network, rules.localOnly, &t) applyDur(errs, loc+".interval", rt.Interval, &t.Interval) applyDur(errs, loc+".timeout", rt.Timeout, &t.Timeout) @@ -427,11 +436,44 @@ func resolveTarget(errs *[]error, i int, rt rawTarget, base timing, seen map[str *errs = append(*errs, fmt.Errorf("%s.flap_limit: target %s: must be >= 1", loc, targetLabel(t))) } - ok := resolveChecks(errs, loc, rt.Checks, &t) + ok := resolveChecks(errs, loc, rt.Checks, &t, rules.localOnly) return t, ok } -func resolveChecks(errs *[]error, loc string, rawChecks []string, t *Target) bool { +// resolveNetwork sets t.Network. localOnly defaults an unset network to +// local and rejects an explicit tailnet. +func resolveNetwork(errs *[]error, loc, raw string, localOnly bool, t *Target) { + switch raw { + case "": + if localOnly { + t.Network = NetworkLocal + } else { + t.Network = NetworkTailnet + } + case string(NetworkTailnet): + if localOnly { + *errs = append( + *errs, + fmt.Errorf( + "%s.network: target %s: network %q is not valid in local-only mode", + loc, + targetLabel(*t), + raw, + ), + ) + } + t.Network = NetworkTailnet + case string(NetworkLocal): + t.Network = NetworkLocal + default: + *errs = append( + *errs, + fmt.Errorf("%s.network: target %s: network %q must be tailnet or local", loc, targetLabel(*t), raw), + ) + } +} + +func resolveChecks(errs *[]error, loc string, rawChecks []string, t *Target, localOnly bool) bool { if len(rawChecks) == 0 { *errs = append(*errs, fmt.Errorf("%s.checks: target %s: checks must not be empty", loc, targetLabel(*t))) return false @@ -455,10 +497,14 @@ func resolveChecks(errs *[]error, loc string, rawChecks []string, t *Target) boo ) } seenChk[c.String()] = struct{}{} - if t.Network == NetworkLocal && c.Kind() == checkTSPing { + if c.Kind() == checkTSPing && (t.Network == NetworkLocal || localOnly) { + msg := "tsping is not valid for network local" + if localOnly { + msg = "tsping is not valid in local-only mode" + } *errs = append( *errs, - fmt.Errorf("%s.checks[%d]: target %s: tsping is not valid for network local", loc, j, targetLabel(*t)), + fmt.Errorf("%s.checks[%d]: target %s: %s", loc, j, targetLabel(*t), msg), ) } if h, isHTTP := c.(httpCheck); isHTTP && t.Addr != "" && !sameHost(t.Addr, h.host) { @@ -620,8 +666,9 @@ var ( func (c *Config) validateCandidate(existing []rawTarget, candidate rawTarget) (Target, []error) { all := append(slices.Clone(existing), candidate) idx := len(existing) - targets, errs := resolveTargets(all, c.Base) - _, baseline := resolveTargets(existing, c.Base) + rules := targetRules{base: c.Base, localOnly: c.Collector.LocalOnly} + targets, errs := resolveTargets(all, rules) + _, baseline := resolveTargets(existing, rules) known := make(map[string]bool, len(baseline)) for _, err := range baseline { known[err.Error()] = true diff --git a/cmd/tail-watch/config_test.go b/cmd/tail-watch/config_test.go index e628d87..e6da250 100644 --- a/cmd/tail-watch/config_test.go +++ b/cmd/tail-watch/config_test.go @@ -327,10 +327,117 @@ func TestExampleConfig(t *testing.T) { } } +//nolint:gocognit // Local-only acceptance and rejection cases keep their inputs next to the assertions. +func TestLocalOnlyConfig(t *testing.T) { + t.Parallel() + t.Run("targets default to local", func(t *testing.T) { + t.Parallel() + cfg, err := parseConfig([]byte(` +collector: {local_only: true} +targets: + - {name: web, addr: web.apps.svc.cluster.local, checks: ['tcp:443']} + - {name: api, addr: api.apps.svc.cluster.local, network: local, checks: ['tcp:80']} +`)) + if err != nil { + t.Fatal(err) + } + if !cfg.Collector.LocalOnly { + t.Fatal("local_only not resolved") + } + for _, target := range cfg.Targets { + if target.Network != NetworkLocal { + t.Errorf("target %s network %s, want local", target.Name, target.Network) + } + } + }) + t.Run("rejections", func(t *testing.T) { + t.Parallel() + tests := []struct { + name string + src string + wants []string + }{ + { + name: "tailnet target", + src: ` +collector: {local_only: true} +targets: + - {name: nas, addr: 100.64.0.1, network: tailnet, checks: [icmp]} +`, + wants: []string{"targets[0].network", "nas", "tailnet", "local-only"}, + }, + { + name: "tsping check", + src: ` +collector: {local_only: true} +targets: + - {name: nas, addr: 100.64.0.1, checks: [tsping]} +`, + wants: []string{"targets[0].checks[0]", "nas", "tsping is not valid in local-only mode"}, + }, + { + name: "ui on", + src: ` +collector: {local_only: true, ui: true, state_dir: /var/lib/tail-watch} +targets: + - {name: web, addr: 10.0.0.1, checks: ['tcp:443']} +`, + wants: []string{"collector.ui", "unavailable in local-only mode"}, + }, + { + name: "several problems together", + src: ` +collector: {local_only: true, ui: true, state_dir: /var/lib/tail-watch} +targets: + - {name: nas, addr: 100.64.0.1, network: tailnet, checks: [tsping]} + - {name: web, addr: 10.0.0.1, checks: ['tcp:443']} +`, + wants: []string{ + "collector.ui", + "targets[0].network", + "targets[0].checks[0]", + "tsping is not valid in local-only mode", + }, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + _, err := parseConfig([]byte(tt.src)) + if err == nil { + t.Fatal("expected error") + } + for _, want := range tt.wants { + if !strings.Contains(err.Error(), want) { + t.Errorf("error %q missing %q", err, want) + } + } + }) + } + }) + t.Run("local_only off keeps today", func(t *testing.T) { + t.Parallel() + cfg, err := parseConfig([]byte(` +collector: {local_only: false} +targets: + - {name: nas, addr: 100.64.0.1, checks: [tsping]} +`)) + if err != nil { + t.Fatal(err) + } + if cfg.Collector.LocalOnly { + t.Fatal("local_only resolved on") + } + if cfg.Targets[0].Network != NetworkTailnet { + t.Fatalf("network %s, want tailnet", cfg.Targets[0].Network) + } + }) +} + func TestListenFlagOverride(t *testing.T) { t.Parallel() cfg := &Config{ - Collector: CollectorConfig{Listen: "127.0.0.1:1", StateDir: "", UI: false}, + Collector: CollectorConfig{Listen: "127.0.0.1:1", StateDir: "", UI: false, LocalOnly: false}, Targets: nil, Sinks: nil, Heartbeat: nil, diff --git a/cmd/tail-watch/event_test.go b/cmd/tail-watch/event_test.go index 80b1d38..f681466 100644 --- a/cmd/tail-watch/event_test.go +++ b/cmd/tail-watch/event_test.go @@ -229,6 +229,29 @@ func TestPayloadContract(t *testing.T) { } } +func TestStartedPayloadLocalOnlyIdentity(t *testing.T) { + t.Parallel() + ev := newStartedEvent( + "evt_L", + CollectorInfo{Name: "pg-ingress-0", TailscaleIP: ""}, + goldenTransition().At, + ) + var m map[string]any + if err := json.Unmarshal(marshalEvent(t, ev), &m); err != nil { + t.Fatal(err) + } + col, ok := m["collector"].(map[string]any) + if !ok { + t.Fatalf("collector %v", m["collector"]) + } + if col["name"] != "pg-ingress-0" { + t.Fatalf("collector.name %v", col["name"]) + } + if _, hasIP := col["tailscale_ip"]; hasIP { + t.Fatal("local-only payload must omit collector.tailscale_ip") + } +} + func TestMapTransition(t *testing.T) { t.Parallel() tests := []struct { diff --git a/cmd/tail-watch/examples_kubernetes_test.go b/cmd/tail-watch/examples_kubernetes_test.go new file mode 100644 index 0000000..cb0d035 --- /dev/null +++ b/cmd/tail-watch/examples_kubernetes_test.go @@ -0,0 +1,216 @@ +package main + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" + "reflect" + "regexp" + "slices" + "strings" + "testing" + + "go.yaml.in/yaml/v3" +) + +// TestSidecarCopiesAgree keeps every embedded copy of the sidecar in step with +// examples/kubernetes/sidecar.yaml. The Kyverno policy and freeze-and-patch are +// compared field by field. The MutatingAdmissionPolicy's CEL and the how-to's +// Go handler cannot be parsed here, so they must contain every key and value of +// the reference. Each copy must also build the ConfigMap and Secret name from +// the ProxyGroup rather than hard-code it. +func TestSidecarCopiesAgree(t *testing.T) { + t.Parallel() + root := repoRoot(t) + read := func(rel string) string { + t.Helper() + b, err := os.ReadFile(filepath.Join(root, rel)) + if err != nil { + t.Fatal(err) + } + return string(b) + } + + var ref podFragment + if err := yaml.Unmarshal([]byte(read("examples/kubernetes/sidecar.yaml")), &ref); err != nil { + t.Fatal(err) + } + want := ref.withoutName(t, "tail-watch-ingress") + + structural := []struct { + name, nameExpr string + fragment func(*testing.T) podFragment + }{ + { + name: "kyverno", + nameExpr: `tail-watch-{{ request.object.metadata.labels."tailscale.com/parent-resource" }}`, + fragment: func(t *testing.T) podFragment { + t.Helper() + return kyvernoFragment(t, read("examples/kubernetes/kyverno-clusterpolicy.yaml")) + }, + }, + { + name: "freeze-and-patch", + nameExpr: "tail-watch-${PG}", + fragment: func(t *testing.T) podFragment { + t.Helper() + return freezeFragment(t, read("examples/kubernetes/freeze-and-patch.sh")) + }, + }, + } + for _, tc := range structural { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + if got := tc.fragment(t).withoutName(t, tc.nameExpr); !reflect.DeepEqual(got, want) { + t.Fatalf("sidecar differs from sidecar.yaml\n got %v\nwant %v", got, want) + } + }) + } + + textual := []struct { + name, src, nameExpr string + nameUses int + }{ + { + name: "mutatingadmissionpolicy", + src: read("examples/kubernetes/mutatingadmissionpolicy.yaml"), + nameExpr: `"tail-watch-" + object.metadata.labels["tailscale.com/parent-resource"]`, + nameUses: 3, + }, + { + name: "webhook handler", + src: howToGoBlock(t, read("docs/how-to/kubernetes-proxygroup.md")), + nameExpr: `"tail-watch-" + pod.Labels["tailscale.com/parent-resource"]`, + nameUses: 1, + }, + } + for _, tc := range textual { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + if n := strings.Count(tc.src, tc.nameExpr); n != tc.nameUses { + t.Errorf("%s appears %d times, want %d", tc.nameExpr, n, tc.nameUses) + } + for _, tok := range fragmentTokens(want["container"], want["volume"]) { + if !slices.ContainsFunc(tok.forms, func(f string) bool { return strings.Contains(tc.src, f) }) { + t.Errorf("missing %s from sidecar.yaml", tok.label) + } + } + }) + } +} + +func kyvernoFragment(t *testing.T, src string) podFragment { + t.Helper() + var policy struct { + Spec struct { + Rules []struct { + Mutate struct { + PatchStrategicMerge struct { + Spec podFragment `yaml:"spec"` + } `yaml:"patchStrategicMerge"` + } `yaml:"mutate"` + } `yaml:"rules"` + } `yaml:"spec"` + } + if err := yaml.Unmarshal([]byte(src), &policy); err != nil { + t.Fatal(err) + } + return policy.Spec.Rules[0].Mutate.PatchStrategicMerge.Spec +} + +func freezeFragment(t *testing.T, src string) podFragment { + t.Helper() + m := regexp.MustCompile(`(?s)read -r -d '' PATCH <= maxDelete { + return false + } + deleted++ + t.Logf("pod %s recreated injected=%v, deleting it again", name, !want) + h.kubectl(t, "delete", "pod", name, "-n", "tailscale", "--wait=false") + return false + } + return true + }) +} + +// rollbackMethod removes one method's injection and returns the ProxyGroup to +// its operator-managed state for the next method. +func (h *kindHarness) rollbackMethod(t *testing.T, manifest string) { + t.Helper() + h.kubectlDelete(t, "-f", filepath.Join(h.root, "examples", "kubernetes", manifest)) + h.kubectl(t, "patch", "proxygroup", "ingress", "--type", "merge", + "-p", `{"spec":{"proxyClass":null}}`) + h.restartIngress(t) + h.ensureInjection(t, "ingress", false, 0) +} + +func (h *kindHarness) stsContainers(t *testing.T) []string { + t.Helper() + out := h.kubectl(t, "get", "statefulset", "ingress", "-n", "tailscale", + "-o", "jsonpath={.spec.template.spec.containers[*].name}") + return strings.Fields(out) +} + +// assertSidecarLive waits for the sidecar's collector.started at the +// receiver, which proves it runs and its sink reaches the receiver. +func (h *kindHarness) assertSidecarLive(t *testing.T) { + t.Helper() + h.waitFor(t, 3*time.Minute, "collector.started at the receiver", func() bool { + return strings.Contains(h.receiverEvents(t), `"type":"collector.started"`) + }) +} + +func (h *kindHarness) waitPodContainers(t *testing.T, pod, container string) { + t.Helper() + h.waitFor(t, 3*time.Minute, pod+" recreated with "+container, func() bool { + out := h.kubectl(t, "get", "pod", pod, "-n", "tailscale", + "-o", "jsonpath={.spec.containers[*].name}") + return slices.Contains(strings.Fields(out), container) + }) +} + +// mutationProbePod carries the opt-in labels so the admission chain under +// test mutates it exactly like a pod of a ProxyGroup named probe. +const mutationProbePod = ` +apiVersion: v1 +kind: Pod +metadata: + name: mutation-probe + namespace: tailscale + labels: + tailscale.com/parent-resource-type: proxygroup + tailscale.com/parent-resource: probe + tail-watch/inject: "true" +spec: + containers: + - name: pause + image: registry.k8s.io/pause:3.10 +` + +// waitMutationLive recreates a probe pod until the admission chain adds the +// sidecar to it, proving the injector is serving before the ProxyGroup pods +// are recreated. A pod created while the injector is still registering its +// webhook is admitted unchanged and never re-mutated. The probe's ConfigMap +// and Secret must be named after its own ProxyGroup, not ingress. +func (h *kindHarness) waitMutationLive(t *testing.T) { + t.Helper() + h.waitFor(t, 3*time.Minute, "admission mutation live", func() bool { + h.kubectlDelete(t, "pod", "mutation-probe", "-n", "tailscale") + h.applyStdin(t, mutationProbePod) + out := h.kubectl(t, "get", "pod", "mutation-probe", "-n", "tailscale", + "-o", "jsonpath={.spec.containers[*].name}") + return slices.Contains(strings.Fields(out), "tail-watch") + }) + names := h.kubectl(t, "get", "pod", "mutation-probe", "-n", "tailscale", "-o", + "jsonpath={.spec.volumes[?(@.name=='tail-watch-config')].configMap.name} "+ + "{.spec.containers[?(@.name=='tail-watch')].envFrom[0].secretRef.name}") + if names != "tail-watch-probe tail-watch-probe" { + t.Fatalf("probe pod ConfigMap and Secret names %q, want tail-watch-probe for both", names) + } + h.kubectlDelete(t, "pod", "mutation-probe", "-n", "tailscale") +} + +// assertStillAdvertising checks the two signals that stay green while hop 2 +// is broken: the workload has Ready endpoints and the operator reports the +// Service advertised. Terminating endpoints, left over from a workload that +// was just recreated, are not part of the claim; an empty slice fails it. +func (h *kindHarness) assertStillAdvertising(t *testing.T) { + t.Helper() + conds := h.kubectl(t, "get", "endpointslice", "-n", "apps", + "-l", "kubernetes.io/service-name=web", "-o", + `jsonpath={range .items[*].endpoints[*]}{.conditions.ready} {.conditions.terminating}{"\n"}{end}`) + ready := 0 + for line := range strings.Lines(conds) { + fields := strings.Fields(line) + if len(fields) == 0 || (len(fields) > 1 && fields[1] == "true") { + continue + } + if fields[0] != "true" { + t.Fatalf("workload endpoint not Ready while hop 2 is broken:\n%s", conds) + } + ready++ + } + if ready == 0 { + t.Fatalf("workload has no Ready endpoints, so hop 2 is not what is broken:\n%s", conds) + } + status := h.kubectl(t, "get", "svc", "web-ts", "-n", "apps", + "-o", "jsonpath={.status.conditions[?(@.type=='TailscaleIngressSvcConfigured')].status}") + msg := h.kubectl(t, "get", "svc", "web-ts", "-n", "apps", + "-o", "jsonpath={.status.conditions[?(@.type=='TailscaleIngressSvcConfigured')].message}") + if status != "True" || !strings.HasSuffix(msg, "proxy backends ready and advertising") { + t.Fatalf("operator condition status=%s message=%q", status, msg) + } +} + +// patchWorkloadWrongPort runs the workload on 8080 with a readiness probe on +// 8080 while the Service's targetPort stays 80: the endpoints stay Ready and +// nothing listens where the Service points. +func (h *kindHarness) patchWorkloadWrongPort(t *testing.T) { + t.Helper() + h.kubectl(t, "patch", "deploy", "web", "-n", "apps", "--type", "strategic", "-p", `{ + "spec": {"template": {"spec": {"containers": [{ + "name": "web", + "image": "python:3.13-alpine", + "command": ["python", "-m", "http.server", "8080"], + "ports": [{"containerPort": 8080}], + "readinessProbe": {"tcpSocket": {"port": 8080}} + }]}}} +}`) + h.kubectl(t, "-n", "apps", "rollout", "status", "deploy/web", "--timeout=600s") +} + +// restoreWorkload recreates the documented workload. A kubectl apply is not +// enough: the strategic-merge patch added command and readinessProbe fields +// the example file does not own, and apply would leave them behind (an nginx +// image with a python command crash-loops). +func (h *kindHarness) restoreWorkload(t *testing.T) { + t.Helper() + h.kubectlDelete(t, "deploy", "web", "-n", "apps") + h.kubectl( + t, + "apply", + "-f", + filepath.Join(h.root, "cmd", "tail-watch", "testdata", "kind", "workload.yaml"), + ) + h.kubectl(t, "-n", "apps", "rollout", "status", "deploy/web", "--timeout=600s") +} + +// receiverEvents port-forwards to the in-cluster receiver and returns every +// event body it has collected. +func (h *kindHarness) receiverEvents(t *testing.T) string { + t.Helper() + port := freePort(t) + cmd := h.command("kubectl", "-n", "apps", "port-forward", "svc/alert-receiver", + fmt.Sprintf("%d:8080", port)) + var stderr bytes.Buffer + cmd.Stderr = &stderr + if err := cmd.Start(); err != nil { + t.Fatalf("port-forward: %v", err) + } + defer func() { _ = cmd.Process.Kill(); _ = cmd.Wait() }() + client := &http.Client{Timeout: 2 * time.Second} + var body string + for range 40 { + resp, err := client.Get(fmt.Sprintf("http://127.0.0.1:%d/events", port)) + if err == nil { + b, readErr := io.ReadAll(resp.Body) + _ = resp.Body.Close() + if readErr == nil && resp.StatusCode == http.StatusOK { + body = string(b) + break + } + } + time.Sleep(250 * time.Millisecond) + } + return body +} + +// waitEvent waits until the receiver holds an event of eventType beyond the +// first since bytes it already held, so an event from an earlier subtest does +// not satisfy a later one. +func (h *kindHarness) waitEvent(t *testing.T, since int, eventType string) { + t.Helper() + h.waitFor(t, 3*time.Minute, eventType+" at the receiver", func() bool { + events := h.receiverEvents(t) + if len(events) < since { + return false + } + return strings.Contains(events[since:], `"type":"`+eventType+`"`) + }) +} + +func (h *kindHarness) waitFor(t *testing.T, timeout time.Duration, what string, cond func() bool) { + t.Helper() + deadline := time.Now().Add(timeout) + for time.Now().Before(deadline) { + if cond() { + return + } + time.Sleep(2 * time.Second) + } + t.Fatalf("timed out waiting for %s", what) +} diff --git a/cmd/tail-watch/main.go b/cmd/tail-watch/main.go index 4ad5c57..cc41f62 100644 --- a/cmd/tail-watch/main.go +++ b/cmd/tail-watch/main.go @@ -146,10 +146,14 @@ func run(args []string, stdout, stderr io.Writer) int { _, _ = fmt.Fprintf(stderr, "%v\n", dynErr) return 1 } - a.bindResolver() - if preflightErr := a.preflight(context.Background()); preflightErr != nil { - _, _ = fmt.Fprintf(stderr, "%v\n", preflightErr) - return 1 + if cfg.Collector.LocalOnly { + a.localOnlyIdentity() + } else { + a.bindResolver() + if preflightErr := a.preflight(context.Background()); preflightErr != nil { + _, _ = fmt.Fprintf(stderr, "%v\n", preflightErr) + return 1 + } } if *onceFlag { return a.runOnce() @@ -177,12 +181,16 @@ func registerUIFlag(fs *flag.FlagSet) **bool { return &value } -// resolveUI decides whether the monitor UI is on. --once ignores it, and it -// needs a state_dir to keep the monitors it creates. +// resolveUI decides whether the monitor UI is on. --once ignores it, it is +// unavailable in local-only mode, and it needs a state_dir to keep the +// monitors it creates. func resolveUI(cfg *Config, flag *bool, once bool) (bool, error) { if once || !cfg.UIEnabled(flag) { return false, nil } + if cfg.Collector.LocalOnly { + return false, errUILocalOnly + } if cfg.Collector.StateDir == "" { return false, errors.New("the UI needs collector.state_dir to store monitors") } @@ -309,11 +317,14 @@ func (a *app) schedule(ctx context.Context, p *prober.Prober) []func() error { } func (a *app) startCollectors(ctx context.Context, p *prober.Prober) *runtimeRegistry { - a.presence = newPresence(a.log, a.clock, a.watchBus) if a.ui { a.cfg.mergeDynamic(a.dynamic, a.log) } - a.bindResolver() + // presence stays nil in local-only mode; its methods are nil-safe. + if !a.cfg.Collector.LocalOnly { + a.presence = newPresence(a.log, a.clock, a.watchBus) + a.bindResolver() + } targets := make([]*Target, len(a.cfg.Targets)) for i := range a.cfg.Targets { targets[i] = &a.cfg.Targets[i] @@ -352,7 +363,9 @@ func (a *app) startCollectors(ctx context.Context, p *prober.Prober) *runtimeReg a.notify.Start(ctx) } go a.presence.run(ctx) - go a.learner.run(ctx) + if !a.cfg.Collector.LocalOnly { + go a.learner.run(ctx) + } for _, t := range targets { if addErr := reg.AddTarget(t); addErr != nil && a.log != nil { a.log.ErrorContext(ctx, "add target", "target", t.Name, "err", addErr) @@ -500,6 +513,17 @@ func (a *app) logf(format string, args ...any) { a.log.Info(fmt.Sprintf(format, args...)) } +// localOnlyIdentity sets the payload identity from the hostname and logs it. +// Local-only mode never contacts tailscaled, so there is no Tailscale IP. +func (a *app) localOnlyIdentity() { + host, err := os.Hostname() + if err != nil { + a.log.Warn("hostname unavailable", "err", err) + } + a.collector = CollectorInfo{Name: host, TailscaleIP: ""} + a.log.Info("collector identity", "hostname", host, "local_only", true) +} + func (a *app) preflight(ctx context.Context) error { st, err := a.env.LocalClient().Status(ctx) if err != nil { diff --git a/cmd/tail-watch/main_test.go b/cmd/tail-watch/main_test.go index 2225312..b0f489e 100644 --- a/cmd/tail-watch/main_test.go +++ b/cmd/tail-watch/main_test.go @@ -10,18 +10,23 @@ import ( "log/slog" "net" "net/http" + "net/http/httptest" "net/netip" "os" "os/exec" "path/filepath" "runtime" "strings" + "sync" + "sync/atomic" "syscall" "testing" "time" + "tailscale.com/client/local" "tailscale.com/ipn" "tailscale.com/ipn/ipnstate" + "tailscale.com/tstime" "tailscale.com/types/views" ) @@ -602,6 +607,230 @@ targets: } } +// countingLocalEnv counts LocalClient lookups so a local-only run can prove +// it never reaches for the LocalAPI. +type countingLocalEnv struct { + netEnv + + calls *atomic.Int32 +} + +func (e countingLocalEnv) LocalClient() *local.Client { + e.calls.Add(1) + return e.netEnv.LocalClient() +} + +//nolint:paralleltest // localOnlyIdentity reads the process hostname and runOnce installs signal handlers. +func TestLocalOnlyRuntime(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = io.WriteString(w, "ok\n") + })) + t.Cleanup(srv.Close) + port := srv.Listener.Addr().(*net.TCPAddr).Port + cfg, err := parseConfig([]byte(fmt.Sprintf(` +collector: {local_only: true} +targets: + - {name: web, addr: 127.0.0.1, interval: 1s, timeout: 500ms, checks: ['tcp:%d']} +`, port))) + if err != nil { + t.Fatal(err) + } + var calls atomic.Int32 + missing := filepath.Join(t.TempDir(), "tailscaled.sock") + var stdout bytes.Buffer + a := &app{ + cfg: cfg, + env: countingLocalEnv{netEnv: newKernelEnv(missing), calls: &calls}, + log: slog.New(slog.DiscardHandler), + clock: tstime.StdClock{}, + pub: newPublisher(), + stdout: &stdout, + socket: missing, + listen: "127.0.0.1:0", + ui: false, + + dynamic: nil, + registry: nil, + resolver: nil, + presence: nil, + class: nil, + learner: nil, + alerts: nil, + notify: nil, + metrics: nil, + watch: nil, + probeMu: sync.Mutex{}, + probes: nil, + + collector: CollectorInfo{Name: "", TailscaleIP: ""}, + } + a.localOnlyIdentity() + host, hostErr := os.Hostname() + if hostErr != nil { + t.Fatal(hostErr) + } + if a.collector.Name != host || a.collector.TailscaleIP != "" { + t.Fatalf("identity %+v, want hostname %q and no Tailscale IP", a.collector, host) + } + if code := a.runOnce(); code != 0 { + t.Fatalf("runOnce code %d stdout %s", code, stdout.String()) + } + if !strings.Contains(stdout.String(), "web tcp:") || !strings.Contains(stdout.String(), " ok ") { + t.Fatalf("stdout %s", stdout.String()) + } + if got := calls.Load(); got != 0 { + t.Fatalf("LocalAPI client requested %d times in local-only mode", got) + } +} + +//nolint:gocognit // Independent CLI scenarios keep their setup next to the assertions. +func TestLocalOnlyCLI(t *testing.T) { + t.Parallel() + bin := buildTailWatch(t) + missing := filepath.Join(t.TempDir(), "tailscaled.sock") + + ln, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = ln.Close() }) + go func() { + for { + c, acceptErr := ln.Accept() + if acceptErr != nil { + return + } + _ = c.Close() + } + }() + openPort := ln.Addr().(*net.TCPAddr).Port + closed, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + closedPort := closed.Addr().(*net.TCPAddr).Port + _ = closed.Close() + + localCfg := func(port int) string { + return writeCfg(t, fmt.Sprintf(` +collector: {local_only: true} +targets: + - name: loop + addr: 127.0.0.1 + interval: 200ms + timeout: 100ms + checks: ["tcp:%d"] +`, port)) + } + + t.Run("check-config rejects tsping", func(t *testing.T) { + t.Parallel() + path := writeCfg(t, ` +collector: {local_only: true} +targets: + - {name: nas, addr: 100.64.0.1, checks: [tsping]} +`) + _, errOut, code := runBin(t, bin, nil, "check-config", "--config", path) + if code != 1 || !strings.Contains(errOut, "tsping is not valid in local-only mode") { + t.Fatalf("code %d stderr %s", code, errOut) + } + }) + t.Run("ui flag rejected", func(t *testing.T) { + t.Parallel() + path := writeCfg(t, fmt.Sprintf(` +collector: {local_only: true, state_dir: %s} +targets: + - {name: loop, addr: 127.0.0.1, checks: ["tcp:%d"]} +`, t.TempDir(), openPort)) + _, errOut, code := runBin(t, bin, nil, "--config", path, "--ui") + if code != 1 || !strings.Contains(errOut, "the UI is unavailable in local-only mode") { + t.Fatalf("code %d stderr %s", code, errOut) + } + }) + t.Run("once pass without tailscaled", func(t *testing.T) { + t.Parallel() + out, errOut, code := runBin(t, bin, nil, "--once", "--config", localCfg(openPort), "--socket", missing) + if code != 0 { + t.Fatalf("code %d\nstdout %s\nstderr %s", code, out, errOut) + } + if !strings.Contains(out, "loop tcp:") || !strings.Contains(out, " ok ") { + t.Fatalf("stdout %s", out) + } + }) + t.Run("once fail without tailscaled", func(t *testing.T) { + t.Parallel() + out, errOut, code := runBin(t, bin, nil, "--once", "--config", localCfg(closedPort), "--socket", missing) + if code != 1 { + t.Fatalf("code %d\nstdout %s\nstderr %s", code, out, errOut) + } + if !strings.Contains(out, "fail") || !strings.Contains(out, "connection refused") { + t.Fatalf("stdout %s", out) + } + }) + t.Run("server schedules checks without tailscaled", func(t *testing.T) { + t.Parallel() + listen := freePort(t) + cmd := exec.Command(bin, "--config", localCfg(openPort), "--socket", missing, + "--listen", fmt.Sprintf("127.0.0.1:%d", listen)) + var stderr syncBuffer + cmd.Stderr = &stderr + if startErr := cmd.Start(); startErr != nil { + t.Fatal(startErr) + } + t.Cleanup(func() { _ = cmd.Process.Kill(); _ = cmd.Wait() }) + metrics := fmt.Sprintf("http://127.0.0.1:%d/metrics", listen) + deadline := time.Now().Add(10 * time.Second) + for !strings.Contains(httpBody(metrics), `tailwatch_target_state{target="loop"} 1`) { + if time.Now().After(deadline) { + t.Fatalf("loop never reached up\nmetrics %s\nstderr %s", httpBody(metrics), stderr.String()) + } + time.Sleep(100 * time.Millisecond) + } + host, hostErr := os.Hostname() + if hostErr != nil { + t.Fatal(hostErr) + } + want := fmt.Sprintf(`"msg":"collector identity","hostname":%q,"local_only":true`, host) + if !strings.Contains(stderr.String(), want) { + t.Fatalf("stderr lacks %s\n%s", want, stderr.String()) + } + if signalErr := cmd.Process.Signal(syscall.SIGTERM); signalErr != nil { + t.Fatal(signalErr) + } + if waitErr := cmd.Wait(); waitErr != nil { + t.Fatalf("exit %v\nstderr %s", waitErr, stderr.String()) + } + }) +} + +// syncBuffer is a [bytes.Buffer] safe to read while a child process writes it. +type syncBuffer struct { + mu sync.Mutex + buf bytes.Buffer +} + +func (b *syncBuffer) Write(p []byte) (int, error) { + b.mu.Lock() + defer b.mu.Unlock() + return b.buf.Write(p) +} + +func (b *syncBuffer) String() string { + b.mu.Lock() + defer b.mu.Unlock() + return b.buf.String() +} + +func httpBody(url string) string { + resp, err := http.Get(url) + if err != nil { + return "" + } + defer resp.Body.Close() + b, _ := io.ReadAll(resp.Body) + return string(b) +} + func runningStatus() *ipnstate.Status { tags := views.SliceOf([]string{"tag:monitor"}) return &ipnstate.Status{ diff --git a/cmd/tail-watch/notifytest.go b/cmd/tail-watch/notifytest.go index 7898512..a98fab2 100644 --- a/cmd/tail-watch/notifytest.go +++ b/cmd/tail-watch/notifytest.go @@ -63,7 +63,13 @@ func runNotifyTestWith(args []string, stdout, stderr io.Writer, deps notifyTestD if deps.env == nil { deps.env = newKernelEnv(socketPath(*socketFlag)) } - if deps.identity == nil { + switch { + case deps.identity != nil: + case cfg.Collector.LocalOnly: + deps.identity = func(context.Context) (CollectorInfo, error) { + return CollectorInfo{}, errors.New("local-only mode has no tailscaled") + } + default: deps.identity = func(ctx context.Context) (CollectorInfo, error) { return localIdentity(ctx, deps.env) } } col := collectorOrHostname(deps.identity) diff --git a/cmd/tail-watch/notifytest_test.go b/cmd/tail-watch/notifytest_test.go index ceb12db..d529c51 100644 --- a/cmd/tail-watch/notifytest_test.go +++ b/cmd/tail-watch/notifytest_test.go @@ -4,7 +4,9 @@ import ( "context" "errors" "net/http" + "path/filepath" "strings" + "sync/atomic" "testing" "time" ) @@ -134,6 +136,34 @@ func TestNotifyTestWorksWithoutTailscaled(t *testing.T) { } } +func TestNotifyTestLocalOnlySkipsLocalAPI(t *testing.T) { + rc := newReceiver(t) + t.Setenv("TW_NT_A", rc.srv.URL) + path := writeCfg( + t, + "collector:\n local_only: true\ntargets:\n - name: nas\n addr: 10.0.0.1\n checks: [icmp]\n"+ + "sinks:\n - {name: hook, type: webhook, url_env: TW_NT_A}\n", + ) + var calls atomic.Int32 + env := countingLocalEnv{netEnv: newKernelEnv(filepath.Join(t.TempDir(), "missing.sock")), calls: &calls} + var out, errOut strings.Builder + if code := runNotifyTestWith( + []string{"--config", path}, + &out, + &errOut, + notifyTestDeps{identity: nil, env: env, clock: nil}, + ); code != 0 { + t.Fatalf("exit %d %s", code, errOut.String()) + } + if n := calls.Load(); n != 0 { + t.Fatalf("local-only notify-test reached the LocalAPI %d times", n) + } + body := string(rc.request(0).Body) + if !strings.Contains(body, `"collector":{"name":"`) || strings.Contains(body, "tailscale_ip") { + t.Fatalf("hostname identity expected, body %s", body) + } +} + func TestNotifyTestUnsetSecretIsReported(t *testing.T) { t.Parallel() path := notifyTestConfig(t, "sinks:\n - {name: hook, type: webhook, url_env: TW_NT_NEVER_SET}\n") diff --git a/cmd/tail-watch/testdata/kind-proxygroup.yaml b/cmd/tail-watch/testdata/kind-proxygroup.yaml new file mode 100644 index 0000000..8fc5bb6 --- /dev/null +++ b/cmd/tail-watch/testdata/kind-proxygroup.yaml @@ -0,0 +1,11 @@ +# kind cluster for TestKindProxyGroup (TAIL_WATCH_KIND=1). The +# MutatingAdmissionPolicy API is v1beta1 and is not served by default on the +# kindest/node v1.35.0 image, so the cluster turns on the feature gate and the +# runtime config. kindnet on this image enforces NetworkPolicy, which the +# hop-2 break in the test relies on; no alternate CNI is installed. +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +featureGates: + MutatingAdmissionPolicy: true +runtimeConfig: + admissionregistration.k8s.io/v1beta1: "true" diff --git a/cmd/tail-watch/testdata/kind/proxygroup.yaml b/cmd/tail-watch/testdata/kind/proxygroup.yaml new file mode 100644 index 0000000..ec20717 --- /dev/null +++ b/cmd/tail-watch/testdata/kind/proxygroup.yaml @@ -0,0 +1,23 @@ +# ingress is the ProxyGroup the test injects the tail-watch sidecar into. It +# has no proxyClass until a doc block from docs/how-to/kubernetes-proxygroup.md +# opts it in. plain never gets the opt-in label and is the negative control: +# its pods must stay exactly as the operator created them. Both use a hostname +# prefix of tw-pg so scripts/kind-stack.sh down can find and delete their +# tailnet devices. +apiVersion: tailscale.com/v1alpha1 +kind: ProxyGroup +metadata: + name: ingress +spec: + type: ingress + replicas: 2 + hostnamePrefix: tw-pg-ingress +--- +apiVersion: tailscale.com/v1alpha1 +kind: ProxyGroup +metadata: + name: plain +spec: + type: ingress + replicas: 1 + hostnamePrefix: tw-pg-plain diff --git a/cmd/tail-watch/testdata/kind/workload.yaml b/cmd/tail-watch/testdata/kind/workload.yaml new file mode 100644 index 0000000..b006784 --- /dev/null +++ b/cmd/tail-watch/testdata/kind/workload.yaml @@ -0,0 +1,152 @@ +# The workload the ProxyGroup forwards to, and the in-cluster receiver the +# sidecar's webhook sink posts to. Nothing here joins the tailnet. +apiVersion: v1 +kind: Namespace +metadata: + name: apps +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: web + namespace: apps +spec: + replicas: 2 + selector: + matchLabels: + app: web + template: + metadata: + labels: + app: web + spec: + containers: + - name: web + image: nginx:1.27-alpine + ports: + - containerPort: 80 +--- +# The ClusterIP Service the sidecar probes. The injected tail-watch config in +# examples/kubernetes/configmap.yaml targets web.apps.svc.cluster.local:80. +apiVersion: v1 +kind: Service +metadata: + name: web + namespace: apps +spec: + selector: + app: web + ports: + - port: 80 + targetPort: 80 +--- +# The L4 Service the operator advertises as Tailscale Service svc:tw-pg-web +# through ProxyGroup ingress. +apiVersion: v1 +kind: Service +metadata: + name: web-ts + namespace: apps + annotations: + tailscale.com/proxy-group: ingress + tailscale.com/hostname: tw-pg-web +spec: + type: LoadBalancer + loadBalancerClass: tailscale + selector: + app: web + ports: + - port: 80 + targetPort: 80 +--- +# alert-receiver stores every POST body and returns them all on GET /events, +# one per line, so the test can assert target.down and target.recovered +# without any external sink. +apiVersion: v1 +kind: ConfigMap +metadata: + name: alert-receiver + namespace: apps +data: + server.py: | + from http.server import BaseHTTPRequestHandler, HTTPServer + + bodies = [] + + class H(BaseHTTPRequestHandler): + def do_POST(self): + n = int(self.headers.get("Content-Length") or 0) + bodies.append(self.rfile.read(n).decode()) + self.send_response(200) + self.end_headers() + self.wfile.write(b"ok") + + def do_GET(self): + if self.path == "/events": + body = "\n".join(bodies).encode() + self.send_response(200) + self.send_header("Content-Type", "text/plain") + self.end_headers() + self.wfile.write(body) + elif self.path == "/healthz": + self.send_response(200) + self.end_headers() + self.wfile.write(b"ok") + else: + self.send_response(404) + self.end_headers() + + def log_message(self, *args): + pass + + HTTPServer(("0.0.0.0", 8080), H).serve_forever() +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: alert-receiver + namespace: apps +spec: + replicas: 1 + selector: + matchLabels: + app: alert-receiver + template: + metadata: + labels: + app: alert-receiver + spec: + securityContext: + runAsUser: 65534 + runAsGroup: 65534 + runAsNonRoot: true + containers: + - name: receiver + image: python:3.13-alpine + command: ["python", "/app/server.py"] + ports: + - containerPort: 8080 + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + volumeMounts: + - name: app + mountPath: /app + readOnly: true + volumes: + - name: app + configMap: + name: alert-receiver +--- +apiVersion: v1 +kind: Service +metadata: + name: alert-receiver + namespace: apps +spec: + selector: + app: alert-receiver + ports: + - port: 8080 + targetPort: 8080 diff --git a/cmd/tail-watch/testdata/snippet.hujson b/cmd/tail-watch/testdata/snippet.hujson index a7b19f6..873e113 100644 --- a/cmd/tail-watch/testdata/snippet.hujson +++ b/cmd/tail-watch/testdata/snippet.hujson @@ -5,6 +5,17 @@ { "tagOwners": { "tag:monitor": ["autogroup:admin"], + // The Kubernetes operator and the ProxyGroup devices it creates, for + // TestKindProxyGroup. + "tag:k8s-operator": ["autogroup:admin"], + "tag:k8s": ["tag:k8s-operator"], + }, + "autoApprovers": { + // Let the operator's ProxyGroups advertise Tailscale Services without a + // manual approval per Service. + "services": { + "svc:*": ["tag:k8s"], + }, }, "grants": [ // The peer the test discovers is a member device. The grant also puts diff --git a/docs/README.md b/docs/README.md index cb06c9b..105ded2 100644 --- a/docs/README.md +++ b/docs/README.md @@ -15,6 +15,8 @@ need to do. and Compose without a dedicated host. - [Write the ACLs](how-to/acls.md): grants for `tag:monitor`. - [Monitor remote subnets and local LANs](how-to/remote-subnets.md) +- [Watch a Kubernetes ProxyGroup](how-to/kubernetes-proxygroup.md): an opt-in + sidecar in the operator's proxy pods. - [Manage monitors in the UI](how-to/monitor-ui.md) - [Send alerts to Slack](how-to/slack-sink.md) - [Send alerts to a webhook](how-to/webhook-sink.md): signing, batching, retries. @@ -35,10 +37,13 @@ need to do. - [Alert state](explanation/alert-state.md) - [Incidents and queues](explanation/incidents-and-queues.md) - [Monitor UI risks](explanation/monitor-ui-risks.md) +- [ProxyGroup sidecar](explanation/proxygroup-sidecar.md): why Kubernetes + misses a broken hop 2, and what the sidecar cannot see. ## Contributing - [Live notification test](contributing/live-notification-test.md) +- [kind ProxyGroup test](contributing/kind-proxygroup-test.md) - [VM systemd test](contributing/vm-systemd-test.md) - [Monitor UI browser tests](contributing/ui-browser-test.md) - [Development and releases](contributing/releases.md) diff --git a/docs/contributing/kind-proxygroup-test.md b/docs/contributing/kind-proxygroup-test.md new file mode 100644 index 0000000..7a1c19c --- /dev/null +++ b/docs/contributing/kind-proxygroup-test.md @@ -0,0 +1,88 @@ +# kind ProxyGroup test + +This page is for contributors who need to run the ProxyGroup sidecar test +against a real cluster and tailnet. + +`TestKindProxyGroup` runs the documented sidecar recipes from +[Watch a Kubernetes ProxyGroup](../how-to/kubernetes-proxygroup.md) on a kind +cluster with the Tailscale Kubernetes operator and an ingress ProxyGroup. It is +off unless `TAIL_WATCH_KIND=1`, and it skips when `kind`, `kubectl`, `helm` or +`docker` is missing. For each injection method (MutatingAdmissionPolicy, Kyverno, +freeze-and-patch) it applies the how-to's bash blocks, then breaks hop 2 — a +NetworkPolicy denying the operator namespace, and a readiness probe on a port +other than `targetPort` — while the Service's endpoints stay `Ready` and the +operator keeps advertising, and expects the sidecar's `target.down` and +`target.recovered` at an in-cluster webhook receiver. Before each +admission-time method touches the proxy pods, a probe pod labelled as +ProxyGroup `probe` must come back with ConfigMap and Secret `tail-watch-probe`, +which proves the names follow the pod's ProxyGroup rather than `ingress`. + +The unit test `TestSidecarCopiesAgree` needs no cluster: it checks every +embedded copy of the sidecar against `examples/kubernetes/sidecar.yaml`, so +change that file and the copies together. + +```bash +TAIL_WATCH_KIND=1 go test -count=1 -timeout 30m -run TestKindProxyGroup -v ./cmd/tail-watch +``` + +## Prerequisites + +- `kind`, `kubectl`, `helm` and a running Docker daemon. +- The cluster config is `cmd/tail-watch/testdata/kind-proxygroup.yaml`. The + MutatingAdmissionPolicy API is v1beta1 and is not served by default on the + kindest/node v1.35.0 image, so the config turns on the + `MutatingAdmissionPolicy` feature gate and the + `admissionregistration.k8s.io/v1beta1` runtime config. +- kindnet on the v1.35.0 node image enforces NetworkPolicy (verified: a + deny-all ingress policy blocks pod-to-pod traffic), so no alternate CNI is + installed. If a future node image changes that, the hop-2 NetworkPolicy + break stops working and the test fails at the `target.down` assertion. +- The fixtures are in `cmd/tail-watch/testdata/kind/`: an nginx workload and + its ClusterIP Service `web.apps.svc.cluster.local`, the L4 Service `web-ts` + advertised as Tailscale Service `svc:tw-pg-web`, the ProxyGroups `ingress` + (the test subject) and `plain` (the negative control), and the + `alert-receiver` webhook receiver the sidecar's sink posts to. + +## Tailnet policy and credentials + +The operator needs the same policy as a production install: the +`tag:k8s-operator` and `tag:k8s` tagOwners and the Tailscale Services +auto-approver in `cmd/tail-watch/testdata/snippet.hujson`, merged into the +test tailnet policy. + +Credentials come from the `tailscale-phase-2/tailscale` pulumi env +(`TAIL_WATCH_PULUMI_ENV` overrides it), next to `tailscale:apiKey`: + +- `tailscale:operatorClientId` and `tailscale:operatorClientSecret`: an OAuth + client with Devices Core, Auth Keys and Services write scopes, tagged + `tag:k8s-operator`. The operator chart is pinned to the newest chart version + at or below the `tailscale.com` version in `go.mod`. +- `tailscale:apiKey` and `tailscale:tailnet`: teardown deletes the Tailscale + Service `svc:tw-pg-web` and every device whose hostname starts with `tw-pg`. + +The test and the helper script never print or persist these values; the OAuth +secret reaches Helm through a mode-0600 values file that is deleted +afterwards. + +## The harness by hand + +`scripts/kind-stack.sh` does everything except run the scenarios: + +```bash +scripts/kind-stack.sh up # cluster, operator, fixtures, Ready ProxyGroup +scripts/kind-stack.sh down # cluster, Tailscale Service and tw-pg devices gone +``` + +`TAIL_WATCH_KIND_CLUSTER` overrides the cluster name +(`tail-watch-proxygroup`), `TAIL_WATCH_KIND_IMAGE` the image built and loaded +(`tail-watch:0.1.0`). The test builds the same harness itself when the cluster +is missing, so a bare `go test` run works; `kind-stack.sh up` is for iterating +on the scenarios without paying cluster startup each time. + +## Cleanup + +The test leaves the harness running so the next run can reuse it. Tear it down +with `scripts/kind-stack.sh down`, and rerun it if a failed run leaves tailnet +state behind: it deletes the kind cluster, the Tailscale Service and any device +with the `tw-pg` hostname prefix, which covers the operator's proxies +(`tw-pg-ingress-*`, `tw-pg-plain-*`). diff --git a/docs/explanation/failure-layers.md b/docs/explanation/failure-layers.md index 1404cad..faf469a 100644 --- a/docs/explanation/failure-layers.md +++ b/docs/explanation/failure-layers.md @@ -82,6 +82,14 @@ no handling node to consult. Presence, routes and disco do not apply, and the only layers available are `service` (the device answered in some way) and `lan` (it did not). +Local checks can also run where there is no tailscaled at all. With +`collector.local_only`, tail-watch skips the LocalAPI and the Kubernetes +ProxyGroup sidecar uses this to probe workload Services from the proxy pod's +network namespace. Those failures are always `service` or `lan` — the pod has +no tailnet layers to blame — which is exactly the claim the sidecar makes: +the path from proxy to workload is broken. See +[Watch a Kubernetes ProxyGroup](../how-to/kubernetes-proxygroup.md). + ## What the classifier knows, and what it does not The classifier uses the tailnet's own view of itself. That view comes from diff --git a/docs/explanation/proxygroup-sidecar.md b/docs/explanation/proxygroup-sidecar.md new file mode 100644 index 0000000..ba60f74 --- /dev/null +++ b/docs/explanation/proxygroup-sidecar.md @@ -0,0 +1,48 @@ +# ProxyGroup sidecar + +A Tailscale Kubernetes operator ingress ProxyGroup can advertise a Tailscale +Service while traffic from the proxy pod to the workload is broken. This page +explains why Kubernetes does not notice, what a tail-watch sidecar in the +proxy pod adds, and what it still cannot see. + +To install the sidecar, see +[Watch a Kubernetes ProxyGroup](../how-to/kubernetes-proxygroup.md). The +`collector.local_only` key it relies on is in the +[configuration reference](../reference/config.md#local-only-mode). + +## Why Kubernetes does not notice hop 2 + +An ingress ProxyGroup has two hops: hop 1 from the tailnet client to the +proxy pod, and hop 2 from the proxy pod to the workload Service. Kubernetes +health signals do not cover hop 2: + +- Containerboot's `/healthz` reports healthy when the node has tailnet + addresses. It says nothing about the path to the workload. +- The operator advertises an L4 Tailscale Service as soon as any + EndpointSlice endpoint is `Ready`. Readiness is the kubelet's view from the + workload's node, not reachability from the proxy pod. + +So a NetworkPolicy that drops the proxy's traffic, a readiness probe on a +different port than the Service's `targetPort`, or a broken CNI path on one +proxy's node all leave the Service advertised while clients cannot use it. + +A tail-watch sidecar in the proxy pod probes the workload Services from the +pod's own network namespace, which is exactly the vantage point hop 2 needs. + +## What a crash-looping sidecar does + +A sidecar that keeps exiting marks its proxy pod NotReady. The kind test +measured what follows: the Service stays advertised, because the operator's +advertising gate ignores pod readiness and `ProxyGroupReady` stays `True`. A +StatefulSet rolling update, though, stalls at the affected pod until the +sidecar runs again, because the update waits for each pod to be Ready. That is +why the how-to validates the config before applying it. + +## What the sidecar cannot see + +The sidecar's own connections leave the pod through OUTPUT, not through the +PREROUTING DNAT that forwarded tailnet traffic takes. A broken containerboot +forwarding rule is invisible to it. Cover hop 1 — tailnet client to proxy, +including the DNAT — end to end with a regular tail-watch collector probing +the Service's tailnet name, as in +[Monitor remote subnets and local LANs](../how-to/remote-subnets.md). diff --git a/docs/how-to/kubernetes-proxygroup.md b/docs/how-to/kubernetes-proxygroup.md new file mode 100644 index 0000000..75c5e70 --- /dev/null +++ b/docs/how-to/kubernetes-proxygroup.md @@ -0,0 +1,341 @@ +# Watch a Kubernetes ProxyGroup from inside the proxy pod + +A Tailscale Kubernetes operator ingress ProxyGroup can advertise a Tailscale +Service while traffic from the proxy pod to the workload is broken, and +nothing notices. This page runs tail-watch as an opt-in sidecar in the proxy +pods to close that gap, with four injection methods to choose from. + +It assumes the operator is already installed in the `tailscale` namespace and +that you have an ingress ProxyGroup, called `ingress` here, forwarding to a +ClusterIP Service in the `apps` namespace. + +Why Kubernetes misses this, and what the sidecar cannot see, is in +[ProxyGroup sidecar](../explanation/proxygroup-sidecar.md). + +## The contract + +Every injection method below produces the same pod, selected the same way: + +- **Opt-in selector.** The sidecar is added only to pods carrying both + `tailscale.com/parent-resource-type: proxygroup`, which the operator sets, + and `tail-watch/inject: "true"`, which you set through the ProxyGroup's + ProxyClass pod labels. All other pods are admitted unchanged. Removing the + label from the ProxyClass stops injection for pods created afterwards. +- **Per-ProxyGroup config.** The sidecar reads `tail-watch.yaml` from the + ConfigMap named `tail-watch-` in the operator's namespace, + mounted read-only at `/etc/tail-watch`. Each method builds the name from the + pod's `tailscale.com/parent-resource` label, which the operator sets, so one + policy serves every opted-in ProxyGroup. Sink secrets come from the Secret of + the same name through `envFrom` and the existing `*_env` keys, never from the + ConfigMap. +- **Local-only mode.** The config sets + [`collector.local_only`](../reference/config.md#local-only-mode): no + tailscaled, targets default to `network: local`, and the payload identifies + the collector by pod hostname. Do not use `icmp` checks in the sidecar — + the container drops all capabilities, so there is no `NET_RAW`. Use `tcp:` + and `http:` checks only. +- **Container shape.** + [`examples/kubernetes/sidecar.yaml`](../../examples/kubernetes/sidecar.yaml) + is the exact fragment every method embeds: image, args, CPU and memory + requests and limits, a non-root user, a read-only root filesystem, and all + capabilities dropped. +- **Never blocks a proxy pod.** Both volumes and every method's failure + policy are `Ignore` or optional, so opting in can never stop a proxy pod + from starting or withdraw a Service. If the injector fails or the ConfigMap + is missing, the pod starts without monitoring and the `tailscale` container + serves as before; it gets the sidecar when it is next recreated after the + injector recovers. That recreation can take a few tries right after the + injector comes back, while its webhook is still being registered. + +A sidecar that crash-loops on a bad config stalls the ProxyGroup's rolling +updates (see [what a crash-looping sidecar +does](../explanation/proxygroup-sidecar.md#what-a-crash-looping-sidecar-does)), +so run `tail-watch check-config` on the ConfigMap contents before applying: + +```bash +kubectl get configmap tail-watch-ingress -n tailscale \ + -o jsonpath='{.data.tail-watch\.yaml}' | tail-watch check-config --config /dev/stdin +``` + +## Build the image + +The project does not publish to a registry yet. Build the image from the +repository at the release tag and push it somewhere the cluster can pull: + +```bash +git checkout v0.1.0 +docker build -t tail-watch:0.1.0 . +``` + +For a local kind cluster, `kind load docker-image tail-watch:0.1.0` is +enough. The manifests use `imagePullPolicy: IfNotPresent`. + +## MutatingAdmissionPolicy + +Version floor: `admissionregistration.k8s.io/v1beta1` MutatingAdmissionPolicy +exists from Kubernetes 1.34 (beta). It is not served by default everywhere — +on the kindest/node v1.35.0 image the API server must start with the +MutatingAdmissionPolicy feature gate and the +`admissionregistration.k8s.io/v1beta1` runtime config (the kind config in this +repository enables both). On 1.32–1.33, enable the same feature gate and the +v1alpha1 (1.32) or v1beta1 (1.33) API, and change the manifest's `apiVersion` +to match. No external component runs, and +`failurePolicy: Ignore` means a failed mutation admits the pod unchanged, so +proxy pods always start. + +The policy matches Pod CREATE and its `matchConditions` select only opted-in +ProxyGroup pods that carry the `tailscale.com/parent-resource` label. Apply +the manifests and opt the ProxyGroup in: + +```bash +kubectl apply -f examples/kubernetes/proxyclass.yaml +kubectl apply -f examples/kubernetes/secret.yaml +kubectl apply -f examples/kubernetes/configmap.yaml +kubectl apply -f examples/kubernetes/mutatingadmissionpolicy.yaml +kubectl patch proxygroup ingress --type merge \ + -p '{"spec":{"proxyClass":"tail-watch"}}' +kubectl rollout restart statefulset ingress -n tailscale +kubectl rollout status statefulset ingress -n tailscale +``` + +Verify the sidecar is there and the StatefulSet template is not: + +```bash +kubectl get pod ingress-0 -n tailscale -o jsonpath='{.spec.containers[*].name}' +# tailscale tail-watch +kubectl get statefulset ingress -n tailscale \ + -o jsonpath='{.spec.template.spec.containers[*].name}' +# tailscale +``` + +## Kyverno + +Version floor: any supported Kyverno v1 (`kyverno.io/v1` ClusterPolicy; the +recipe is tested with chart kyverno-3.9.1, which deprecates the API in favour +of the CEL-based `policies.kyverno.io` MutatingPolicy but still enforces it). +`failurePolicy: Ignore` admits pods unchanged when Kyverno fails, and +`background: false` keeps the rule on the admission path only. + +The annotation `pod-policies.kyverno.io/autogen-controllers: "none"` is +required: Kyverno's autogen would otherwise generate a rule that mutates the +ProxyGroup's StatefulSet, which the operator overwrites on every reconcile. +The rule mutates Pod CREATE only: + +```bash +helm repo add kyverno https://kyverno.github.io/kyverno/ --force-update +helm upgrade --install kyverno kyverno/kyverno -n kyverno --create-namespace --wait +kubectl apply -f examples/kubernetes/proxyclass.yaml +kubectl apply -f examples/kubernetes/secret.yaml +kubectl apply -f examples/kubernetes/configmap.yaml +kubectl apply -f examples/kubernetes/kyverno-clusterpolicy.yaml +kubectl patch proxygroup ingress --type merge \ + -p '{"spec":{"proxyClass":"tail-watch"}}' +kubectl rollout restart statefulset ingress -n tailscale +kubectl rollout status statefulset ingress -n tailscale +``` + +Verify as above: + +```bash +kubectl get pod ingress-0 -n tailscale -o jsonpath='{.spec.containers[*].name}' +# tailscale tail-watch +``` + +## Custom webhook + +Version floor: any cluster with `admissionregistration.k8s.io/v1`, which is +every supported Kubernetes release. No injector image ships and this project +does not build or test one; the code below is a teaching aid for writing your +own. + +The webhook answers an AdmissionReview request for a Pod CREATE with an +`allowed: true` response carrying a base64 JSON patch that appends the +sidecar container and volume. Two guards keep proxy pods starting: the +`objectSelector` restricts calls to pods with both opt-in labels, and +`failurePolicy: Ignore` admits the pod unchanged if the webhook is down. + +```yaml +apiVersion: admissionregistration.k8s.io/v1 +kind: MutatingWebhookConfiguration +metadata: + name: tail-watch-sidecar +webhooks: + - name: tail-watch-sidecar.example.com + admissionReviewVersions: ["v1"] + sideEffects: None + failurePolicy: Ignore + reinvocationPolicy: Never + objectSelector: + matchLabels: + tailscale.com/parent-resource-type: proxygroup + tail-watch/inject: "true" + matchExpressions: + - key: tailscale.com/parent-resource + operator: Exists + clientConfig: + service: + name: tail-watch-injector + namespace: tailscale + path: /mutate + port: 443 + caBundle: "" # PEM of the CA that signed the serving certificate + rules: + - apiGroups: [""] + apiVersions: ["v1"] + operations: ["CREATE"] + resources: ["pods"] +``` + +TLS is your responsibility. With cert-manager, issue a `Certificate` for +`tail-watch-injector.tailscale.svc` and copy its CA into `caBundle` (or use a +`caInjector` annotation). Manually, create a serving certificate signed by a +CA you control — a `CertificateSigningRequest` with signer +`kubernetes.io/kubelet-serving` does not work here; generate one with your +own CA or the cluster's — and base64 the CA into `caBundle`. + +The handler, listening on 8443 behind that Service: + +```go +// Teaching aid for docs/how-to/kubernetes-proxygroup.md. Not shipped. +package main + +import ( + "encoding/json" + "log" + "net/http" + + admissionv1 "k8s.io/api/admission/v1" + corev1 "k8s.io/api/core/v1" +) + +// sidecar and volume are examples/kubernetes/sidecar.yaml with the ConfigMap +// and Secret named tail-watch-. +func sidecar(name string) map[string]any { + return map[string]any{ + "name": "tail-watch", + "image": "tail-watch:0.1.0", + "imagePullPolicy": "IfNotPresent", + "args": []string{"--config", "/etc/tail-watch/tail-watch.yaml", + "--listen", "127.0.0.1:9109"}, + "envFrom": []any{map[string]any{ + "secretRef": map[string]any{"name": name, "optional": true}}}, + "volumeMounts": []any{map[string]any{ + "name": "tail-watch-config", "mountPath": "/etc/tail-watch", "readOnly": true}}, + "resources": map[string]any{ + "requests": map[string]any{"cpu": "10m", "memory": "32Mi"}, + "limits": map[string]any{"cpu": "100m", "memory": "64Mi"}}, + "securityContext": map[string]any{ + "runAsUser": 65532, "runAsGroup": 65532, + "readOnlyRootFilesystem": true, "allowPrivilegeEscalation": false, + "capabilities": map[string]any{"drop": []string{"ALL"}}}, + } +} + +func volume(name string) map[string]any { + return map[string]any{ + "name": "tail-watch-config", + "configMap": map[string]any{"name": name, "optional": true}, + } +} + +func mutate(w http.ResponseWriter, r *http.Request) { + var review admissionv1.AdmissionReview + if err := json.NewDecoder(r.Body).Decode(&review); err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + resp := &admissionv1.AdmissionResponse{UID: review.Request.UID, Allowed: true} + + var pod corev1.Pod + if err := json.Unmarshal(review.Request.Object.Raw, &pod); err == nil { + name := "tail-watch-" + pod.Labels["tailscale.com/parent-resource"] + ops := []map[string]any{ + {"op": "add", "path": "/spec/containers/-", "value": sidecar(name)}, + } + if len(pod.Spec.Volumes) == 0 { + ops = append(ops, map[string]any{ + "op": "add", "path": "/spec/volumes", "value": []any{volume(name)}}) + } else { + ops = append(ops, map[string]any{ + "op": "add", "path": "/spec/volumes/-", "value": volume(name)}) + } + patch, err := json.Marshal(ops) + if err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + // Patch is []byte, so encoding/json base64-encodes it on the wire. + resp.Patch = patch + pt := admissionv1.PatchTypeJSONPatch + resp.PatchType = &pt + } + + review.Response = resp + review.Request = nil + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(review) +} + +func main() { + http.HandleFunc("/mutate", mutate) + log.Fatal(http.ListenAndServeTLS(":8443", "/tls/tls.crt", "/tls/tls.key", nil)) +} +``` + +The `objectSelector` already restricts calls to opted-in ProxyGroup pods that +carry `tailscale.com/parent-resource`, so the handler can trust that every +request it sees wants the sidecar and names its ProxyGroup. Deploy it +with the same opt-in, ConfigMap and rollback contract as the other methods. + +## Freeze and patch (evaluation only) + +No version floor. This method scales the operator to zero and patches the +ProxyGroup's StatefulSet directly — the one thing the other methods avoid — +which works only because nothing reconciles while the operator is down. That +is also why it is evaluation only: while the operator is scaled to zero, +ProxyGroups, Services and Ingresses stop reacting to changes and the proxy +pods run unmanaged. Use it to try the sidecar on a test cluster: + +```bash +kubectl apply -f examples/kubernetes/proxyclass.yaml +kubectl apply -f examples/kubernetes/secret.yaml +kubectl apply -f examples/kubernetes/configmap.yaml +examples/kubernetes/freeze-and-patch.sh ingress +kubectl rollout status statefulset ingress -n tailscale +kubectl get pod ingress-0 -n tailscale -o jsonpath='{.spec.containers[*].name}' +# tailscale tail-watch +``` + +The script prints its own undo: scale the operator back to 1. The next +reconcile overwrites the whole StatefulSet spec, which removes the sidecar +(see [Roll back](#roll-back)). + +## What it costs + +Each replica probes every configured Service, so check load scales with +replicas × Services ÷ interval. The default interval is 30s, and the requests +and limits in +[`examples/kubernetes/sidecar.yaml`](../../examples/kubernetes/sidecar.yaml) +bound what opting in costs per pod. Only the ProxyGroups that use the opt-in +ProxyClass pay it. The sidecar does not cover hop 1; see [what the sidecar +cannot see](../explanation/proxygroup-sidecar.md#what-the-sidecar-cannot-see). + +## Roll back + +Remove the label from the ProxyClass (or point the ProxyGroup at another +ProxyClass) and delete the policy you applied. Existing pods keep the +sidecar until they are recreated; new pods are admitted unchanged: + +```bash +kubectl patch proxygroup ingress --type merge \ + -p '{"spec":{"proxyClass":null}}' +kubectl delete -f examples/kubernetes/mutatingadmissionpolicy.yaml +kubectl rollout restart statefulset ingress -n tailscale +``` + +For freeze-and-patch, scaling the operator back to 1 both unfreezes it and +removes the sidecar at the next reconcile: + +```bash +kubectl scale deployment operator -n tailscale --replicas=1 +``` diff --git a/docs/how-to/troubleshoot.md b/docs/how-to/troubleshoot.md index a311ca7..7fac0a9 100644 --- a/docs/how-to/troubleshoot.md +++ b/docs/how-to/troubleshoot.md @@ -18,6 +18,10 @@ print to stderr and exit 1; log lines are JSON on stderr. | `sink : url: environment variable is unset or empty` (also `secret:`, `header :`, `heartbeat:`) | A `*_env` variable is unset or empty at startup. Secret files report `secret file is empty` | Export the variable in the service's environment. Compose and systemd do not inherit your shell. Run `notify-test` to confirm. See [Slack](slack-sink.md) and [Heartbeat](heartbeat.md) | | `notification state is corrupt (...); repair or move the file aside and restart, but note open incidents are forgotten if it is removed` | `notification-state.json` in `collector.state_dir` cannot be decoded, so startup stops rather than forget open incidents | Restore or repair the file from a backup. If you move it aside, open incidents are forgotten and their recoveries are not sent. See [Incidents and queues](../explanation/incidents-and-queues.md) | | `the UI needs collector.state_dir to store monitors` | `--ui` or `collector.ui: true` is set with no `state_dir` | Set `collector.state_dir`, or turn the UI off with `--ui=false`. See [Monitor UI](monitor-ui.md) | +| `the UI is unavailable in local-only mode` (from `--ui` or `collector.ui`) | The UI assumes a tailnet and a state directory, which a local-only collector does not have | Drop the UI; the status page and `/metrics` still serve on `collector.listen`. See [Local-only mode](../reference/config.md#local-only-mode) | +| `network "tailnet" is not valid in local-only mode` | `collector.local_only: true` is set and a target asks for the tailnet | Remove `network: tailnet`; without `network` the target defaults to `local`. See [Local-only mode](../reference/config.md#local-only-mode) | +| `tsping is not valid in local-only mode` | A local-only config lists a `tsping` check, which needs tailscaled | Use `icmp`, `tcp:`, or `http:` checks. See [Local-only mode](../reference/config.md#local-only-mode) | +| `"msg":"collector identity","hostname":"","local_only":true` (informational) | `collector.local_only: true` is set, so tail-watch starts without tailscaled and identifies itself by hostname | Nothing. Payloads carry `collector.name` and omit `collector.tailscale_ip`. See [Payload](../reference/payload.md) | | `queue: disk requires collector.state_dir` | A sink uses the disk queue with no `state_dir` | Set `collector.state_dir` or use `queue: memory` | | `dynamic monitors are inactive: the UI is off` (a warning) | `targets.json` has monitors but the UI is off, so they do not run | Turn the UI on, or ignore it | | `webhook sink is unsigned: no secret_env or secret_file` (a warning) | A webhook sink has no HMAC key | Add `secret_env`. See [Webhook](webhook-sink.md) | diff --git a/docs/reference/cli.md b/docs/reference/cli.md index ced1cba..0a66ec6 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -14,7 +14,7 @@ Without a subcommand, tail-watch runs the collector. | --- | --- | --- | --- | | `-config` | string | see below | Config file path. Overrides `TAIL_WATCH_CONFIG`. | | `-listen` | string | `collector.listen`, else `127.0.0.1:9109` | HTTP listen address. | -| `-socket` | string | platform default | tailscaled LocalAPI socket. | +| `-socket` | string | platform default | tailscaled LocalAPI socket. Ignored when `collector.local_only` is `true`. | | `-log-level` | string | `info` | `debug`, `info`, `warn`, or `error`. Overrides `TAIL_WATCH_LOG_LEVEL`. | | `-once` | bool | `false` | Run every check once, print one line each, and exit. | | `-ui` | bool | `collector.ui`, else `false` | Serve the monitor UI and API. `--ui=false` turns off a config that enables it. | @@ -61,7 +61,8 @@ no secrets, sends no heartbeat, and ignores the UI. Before serving, the collector fails with exit code 1 when: - the config file cannot be read or is invalid; -- the UI is on and `collector.state_dir` is unset; +- the UI is on and `collector.state_dir` is unset, or the UI is on with + `collector.local_only: true`; - tailscaled is unreachable, or its state is not `Running`; - tailscaled uses userspace networking (kernel mode is required); - a target is a remote subnet address on Linux and the host has not accepted @@ -69,6 +70,10 @@ Before serving, the collector fails with exit code 1 when: - a sink or heartbeat secret cannot be read or is empty (server mode only); - the notification state file cannot be decoded (server mode only). +With `collector.local_only: true`, every tailscaled check above is skipped: the +collector makes no LocalAPI call, logs its hostname and local-only mode, and +schedules its checks. + It logs a warning, and continues, when the node lacks `tag:monitor`, when an exit node is active, when a webhook sink has no secret, and when the UI is reachable beyond loopback. @@ -112,7 +117,7 @@ omitted. | `-config` | string | see [Precedence](#precedence) | Config file path. | | `-sink` | string | every sink | Send only to the sink with this name. | | `-timeout` | duration | `30s` | Give up on a sink after this long, including retries. | -| `-socket` | string | platform default | tailscaled LocalAPI socket, used to read the collector identity. | +| `-socket` | string | platform default | tailscaled LocalAPI socket, used to read the collector identity. Ignored when `collector.local_only` is `true`. | Sinks are tried in parallel. One line is printed per sink, in config order: diff --git a/docs/reference/config.md b/docs/reference/config.md index 8f75253..c347ec5 100644 --- a/docs/reference/config.md +++ b/docs/reference/config.md @@ -124,6 +124,7 @@ heartbeat: | `listen` | string | no | `127.0.0.1:9109` | HTTP listen address, `host:port`. `--listen` overrides it. | | `state_dir` | string | no | unset | Directory for durable state. Created with mode `0700` when missing. | | `ui` | bool | no | `false` | Serve the monitor UI and API. `--ui` overrides it, and `--ui=false` turns it off. | +| `local_only` | bool | no | `false` | Run without tailscaled. See below. | `state_dir` holds: @@ -136,6 +137,35 @@ heartbeat: `state_dir` is required when `ui` is on (startup exits with an error otherwise) and when any sink sets `queue: disk`. `--once` ignores `ui`. +### Local-only mode + +With `local_only: true`, tail-watch runs without tailscaled: it makes no +LocalAPI call, skips the startup preflight, and takes the collector identity +from the hostname, so payloads carry `collector.name` and omit +`collector.tailscale_ip`. This mode is for running next to something that is +not a tailnet device, such as the ProxyGroup sidecar in +[Watch a Kubernetes ProxyGroup](../how-to/kubernetes-proxygroup.md). + +Local-only mode changes what the loader accepts: + +- A target without `network` defaults to `local` instead of `tailnet`. +- A target with `network: tailnet` is rejected. +- A `tsping` check is rejected. +- `ui: true` is rejected, and so is the `--ui` flag. + +Each problem is reported with its key path, and `check-config` applies the same +rules. A minimal local-only config: + +```yaml +collector: + local_only: true + +targets: + - name: web + addr: web.apps.svc.cluster.local + checks: [tcp:443] +``` + ## `defaults` Each key is the fallback for the same key on every target, including monitors diff --git a/docs/reference/payload.md b/docs/reference/payload.md index 2c2019c..0381438 100644 --- a/docs/reference/payload.md +++ b/docs/reference/payload.md @@ -36,7 +36,7 @@ failing at startup produces one `target.down`. | `late` | boolean | any, when true | The first attempt happened more than one minute after `time`. The original `time` is kept. | | `collector` | object | all | The sending collector. | | `collector.name` | string | all | The node's hostname. | -| `collector.tailscale_ip` | string | when known | The node's first Tailscale IPv4 address, or IPv6 when it has none. | +| `collector.tailscale_ip` | string | when known; omitted in local-only mode | The node's first Tailscale IPv4 address, or IPv6 when it has none. | | `target` | object | target events | The affected target. | | `target.name` | string | target events | Target name. | | `target.addr` | string | target events | Target address. | diff --git a/examples/kubernetes/configmap.yaml b/examples/kubernetes/configmap.yaml new file mode 100644 index 0000000..b19eeac --- /dev/null +++ b/examples/kubernetes/configmap.yaml @@ -0,0 +1,33 @@ +# Sidecar config for ProxyGroup "ingress" in the tailscale namespace. +# The name must be tail-watch-; the injected container mounts +# key tail-watch.yaml read-only at /etc/tail-watch. One ConfigMap per opted-in +# ProxyGroup. +# +# Validate the embedded config before applying: +# tail-watch check-config --config <(kubectl get configmap tail-watch-ingress \ +# -n tailscale -o jsonpath='{.data.tail-watch\.yaml}') +apiVersion: v1 +kind: ConfigMap +metadata: + name: tail-watch-ingress + namespace: tailscale +data: + tail-watch.yaml: | + # local_only runs with no tailscaled. Targets default to network: local, + # checks use the pod's own network namespace, and identity is the pod + # hostname. Do not add icmp checks: the container has no NET_RAW. + collector: + local_only: true + + targets: + - name: web + addr: web.apps.svc.cluster.local + site: apps + checks: [tcp:80] + + # Secrets come from the Secret mounted by envFrom, never from this file. + sinks: + - name: ops-webhook + type: webhook + url_env: OPS_WEBHOOK_URL + secret_env: OPS_WEBHOOK_SECRET diff --git a/examples/kubernetes/freeze-and-patch.sh b/examples/kubernetes/freeze-and-patch.sh new file mode 100755 index 0000000..aa06372 --- /dev/null +++ b/examples/kubernetes/freeze-and-patch.sh @@ -0,0 +1,95 @@ +#!/usr/bin/env bash +# EVALUATION ONLY. +# +# This freezes the Tailscale operator (scales it to zero) and patches the +# ProxyGroup's StatefulSet directly with the tail-watch sidecar. While the +# operator is scaled to zero it reconciles nothing: ProxyGroups, Services and +# Ingresses stop reacting to changes, and existing proxy pods keep running +# unmanaged. Use this to evaluate the sidecar on a test cluster, not in +# production. The manifest-driven methods in +# docs/how-to/kubernetes-proxygroup.md mutate Pod CREATE instead and leave +# the operator running. +# +# Usage: freeze-and-patch.sh +# +# Undo: scale the operator back to 1. Its next reconcile overwrites the whole +# StatefulSet spec, which removes the sidecar: +# kubectl scale deployment operator -n tailscale --replicas=1 +set -euo pipefail + +usage() { + echo "usage: $0 " >&2 + echo "example: $0 ingress" >&2 +} + +if [ "$#" -ne 1 ]; then + usage + exit 2 +fi + +PG="$1" +NS="tailscale" +OPERATOR_DEPLOYMENT="operator" +STS="${PG}" + +# The same container and volume as examples/kubernetes/sidecar.yaml, as a +# strategic merge patch. Containers and volumes merge by name. +read -r -d '' PATCH <, taken + # from the label the operator sets. A strategic merge patch merges + # containers and volumes by name, so this appends without touching the + # pod's other settings. + patchStrategicMerge: + spec: + containers: + - name: tail-watch + image: tail-watch:0.1.0 + imagePullPolicy: IfNotPresent + args: + - --config + - /etc/tail-watch/tail-watch.yaml + - --listen + - 127.0.0.1:9109 + envFrom: + - secretRef: + name: 'tail-watch-{{ request.object.metadata.labels."tailscale.com/parent-resource" }}' + optional: true + volumeMounts: + - name: tail-watch-config + mountPath: /etc/tail-watch + readOnly: true + resources: + requests: + cpu: 10m + memory: 32Mi + limits: + cpu: 100m + memory: 64Mi + securityContext: + runAsUser: 65532 + runAsGroup: 65532 + readOnlyRootFilesystem: true + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + volumes: + - name: tail-watch-config + configMap: + name: 'tail-watch-{{ request.object.metadata.labels."tailscale.com/parent-resource" }}' + optional: true diff --git a/examples/kubernetes/mutatingadmissionpolicy.yaml b/examples/kubernetes/mutatingadmissionpolicy.yaml new file mode 100644 index 0000000..7d9c570 --- /dev/null +++ b/examples/kubernetes/mutatingadmissionpolicy.yaml @@ -0,0 +1,134 @@ +# Inject the tail-watch sidecar with a MutatingAdmissionPolicy (CEL mutation +# inside the API server, no external component). +# +# The version floor and the feature gate it may need are in +# docs/how-to/kubernetes-proxygroup.md; on 1.32-1.33 use the apiVersion that +# page names. +# +# failurePolicy: Ignore means a failed mutation admits the pod unchanged, so a +# proxy pod can never be blocked by this policy; it starts without the +# sidecar. The policy mutates Pod CREATE only and never touches the +# operator-owned StatefulSet. +apiVersion: admissionregistration.k8s.io/v1beta1 +kind: MutatingAdmissionPolicy +metadata: + name: tail-watch-sidecar +spec: + failurePolicy: Ignore + matchConstraints: + resourceRules: + - apiGroups: [""] + apiVersions: ["v1"] + operations: ["CREATE"] + resources: ["pods"] + matchConditions: + - name: is-proxygroup-pod + expression: >- + object.metadata.?labels["tailscale.com/parent-resource-type"].orValue("") + == "proxygroup" + - name: is-opted-in + expression: >- + object.metadata.?labels["tail-watch/inject"].orValue("") == "true" + - name: has-proxygroup-name + expression: >- + object.metadata.?labels["tailscale.com/parent-resource"].orValue("") != "" + reinvocationPolicy: Never + mutations: + - patchType: JSONPatch + jsonPatch: + # The same container and volume as examples/kubernetes/sidecar.yaml. + # ConfigMap and Secret names are tail-watch-, taken + # from the label the operator sets. The patch appends to spec.containers and + # spec.volumes without touching the pod's other settings. Apply + # configurations cannot set atomic lists such as args and envFrom, + # which is why this is a JSON patch. + expression: | + [ + JSONPatch{ + op: "add", + path: "/spec/containers/-", + value: Object.spec.containers{ + name: "tail-watch", + image: "tail-watch:0.1.0", + imagePullPolicy: "IfNotPresent", + args: [ + "--config", "/etc/tail-watch/tail-watch.yaml", + "--listen", "127.0.0.1:9109" + ], + envFrom: [ + Object.spec.containers.envFrom{ + secretRef: Object.spec.containers.envFrom.secretRef{ + name: "tail-watch-" + object.metadata.labels["tailscale.com/parent-resource"], + optional: true + } + } + ], + volumeMounts: [ + Object.spec.containers.volumeMounts{ + name: "tail-watch-config", + mountPath: "/etc/tail-watch", + readOnly: true + } + ], + resources: Object.spec.containers.resources{ + requests: { + "cpu": "10m", + "memory": "32Mi" + }, + limits: { + "cpu": "100m", + "memory": "64Mi" + } + }, + securityContext: Object.spec.containers.securityContext{ + runAsUser: 65532, + runAsGroup: 65532, + readOnlyRootFilesystem: true, + allowPrivilegeEscalation: false, + capabilities: Object.spec.containers.securityContext.capabilities{ + drop: ["ALL"] + } + } + } + } + ] + (has(object.spec.volumes) && size(object.spec.volumes) > 0 + ? [ + JSONPatch{ + op: "add", + path: "/spec/volumes/-", + value: Object.spec.volumes{ + name: "tail-watch-config", + configMap: Object.spec.volumes.configMap{ + name: "tail-watch-" + object.metadata.labels["tailscale.com/parent-resource"], + optional: true + } + } + } + ] + : [ + JSONPatch{ + op: "add", + path: "/spec/volumes", + value: [ + Object.spec.volumes{ + name: "tail-watch-config", + configMap: Object.spec.volumes.configMap{ + name: "tail-watch-" + object.metadata.labels["tailscale.com/parent-resource"], + optional: true + } + } + ] + } + ]) +--- +# A policy does nothing until a binding activates it. Empty matchResources +# matches every namespace; the policy itself is already scoped to Pod CREATE +# and its matchConditions limit mutation to opted-in ProxyGroup pods, so all +# other pods are admitted unchanged. +apiVersion: admissionregistration.k8s.io/v1beta1 +kind: MutatingAdmissionPolicyBinding +metadata: + name: tail-watch-sidecar +spec: + policyName: tail-watch-sidecar + matchResources: {} diff --git a/examples/kubernetes/proxyclass.yaml b/examples/kubernetes/proxyclass.yaml new file mode 100644 index 0000000..58a4f6b --- /dev/null +++ b/examples/kubernetes/proxyclass.yaml @@ -0,0 +1,17 @@ +# Opt a ProxyGroup in to the tail-watch sidecar. +# ProxyClass can set pod labels but not containers, so this label is the +# opt-in signal: every injection method in docs/how-to/kubernetes-proxygroup.md +# adds the sidecar only to pods carrying both this label and the operator's +# own tailscale.com/parent-resource-type: proxygroup label. +# Reference it from a ProxyGroup with: +# kubectl patch proxygroup --type merge \ +# -p '{"spec":{"proxyClass":"tail-watch"}}' +apiVersion: tailscale.com/v1alpha1 +kind: ProxyClass +metadata: + name: tail-watch +spec: + statefulSet: + pod: + labels: + tail-watch/inject: "true" diff --git a/examples/kubernetes/secret.yaml b/examples/kubernetes/secret.yaml new file mode 100644 index 0000000..45daf00 --- /dev/null +++ b/examples/kubernetes/secret.yaml @@ -0,0 +1,15 @@ +# PLACEHOLDER VALUES ONLY. Replace them before relying on alerts. +# Sink secrets for the tail-watch sidecar of ProxyGroup "ingress". The +# container reads them through envFrom; the *_env keys in the ConfigMap name +# these variables. check-config does not resolve them, but the collector +# refuses to start when they are unset or empty. The envFrom reference is +# optional, so a missing Secret never blocks the pod. +apiVersion: v1 +kind: Secret +metadata: + name: tail-watch-ingress + namespace: tailscale +type: Opaque +stringData: + OPS_WEBHOOK_URL: "https://example.invalid/REPLACE-ME" + OPS_WEBHOOK_SECRET: "REPLACE-ME" diff --git a/examples/kubernetes/sidecar.yaml b/examples/kubernetes/sidecar.yaml new file mode 100644 index 0000000..289f207 --- /dev/null +++ b/examples/kubernetes/sidecar.yaml @@ -0,0 +1,54 @@ +# The tail-watch sidecar, exactly as every injection method in +# docs/how-to/kubernetes-proxygroup.md produces it. This file is the single +# copy shown for reference; each method's manifest (MutatingAdmissionPolicy, +# Kyverno ClusterPolicy, the webhook's JSON patch, freeze-and-patch) embeds +# the same container and volumes; TestSidecarCopiesAgree fails when one +# drifts. Do not apply this file directly; it is a fragment, not a resource. +# +# The ConfigMap and Secret names are tail-watch-: each method +# builds them from the pod's tailscale.com/parent-resource label, which the +# operator sets. They are shown for a ProxyGroup named "ingress". Both +# references are optional, so +# a missing ConfigMap or Secret never blocks the proxy pod: the tailscale +# container starts and tail-watch exits with "config not found". +containers: + - name: tail-watch + # The project does not publish to a registry yet. Build at the release + # tag (docker build -t tail-watch:0.1.0 .) and push it somewhere the + # cluster can pull, or kind load it for a local test. + image: tail-watch:0.1.0 + imagePullPolicy: IfNotPresent + args: + - --config + - /etc/tail-watch/tail-watch.yaml + # Metrics and the status page stay inside the pod. + - --listen + - 127.0.0.1:9109 + envFrom: + - secretRef: + name: tail-watch-ingress + optional: true + volumeMounts: + - name: tail-watch-config + mountPath: /etc/tail-watch + readOnly: true + resources: + requests: + cpu: 10m + memory: 32Mi + limits: + cpu: 100m + memory: 64Mi + securityContext: + runAsUser: 65532 + runAsGroup: 65532 + readOnlyRootFilesystem: true + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL +volumes: + - name: tail-watch-config + configMap: + name: tail-watch-ingress + optional: true diff --git a/openspec/changes/archive/2026-10-01-add-proxygroup-sidecar/.openspec.yaml b/openspec/changes/archive/2026-10-01-add-proxygroup-sidecar/.openspec.yaml new file mode 100644 index 0000000..480ad4d --- /dev/null +++ b/openspec/changes/archive/2026-10-01-add-proxygroup-sidecar/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-10-01 diff --git a/openspec/changes/archive/2026-10-01-add-proxygroup-sidecar/design.md b/openspec/changes/archive/2026-10-01-add-proxygroup-sidecar/design.md new file mode 100644 index 0000000..c4e0f8c --- /dev/null +++ b/openspec/changes/archive/2026-10-01-add-proxygroup-sidecar/design.md @@ -0,0 +1,199 @@ +# Design + +## Context + +See proposal.md (Why) for the blind spot. These facts come from `tailscale.com@v1.102.5`, +the module version `go.mod` pins, and shape the approach: + +- Containerboot's `/healthz` reports healthy when the node has tailnet addresses + (`cmd/containerboot/main.go`, `healthCheck.Update(len(addrs) != 0)`). +- For an L4 Service on an ingress ProxyGroup, the operator advertises the Tailscale + Service when any EndpointSlice endpoint is `Ready` (`checkEndpointsReady` in + `cmd/k8s-operator/svc-for-pg.go`). It also needs the replica's forwarding config + to be current, which it judges from the pod's IPs and the config status in the + replica's Secret (`backendRoutesSetup`, `isCurrentStatus`). Pod readiness plays no + part. The L7 Ingress path (`ingress-for-pg.go`) has no endpoint gate at all. +- The operator replaces the whole StatefulSet spec on every reconcile + (`proxygroup.go`: `s.Spec = ss.Spec` inside `createOrUpdate`). A container added to + the StatefulSet is gone on the next reconcile. +- ProxyClass can set pod labels, annotations, affinity, node selection, tolerations, + and the `tailscale` and init containers, but not extra containers or volumes + (`types_proxyclass.go`, type `Pod`). +- ProxyGroup pods carry `tailscale.com/managed: "true"`, + `tailscale.com/parent-resource-type: proxygroup` and + `tailscale.com/parent-resource: ` (`pgLabels`). +- Containerboot's tailscaled socket defaults to `/tmp/tailscaled.sock` inside the + `tailscale` container, which other containers cannot reach without a shared volume. +- tail-watch refuses to start unless tailscaled is reachable, `Running` and in kernel + mode. It already classifies `network: local` failures from device evidence alone, + with no presence, route or disco data (failure-classification, "Layer decision + order"). + +## Goals / Non-Goals + +**Goals:** + +- One sidecar contract (opt-in label, ConfigMap name, container shape) that every + injection method produces, so the admin's choice of method changes nothing else. +- Opting in can never stop a proxy pod from starting or serving, or withdraw a Service. +- Every documented recipe runs in CI-like conditions on kind, so the docs cannot drift + from what works. + +**Non-Goals:** + +- Changing the `tailscale` container, containerboot or the operator. +- Checks of hop 1 from inside the pod. The existing tailnet collector covers it from + outside. +- An L7 Ingress-specific check type. An `http:` check against the Ingress backend + Service covers it with today's checks. + +## Decisions + +### Local-only mode instead of sharing containerboot's tailscaled + +The sidecar's hop-2 checks are `tcp:` and `http:` checks on cluster Services from the +pod's network namespace. They need no LocalAPI. `collector.local_only: true` skips the +LocalAPI entirely, so the sidecar needs nothing from the `tailscale` container. + +Alternative: mount a shared `emptyDir` over the socket directory and set `TS_SOCKET`. +Rejected: it modifies the operator's `tailscale` container, which the isolation +requirement forbids. It would also require kernel-mode tailscaled in ProxyGroup pods, +which is unverified, and it buys nothing hop-2 checks use. + +In local-only mode, a target without `network` defaults to `local`, so a sidecar +config does not repeat `network: local` on every target. `network: tailnet`, `tsping` +and the UI are rejected. The UI is refused because its address picker and dynamic +monitors assume a tailnet and a state directory, which are both meaningless here. +Identity is `os.Hostname()`, which is the pod name in Kubernetes. The payload already +marks `collector.tailscale_ip` "when known", so no schema change is needed. + +### Inject into Pods, never the StatefulSet + +Every method mutates Pod `CREATE` only. Mutating the StatefulSet would fight the +operator's full-spec overwrite. Pods are recreated from the operator's template and +re-injected, so injection survives reconciles and operator upgrades. + +### Opt-in through a ProxyClass pod label + +The selector is `tailscale.com/parent-resource-type: proxygroup` plus +`tail-watch/inject: "true"`. ProxyClass already sets pod labels, so the admin opts a +ProxyGroup in or out without any operator change, and only the ProxyGroups that use +that ProxyClass pay the resource cost. A cluster-wide namespace label was rejected +because it opts in every ProxyGroup at once. + +### Per-ProxyGroup ConfigMap, mounted optional + +The container mounts ConfigMap `tail-watch-`, key `tail-watch.yaml`, +with `optional: true`. Every method builds the name from the pod's +`tailscale.com/parent-resource` label and only mutates pods that carry it, since a +name of `tail-watch-` alone is invalid and would fail the pod. A required ConfigMap volume that is missing blocks the whole pod +in `ContainerCreating`, `tailscale` container included. An optional one lets the pod +start, and tail-watch exits with its existing "config not found" error. Sink secrets +use the existing `*_env` keys filled from a Secret by `envFrom`, also optional. + +### Container shape + +The sidecar is a regular container appended to `spec.containers`, named `tail-watch`. +It uses the release image with a pinned tag, `--config /etc/tail-watch/tail-watch.yaml`, +and `--listen 127.0.0.1:9109` so the metrics and status page stay inside the pod. It +sets requests of 10m CPU and 32Mi memory and limits of 100m CPU and 64Mi memory. It runs as uid +65532, the image's user, with `readOnlyRootFilesystem` and no capabilities. TCP and HTTP +checks need none, and the docs tell the admin not to use `icmp`. A Kubernetes native +sidecar (restartable init container) was considered. It is rejected because the +ordering it buys does not matter here and it needs a newer minimum version. + +### Four methods, one contract + +| Method | Mechanism | Failure isolation | Ships | +| --- | --- | --- | --- | +| MutatingAdmissionPolicy | CEL mutation in the API server, `matchConstraints` on Pod `CREATE`, label `matchConditions` | No external component. `failurePolicy: Ignore` | manifest | +| Kyverno | `ClusterPolicy`, `mutate.patchStrategicMerge`, `pod-policies.kyverno.io/autogen-controllers: none` | `failurePolicy: Ignore`, Pod-only match | manifest | +| Custom webhook | `MutatingWebhookConfiguration` with `objectSelector` on both labels and `tailscale.com/parent-resource`, `sideEffects: None`, JSON patch response | `failurePolicy: Ignore`, and `objectSelector` keeps other pods from calling it | docs and example Go handler | +| Freeze and patch | Scale the operator to 0 and `kubectl patch` the StatefulSet | n/a, evaluation only | manifest patch | + +Each recipe states its version floor. The MutatingAdmissionPolicy floor is the +Kubernetes release where the API is served. The task list checks it on the kind node +image instead of assuming it. Kyverno's autogen annotation is set explicitly, because +autogen would otherwise copy the rule onto the StatefulSet and fight the operator. The +custom webhook's example handler lives only in the how-to, as a teaching aid. It is not +built, released or tested, which matches the user's choice. + +### kind live test runs the documented recipes + +`TestKindProxyGroup` is gated on `TAIL_WATCH_KIND=1` and skips when `kind`, `kubectl` +or `helm` is missing. + +1. Create a kind cluster from `cmd/tail-watch/testdata/kind-proxygroup.yaml`, which also + turns on MutatingAdmissionPolicy if the node image's Kubernetes needs it. +2. Install the operator chart, pinned to the `tailscale.com` version in `go.mod`, with + the OAuth client from the pulumi env. +3. `kind load` a locally built tail-watch image. +4. Create an ingress ProxyGroup, a workload, and an L4 Service on that ProxyGroup. +5. Deploy an in-cluster webhook receiver. The sidecar's sink posts to it and the test + reads it, with no external secrets. +6. For each manifest method in turn (MutatingAdmissionPolicy, Kyverno, + freeze-and-patch), apply the fenced blocks and manifests from the how-to. The block + extraction follows `systemdDocBlocks` in `vm_systemd_test.go`. +7. Assert that the pods have the `tail-watch` container and the StatefulSet does not. +8. Break hop 2 with a NetworkPolicy, then with a readiness probe on the wrong port. Each + time, assert that EndpointSlices stay `Ready`, that the operator condition reads + "N/N proxy backends ready and advertising", and that `target.down` arrives, then + `target.recovered` once the fix is applied. +9. Teardown deletes the cluster, the Tailscale Service and the operator's devices. + +### Test tailnet prerequisites + +The operator needs an OAuth client with device, auth key and services write scopes, +and policy for `tag:k8s-operator`, `tag:k8s` and Tailscale Service approval. The client +goes into the `tailscale-phase-2/tailscale` pulumi env next to `apiKey`, and the test +reads it the same way `live-stack.sh` reads credentials. The policy snippet joins +`cmd/tail-watch/testdata/snippet.hujson`. The test never writes or logs the secret. + +## Risks / Trade-offs + +- [A crash-looping sidecar marks the proxy pod NotReady, which can stall the + StatefulSet's rolling updates during operator upgrades] → Advertising is unaffected, + because the gate ignores pod readiness. The docs tell admins to run + `tail-watch check-config` before applying the ConfigMap. The kind test measures + whether an exiting sidecar changes the ProxyGroup's `Ready` condition or blocks a + rollout, and the docs state the result. +- [The sidecar's own connections traverse OUTPUT, not the PREROUTING DNAT that + forwarded traffic takes, so a broken containerboot rule is not seen] → Documented as + the residual gap. Hop 1 and the DNAT are covered end to end by a tailnet collector + probing the Service's tailnet name. +- [A webhook with `failurePolicy: Fail` would block proxy pods while it is down] → + Every recipe uses `Ignore`, and the docs explain why and what is lost: pods start + without the sidecar. +- [Operator changes rename labels or change reconcile behavior] → The chart is pinned + to the module version, and the kind test fails loudly when the selector or the + "StatefulSet does not contain it" assertion breaks. +- [Each replica probes every Service, so cost scales with replicas × Services ÷ + interval] → Documented. Requests and limits are explicit, and the default interval + stays 30s. +- [The kind test creates real devices and a Tailscale Service on the test tailnet] → + Teardown deletes them, and on failure the devices are cleaned up by their tag or + hostname prefix. + +## Deviations from the design doc + +- "Runtime: kernel-mode tailscaled" requires kernel-mode tailscaled. Local-only mode + needs no tailscaled, because hop-2 checks never cross the tailnet. The kernel-mode + rule still holds whenever any target is `network: tailnet`. +- "Placement" says not to run the collector on the device it monitors, because it + cannot see tunnel failures and goes down with the device. The sidecar runs on the + proxy deliberately. Its subject is hop 2, which only the proxy's namespace sees, and + hop 1 stays with a tailnet collector. + +## Migration Plan + +Additive. `local_only` defaults to `false`, so existing configs and deployments are +unchanged. Admins opt in per ProxyGroup and roll back by removing the ProxyClass label +and the policy. Existing pods keep the sidecar until they are recreated. + +## Open Questions + +- The exact Kubernetes release where MutatingAdmissionPolicy is served by default, for + the version floor in the docs. The test's kind config enables it either way, so the + answer only changes one sentence in the how-to. +- Whether kindnet enforces NetworkPolicy on the kind node image. If it does not, the + test installs Calico, which is a setup detail with the same assertions. diff --git a/openspec/changes/archive/2026-10-01-add-proxygroup-sidecar/proposal.md b/openspec/changes/archive/2026-10-01-add-proxygroup-sidecar/proposal.md new file mode 100644 index 0000000..ddf1907 --- /dev/null +++ b/openspec/changes/archive/2026-10-01-add-proxygroup-sidecar/proposal.md @@ -0,0 +1,90 @@ +# Proposal + +## Why + +A Tailscale Kubernetes operator ingress ProxyGroup advertises a Tailscale Service +while traffic from the proxy to the workload (hop 2) is broken, and nothing notices. +The proxy's `/healthz` only says the node has tailnet addresses. The operator +advertises an L4 Service once any EndpointSlice endpoint is `Ready`. Readiness is +the kubelet's view from the workload's node, not reachability from the proxy pod. +A NetworkPolicy that drops the proxy's traffic, a readiness probe on a different +port from `targetPort`, or a broken CNI path on one proxy's node all leave control +advertising a Service that clients cannot use. tail-watch already runs TCP and HTTP +checks on `network: local` targets and alerts on them. Running it next to the proxy +covers that blind spot without operator changes. A Kubernetes operator maintainer +asked for this, as a step toward a containerboot companion. + +## What Changes + +- Add `collector.local_only`. With it on, tail-watch starts without tailscaled: + it skips the LocalAPI preflight, the peer presence view, dependency learning and + disco probes. It accepts only `network: local` targets and refuses `tsping` + checks and the monitor UI. Alerts carry the hostname as `collector.name` and omit + `collector.tailscale_ip`, which the payload already allows. The default keeps + today's behavior. +- Document running tail-watch as an opt-in sidecar in ProxyGroup proxy pods. The + admin opts a ProxyGroup in by adding a pod label through its ProxyClass. Each + ProxyGroup reads its own ConfigMap, named from the `tailscale.com/parent-resource` + pod label. The sidecar probes the ClusterIP Services the proxy forwards to. +- Document four injection methods for the admin to choose from, because the + operator overwrites its StatefulSet spec on every reconcile and ProxyClass cannot + add containers: + - MutatingAdmissionPolicy, for clusters where it is enabled. + - A Kyverno ClusterPolicy that mutates Pods on create, with autogen off. + - A custom mutating webhook: the AdmissionReview contract, the JSON patch, the + webhook configuration and TLS, with example handler code in the docs. No + injector image ships. + - Freeze and patch: scale the operator to zero and patch the StatefulSet. This is + for evaluation only, because nothing reconciles while the operator is down. + + Every method uses the same opt-in label and ConfigMap name and must not stop + proxy pods from starting when the injector fails. +- Ship example manifests under `examples/kubernetes/` for the methods that apply + manifests. +- Add a kind-based live test, gated like the other live tests. It brings up the + operator and an ingress ProxyGroup. It injects the sidecar with each manifest + method from the docs. It breaks hop 2 in two ways while the Service's endpoints + stay `Ready` and the operator still reports the Service as advertised. It expects + the sidecar to alert on each break. + +## Capabilities + +### New Capabilities + +- `proxygroup-sidecar`: tail-watch as an opt-in sidecar in operator ProxyGroup pods. + Covers opt-in selection, the per-ProxyGroup config, the injection methods and + their failure isolation, and detecting a broken proxy-to-workload hop while the + Service is advertised. + +### Modified Capabilities + +- `collector-config`: a `collector.local_only` key, and validation that limits a + local-only config to `network: local` targets with no `tsping` and no UI. +- `collector-runtime`: in local-only mode, startup no longer requires a reachable, + `Running` tailscaled, and the collector identity comes from the hostname. + +## Impact + +- Design doc sections: "Placement" gains a Kubernetes sidecar placement, and + "Runtime: kernel-mode tailscaled" with "Network access" gain the local-only + exception. "What a failed check means" already classifies `network: local` + failures without tailnet evidence and is unchanged. +- Phase: a phase 1 follow-up. Non-goals, deferred to later phases or other work: + - `tsnetEnv`, `target.degraded`, PagerDuty and other phase 2 items, and phase 3 + quorum. + - Acting on a failure, such as unadvertising the Service from a bad replica. + - An operator setting for sidecars. Both of these are tracked in a separate + issue for the operator. + - A shipped injector image. + - Probing hop 1 (tailnet client to proxy) from the sidecar. + - Deriving targets from containerboot's ingress config file. +- Code: `cmd/tail-watch/config.go` (key and validation), `main.go` (startup, + identity, preflight), plus tests. Checks, the payload schema and metric names do + not change. +- New: `docs/how-to/kubernetes-proxygroup.md`, `examples/kubernetes/`, a kind live + test with its compose-free helper script, and doc updates in `reference/config.md` + and `reference/cli.md`. +- Test prerequisites: kind and Helm, which are installed. The test tailnet also + needs an OAuth client for the operator, which the current pulumi env does not + hold, and policy for the operator's tags and Tailscale Services. +- No new Go dependency. diff --git a/openspec/changes/archive/2026-10-01-add-proxygroup-sidecar/specs/collector-config/spec.md b/openspec/changes/archive/2026-10-01-add-proxygroup-sidecar/specs/collector-config/spec.md new file mode 100644 index 0000000..94463d2 --- /dev/null +++ b/openspec/changes/archive/2026-10-01-add-proxygroup-sidecar/specs/collector-config/spec.md @@ -0,0 +1,63 @@ +# Spec Delta + +## MODIFIED Requirements + +### Requirement: Collector section + +The `collector` section SHALL accept `listen` (host:port for the HTTP server, +default `127.0.0.1:9109`), `state_dir` (optional directory path), `ui` (boolean, +default `false`) and `local_only` (boolean, default `false`). The `--listen` flag SHALL +override `collector.listen`. The `--ui` flag SHALL override `collector.ui`, and +`--ui=false` SHALL turn it off. When the UI is on and `collector.state_dir` is unset, +tail-watch SHALL exit non-zero before contacting tailscaled, with an error saying the UI +needs `collector.state_dir` to store monitors. With `--once`, the UI setting SHALL be +ignored and only YAML targets run. + +When `collector.local_only` is `true`, a target that sets no `network` SHALL default to +`local` instead of `tailnet`, and validation SHALL reject, with every problem listed and +each naming its field: a target with `network: tailnet` (`defaults` has no `network` key), any `tsping` +check, and the UI being on from `collector.ui` or the `--ui` flag. `tail-watch +check-config` SHALL apply the same rules. + +#### Scenario: Default listen address + +- **WHEN** the config omits `collector.listen` and no `--listen` flag is given +- **THEN** the HTTP server listens on `127.0.0.1:9109` + +#### Scenario: UI off by default + +- **WHEN** the config omits `collector.ui` and no `--ui` flag is given +- **THEN** the UI and its API are not served + +#### Scenario: Flag overrides config + +- **WHEN** `collector.ui: false` is set and tail-watch runs with `--ui` +- **THEN** the UI is on + +#### Scenario: UI without a state directory + +- **WHEN** the UI is on and `collector.state_dir` is unset +- **THEN** tail-watch exits non-zero with an error naming `collector.state_dir` + +#### Scenario: One-shot mode ignores the UI + +- **WHEN** tail-watch runs with `--once --ui` +- **THEN** it runs each YAML target's checks once and starts no HTTP server + +#### Scenario: Local-only defaults targets to local + +- **WHEN** `collector.local_only: true` is set and a target sets no `network` +- **THEN** the target loads with `network: local` + +#### Scenario: Local-only rejects tailnet targets and tsping + +- **WHEN** `collector.local_only: true` is set, one target sets `network: tailnet` and + another lists a `tsping` check +- **THEN** validation fails with an error for each, naming the target's `network` field + and the `checks` entry + +#### Scenario: Local-only rejects the UI + +- **WHEN** `collector.local_only: true` is set and tail-watch runs with `--ui` +- **THEN** tail-watch exits non-zero with an error saying the UI is unavailable in + local-only mode diff --git a/openspec/changes/archive/2026-10-01-add-proxygroup-sidecar/specs/collector-runtime/spec.md b/openspec/changes/archive/2026-10-01-add-proxygroup-sidecar/specs/collector-runtime/spec.md new file mode 100644 index 0000000..d099a15 --- /dev/null +++ b/openspec/changes/archive/2026-10-01-add-proxygroup-sidecar/specs/collector-runtime/spec.md @@ -0,0 +1,75 @@ +# Spec Delta + +## MODIFIED Requirements + +### Requirement: LocalAPI connection + +tail-watch SHALL connect to the local tailscaled over its LocalAPI socket, using the +platform default path unless the `--socket` flag names another. It SHALL NOT +embed its own Tailscale node. When `collector.local_only` is `true`, tail-watch SHALL +NOT connect to tailscaled at all and SHALL ignore `--socket`. + +#### Scenario: Custom socket + +- **WHEN** tail-watch runs with `--socket /run/ts/tailscaled.sock` +- **THEN** all LocalAPI calls use that socket + +#### Scenario: Local-only makes no LocalAPI call + +- **WHEN** `collector.local_only: true` is set and no tailscaled runs on the host +- **THEN** tail-watch makes no LocalAPI call and runs its checks + +### Requirement: Startup preflight + +Before scheduling checks, tail-watch SHALL confirm that tailscaled is reachable and +its backend state is `Running`, and SHALL exit non-zero with an actionable +error otherwise. It SHALL log the collector's hostname, Tailscale IPs and tags. +It SHALL log a warning when the node lacks the `tag:monitor` tag, and a warning +when the host uses an exit node, because alerts would then depend on that exit +node. For `network: tailnet` targets behind subnet routers on Linux, preflight +SHALL verify route acceptance is enabled and report an actionable error naming +`tailscale set --accept-routes` otherwise. It SHALL reject userspace-networking +tailscaled for this kernel-mode runtime. A route advertised in presence SHALL +NOT be treated as proof of an installed host route. Deployment documentation +SHALL require checking the effective route for each remote subnet and selecting +`network: local` explicitly for locally monitored LANs. + +When `collector.local_only` is `true`, preflight SHALL skip every tailscaled check +above, SHALL log the hostname and that it runs in local-only mode, and SHALL use the +hostname as the collector identity, so notifications carry `collector.name` and omit +`collector.tailscale_ip`. + +#### Scenario: Subnet routes not accepted + +- **WHEN** a Linux collector has a remote subnet target but does not accept routes +- **THEN** startup fails with the route-acceptance instruction before checks run + +#### Scenario: tailscaled not running + +- **WHEN** the LocalAPI socket does not exist or refuses connections +- **THEN** tail-watch exits non-zero with an error saying tailscaled is not + reachable at the socket path + +#### Scenario: Logged out + +- **WHEN** tailscaled reports backend state `NeedsLogin` +- **THEN** tail-watch exits non-zero with an error telling the operator to run + `tailscale up` + +#### Scenario: Exit node in use + +- **WHEN** the host has an active exit node +- **THEN** tail-watch starts and logs a warning that notifications will route + through the exit node + +#### Scenario: Local-only startup without tailscaled + +- **WHEN** `collector.local_only: true` is set and the LocalAPI socket does not exist +- **THEN** tail-watch starts, logs its hostname and local-only mode, and schedules + its checks + +#### Scenario: Local-only collector identity + +- **WHEN** a local-only collector on host `pg-ingress-0` sends `collector.started` +- **THEN** the payload has `collector.name` `pg-ingress-0` and no + `collector.tailscale_ip` diff --git a/openspec/changes/archive/2026-10-01-add-proxygroup-sidecar/specs/proxygroup-sidecar/spec.md b/openspec/changes/archive/2026-10-01-add-proxygroup-sidecar/specs/proxygroup-sidecar/spec.md new file mode 100644 index 0000000..91da0f7 --- /dev/null +++ b/openspec/changes/archive/2026-10-01-add-proxygroup-sidecar/specs/proxygroup-sidecar/spec.md @@ -0,0 +1,132 @@ +# Spec Delta + +## Purpose + +Runs tail-watch as an opt-in sidecar in Tailscale Kubernetes operator ProxyGroup pods, +so a broken path from the proxy to the workload is detected and alerted on even while +the operator and control report the Tailscale Service as advertised. + +## ADDED Requirements + +### Requirement: Opt-in sidecar selection + +The documented injection SHALL add the tail-watch sidecar only to pods that carry both +`tailscale.com/parent-resource-type: proxygroup`, which the operator sets, and +`tail-watch/inject: "true"`, which the admin sets through the ProxyGroup's ProxyClass +pod labels. Any other pod, including a ProxyGroup pod without the opt-in label, SHALL +be admitted unchanged. Removing the label from the ProxyClass SHALL stop injection +for pods created afterwards. + +#### Scenario: Opted-in ProxyGroup gets the sidecar + +- **WHEN** a ProxyGroup uses a ProxyClass that sets the pod label + `tail-watch/inject: "true"` and its pods are created +- **THEN** each proxy pod runs a `tail-watch` container next to the `tailscale` + container + +#### Scenario: Other pods are untouched + +- **WHEN** a ProxyGroup pod without the opt-in label, or any pod that is not a + ProxyGroup pod, is created +- **THEN** the pod is admitted exactly as submitted + +### Requirement: Per-ProxyGroup configuration + +The injected container SHALL run tail-watch in local-only mode with its config read +from the ConfigMap `tail-watch-` in the pod's namespace, key `tail-watch.yaml`, +where `` is the pod's `tailscale.com/parent-resource` label, mounted read-only. +Targets in that config SHALL name the cluster Services the ProxyGroup forwards to, +for example `web.apps.svc.cluster.local` with `tcp:` or `http:` checks. Secrets for +notification sinks SHALL come from a Kubernetes Secret through environment variables, +using the existing `*_env` keys, and SHALL NOT appear in the ConfigMap. + +#### Scenario: Config follows the ProxyGroup name + +- **WHEN** a pod of ProxyGroup `ingress-a` is injected +- **THEN** its `tail-watch` container reads `tail-watch.yaml` from ConfigMap + `tail-watch-ingress-a` + +#### Scenario: Missing ConfigMap + +- **WHEN** the ConfigMap for an opted-in ProxyGroup does not exist +- **THEN** the pod is still created, the `tail-watch` container reports the missing + config, and the `tailscale` container runs, serves traffic and keeps the Service + advertised as it would without the sidecar + +### Requirement: Proxy isolation + +Injection SHALL change only the pod: it SHALL add the `tail-watch` container and its +config volume, and SHALL NOT modify the `tailscale` container, the pod's other +settings, or the operator-owned StatefulSet. A failure of the injector SHALL NOT stop +proxy pods from being created; such a pod SHALL start without the sidecar. The +documented container SHALL set CPU and memory requests and limits and run as a +non-root user, so the admin can see and bound what opting in costs. + +#### Scenario: Sidecar failure keeps the Service advertised + +- **WHEN** the `tail-watch` container exits or restarts repeatedly +- **THEN** the `tailscale` container keeps serving and the operator keeps the + Service advertised from that pod + +#### Scenario: Injector unavailable + +- **WHEN** the injector of a method that relies on an admission webhook is down and + a ProxyGroup pod is created +- **THEN** the pod is created without the `tail-watch` container + +#### Scenario: Operator reconcile keeps the sidecar + +- **WHEN** the operator reconciles an opted-in ProxyGroup and its pods are recreated +- **THEN** the new pods have the `tail-watch` container again and the StatefulSet + spec does not contain it + +### Requirement: Documented injection methods + +The documentation SHALL describe, with the same opt-in label and ConfigMap contract, +four ways to inject the sidecar so the admin can choose: a MutatingAdmissionPolicy; +a Kyverno ClusterPolicy that mutates Pods on create with policy autogeneration for +pod controllers turned off; a custom mutating admission webhook, giving the +AdmissionReview contract, the JSON patch, the webhook configuration and TLS setup, and +example handler code; and freeze-and-patch, marked as evaluation only because the +operator reconciles nothing while it is scaled to zero. Each method SHALL state the +Kubernetes and component versions it needs and how its failure policy keeps proxy pods +starting. The repository SHALL ship example manifests for the MutatingAdmissionPolicy, +Kyverno and freeze-and-patch methods, and no injector image. + +#### Scenario: Admin picks Kyverno + +- **WHEN** an admin with Kyverno applies the documented ClusterPolicy and opts a + ProxyGroup in +- **THEN** the ProxyGroup's pods get the sidecar and the StatefulSet is not mutated + +#### Scenario: Admin picks a custom webhook + +- **WHEN** an admin builds an injector from the documented contract and example code +- **THEN** it produces the same pod as the other methods for the same opt-in label + and ConfigMap name + +### Requirement: Detect a broken proxy-to-workload hop + +With the sidecar running and the Service still advertised, a target for a cluster +Service SHALL go `DOWN` and send `target.down` when traffic from the proxy pod to the +workload fails, even while the Service's EndpointSlice endpoints are `Ready`. It SHALL +send `target.recovered` when the path works again. Detection SHALL rely only on the +sidecar's own checks, not on operator status, endpoint readiness or the LocalAPI. + +#### Scenario: NetworkPolicy blocks the proxy + +- **WHEN** a NetworkPolicy denies ingress from the operator's namespace to the + workload, its endpoints stay `Ready`, and the operator still reports the Service as + advertised +- **THEN** the sidecar's check for that Service fails and `target.down` is sent + +#### Scenario: Readiness probe on another port + +- **WHEN** the workload's readiness probe passes on one port while nothing listens on + the Service's `targetPort` +- **THEN** the sidecar's check for that Service fails and `target.down` is sent + +#### Scenario: Path restored + +- **WHEN** the blocking NetworkPolicy is removed +- **THEN** the target recovers and `target.recovered` is sent diff --git a/openspec/changes/archive/2026-10-01-add-proxygroup-sidecar/tasks.md b/openspec/changes/archive/2026-10-01-add-proxygroup-sidecar/tasks.md new file mode 100644 index 0000000..cc8000a --- /dev/null +++ b/openspec/changes/archive/2026-10-01-add-proxygroup-sidecar/tasks.md @@ -0,0 +1,138 @@ +# Tasks + +## 1. Local-only config + +- [x] 1.1 Add `local_only` to `CollectorConfig` in `cmd/tail-watch/config.go`: when true, + a target without `network` defaults to `local`, and validation reports every + `network: tailnet` target, every `tsping` check, and `collector.ui: + true`, each naming its field; verify table-driven cases in + `cmd/tail-watch/config_test.go` cover the default-to-local, each rejection, several + problems reported together, and `local_only: false` leaving today's behavior unchanged +- [x] 1.2 Make `tail-watch check-config` apply the same rules and reject `--ui` with + `local_only` at flag resolution in `cmd/tail-watch/main.go`; verify tests in + `cmd/tail-watch/main_test.go` show `check-config` failing on a local-only config with + a `tsping` check and `--ui` failing with the local-only message +- [x] 1.3 Document `collector.local_only` in `docs/reference/config.md` (key table, + defaulting rule, what it rejects) and check the new snippet with + `tail-watch check-config`; verify markdownlint passes on the file + +## 2. Local-only runtime + +- [x] 2.1 In `cmd/tail-watch/main.go`, skip the LocalAPI preflight, presence view, + dependency learner and resolver binding when `local_only` is set; take the collector + identity from `os.Hostname()` with no Tailscale IP; log hostname and local-only mode; + verify a test in `cmd/tail-watch/main_test.go` starts the app with a socket path that + does not exist and gets a scheduled `tcp:` result against an `httptest` listener, and + that no LocalAPI call is made (fake env counts calls) +- [x] 2.2 Make `--once` work in local-only mode without tailscaled; verify a `--once` test + with a nonexistent socket exits 0 for a passing local check and 1 for a failing one +- [x] 2.3 Verify the payload identity: a test in `cmd/tail-watch/event_test.go` (or the + notifier tests) shows `collector.started` from a local-only collector carries + `collector.name` and omits `collector.tailscale_ip`, with no golden-file change +- [x] 2.4 Update `docs/reference/cli.md` (`--socket` ignored in local-only mode) and + `docs/how-to/troubleshoot.md` (local-only startup log line and its rejections); + verify markdownlint passes + +## 3. Test tailnet prerequisites + +- [x] 3.1 Ask the user to add an operator OAuth client (device, auth key and services + write scopes) to the `tailscale-phase-2/tailscale` pulumi env; verify + `pulumi env open` lists the new keys by name only, never printing values +- [x] 3.2 Add the operator's `tagOwners` (`tag:k8s-operator`, `tag:k8s`) and Tailscale + Service approval to `cmd/tail-watch/testdata/snippet.hujson`, and confirm with the + user that the test tailnet policy contains them; verify the file still parses with + the existing snippet test, if any, or `hujson` standardization + +## 4. Sidecar recipes and examples + +- [x] 4.1 Add `examples/kubernetes/`: `proxyclass.yaml` (opt-in label + `tail-watch/inject: "true"`), `configmap.yaml` (a local-only config probing a Service + by cluster DNS name), `secret.yaml` (sink env, placeholders only), and the shared + container spec as used by all methods (name `tail-watch`, pinned image tag, + `--listen 127.0.0.1:9109`, requests and limits, uid 65532, read-only root, optional + ConfigMap and Secret); verify the ConfigMap's config passes `tail-watch check-config` + and the manifests pass `kubectl apply --dry-run=client` +- [x] 4.2 Add `examples/kubernetes/mutatingadmissionpolicy.yaml` (Pod `CREATE`, both + labels in `matchConditions`, `failurePolicy: Ignore`) and + `examples/kubernetes/kyverno-clusterpolicy.yaml` (`patchStrategicMerge`, + `pod-policies.kyverno.io/autogen-controllers: none`, `failurePolicy: Ignore`), each + producing the same container and volumes; verify `kubectl apply --dry-run=server` + against a kind cluster with Kyverno installed +- [x] 4.3 Add `examples/kubernetes/freeze-and-patch.sh` (scale the operator to 0, patch + the ProxyGroup StatefulSet, with an "evaluation only" header); verify `bash -n` and + that it refuses to run without the ProxyGroup name +- [x] 4.4 Write `docs/how-to/kubernetes-proxygroup.md`: what hop 2 is and why + readiness misses it, the opt-in label and ConfigMap contract, the four methods with + version floors and failure policy, the custom webhook section (AdmissionReview + contract, JSON patch, `objectSelector`, `sideEffects: None`, TLS, example Go handler), + cost guidance, the residual DNAT gap, and fenced bash blocks under a fixed heading + per method for the test to run; verify markdownlint and that every config snippet + passes `tail-watch check-config` +- [x] 4.5 Link the page from `docs/README.md` and `README.md`, and add the Kubernetes + placement to `docs/explanation/` where placement is explained; verify markdownlint + +## 5. kind harness + +- [x] 5.1 Add `cmd/tail-watch/testdata/kind-proxygroup.yaml` (cluster config, enabling + MutatingAdmissionPolicy when the node image needs it) and testdata manifests for the + workload, its L4 Service on the ProxyGroup, the ProxyGroup, and an in-cluster webhook + receiver; verify `kind create cluster --config` succeeds and + `kubectl api-resources | grep -i mutatingadmission` lists the API +- [x] 5.2 Add `scripts/kind-stack.sh up|down` (cluster, operator chart pinned to the + `tailscale.com` version in `go.mod`, OAuth client read from the pulumi env, `kind + load` of a locally built image; `down` deletes the cluster, the Tailscale Service and + the operator's devices); verify `up` reaches a `Ready` ProxyGroup and `down` leaves no + devices with the stack's hostname prefix +- [x] 5.3 Determine whether kindnet enforces NetworkPolicy on the node image; if not, add + Calico to `kind-stack.sh`; verify a deny-all policy blocks a test `curl` between pods +- [x] 5.4 Document the harness in `docs/contributing/kind-proxygroup-test.md` + (prerequisites, env vars, cleanup); verify markdownlint + +## 6. kind live test + +- [x] 6.1 Add `TestKindProxyGroup` in `cmd/tail-watch/kind_proxygroup_test.go`, gated on + `TAIL_WATCH_KIND=1` and skipping when `kind`, `kubectl` or `helm` is missing, with + doc-block extraction in the style of `systemdDocBlocks`; verify it skips cleanly + without the gate and `go vet` passes +- [x] 6.2 For each of MutatingAdmissionPolicy, Kyverno and freeze-and-patch, run the + how-to's blocks, then assert every proxy pod has the `tail-watch` container, the + StatefulSet does not (except freeze-and-patch), and pods of a ProxyGroup without the + label are unchanged; verify the test passes for all three +- [x] 6.3 Break hop 2 with a NetworkPolicy denying the operator namespace, assert + EndpointSlices stay `Ready` and the operator condition reads "N/N proxy backends ready + and advertising", then assert `target.down` reaches the receiver; remove the policy + and assert `target.recovered`; verify the test passes +- [x] 6.4 Repeat 6.3 with a readiness probe on a port other than `targetPort` and + nothing listening on `targetPort`; verify the test passes +- [x] 6.5 Make the sidecar exit (delete its ConfigMap, recreate a pod) and assert the + Service stays advertised from that pod; record whether the ProxyGroup `Ready` + condition or a StatefulSet rollout is affected, and state the result in + `docs/how-to/kubernetes-proxygroup.md`; verify the test passes and the doc names the + observed behavior +- [x] 6.6 Assert an injector outage leaves proxy pods creatable: scale Kyverno to 0, + recreate a proxy pod, and assert it starts without the sidecar; verify the test passes + +## 7. Follow-up and project notes + +- [x] 7.1 Draft the operator issue (sidecar setting on ProxyClass/ProxyGroup, and + unadvertising a replica whose hop 2 is broken), citing this change's evidence; show + it to the user and file it only after approval; verify the user approved the text. + Outcome: drafted and shown; the user decided not to file the issue — the + admission-mutation methods documented in this change (MutatingAdmissionPolicy, + Kyverno, custom webhook, freeze-and-patch) are the accepted way to run the sidecar, + so no operator feature request is needed. +- [x] 7.2 Add the local-only mode, the sidecar contract and the kind test to `CLAUDE.md` + (Layout and a "kind ProxyGroup test" section); verify markdownlint + +## 8. Validation + +- [x] 8.1 Run `mise exec -- go build ./...`, `go vet ./...` and `go test ./...`; verify + all pass +- [x] 8.2 Run `mise run lint` and `govulncheck ./...`; verify no findings +- [x] 8.3 Run `markdownlint-cli2` over the changed docs and + `openspec/changes/add-proxygroup-sidecar/`; verify zero findings +- [x] 8.4 Run `mise exec -- openspec validate add-proxygroup-sidecar --strict`; verify + it passes +- [x] 8.5 Run `TAIL_WATCH_KIND=1 go test -run TestKindProxyGroup ./cmd/tail-watch` + end to end and then `scripts/kind-stack.sh down`; verify it passes and the test + tailnet has no leftover devices or Tailscale Service diff --git a/openspec/specs/collector-config/spec.md b/openspec/specs/collector-config/spec.md index 76f97dd..fea2164 100644 --- a/openspec/specs/collector-config/spec.md +++ b/openspec/specs/collector-config/spec.md @@ -28,12 +28,19 @@ falling back to the `TAIL_WATCH_CONFIG` environment variable, falling back to ### Requirement: Collector section The `collector` section SHALL accept `listen` (host:port for the HTTP server, -default `127.0.0.1:9109`), `state_dir` (optional directory path) and `ui` (boolean, -default `false`). The `--listen` flag SHALL override `collector.listen`. The `--ui` flag -SHALL override `collector.ui`, and `--ui=false` SHALL turn it off. When the UI is on and -`collector.state_dir` is unset, tail-watch SHALL exit non-zero before contacting -tailscaled, with an error saying the UI needs `collector.state_dir` to store monitors. -With `--once`, the UI setting SHALL be ignored and only YAML targets run. +default `127.0.0.1:9109`), `state_dir` (optional directory path), `ui` (boolean, +default `false`) and `local_only` (boolean, default `false`). The `--listen` flag SHALL +override `collector.listen`. The `--ui` flag SHALL override `collector.ui`, and +`--ui=false` SHALL turn it off. When the UI is on and `collector.state_dir` is unset, +tail-watch SHALL exit non-zero before contacting tailscaled, with an error saying the UI +needs `collector.state_dir` to store monitors. With `--once`, the UI setting SHALL be +ignored and only YAML targets run. + +When `collector.local_only` is `true`, a target that sets no `network` SHALL default to +`local` instead of `tailnet`, and validation SHALL reject, with every problem listed and +each naming its field: a target with `network: tailnet` (`defaults` has no `network` key), any `tsping` +check, and the UI being on from `collector.ui` or the `--ui` flag. `tail-watch +check-config` SHALL apply the same rules. #### Scenario: Default listen address @@ -60,6 +67,24 @@ With `--once`, the UI setting SHALL be ignored and only YAML targets run. - **WHEN** tail-watch runs with `--once --ui` - **THEN** it runs each YAML target's checks once and starts no HTTP server +#### Scenario: Local-only defaults targets to local + +- **WHEN** `collector.local_only: true` is set and a target sets no `network` +- **THEN** the target loads with `network: local` + +#### Scenario: Local-only rejects tailnet targets and tsping + +- **WHEN** `collector.local_only: true` is set, one target sets `network: tailnet` and + another lists a `tsping` check +- **THEN** validation fails with an error for each, naming the target's `network` field + and the `checks` entry + +#### Scenario: Local-only rejects the UI + +- **WHEN** `collector.local_only: true` is set and tail-watch runs with `--ui` +- **THEN** tail-watch exits non-zero with an error saying the UI is unavailable in + local-only mode + ### Requirement: Defaults section The `defaults` section SHALL accept `interval`, `timeout`, `fail_threshold`, diff --git a/openspec/specs/collector-runtime/spec.md b/openspec/specs/collector-runtime/spec.md index 39f06f9..73a1d8f 100644 --- a/openspec/specs/collector-runtime/spec.md +++ b/openspec/specs/collector-runtime/spec.md @@ -12,13 +12,19 @@ one-shot mode and shuts down. tail-watch SHALL connect to the local tailscaled over its LocalAPI socket, using the platform default path unless the `--socket` flag names another. It SHALL NOT -embed its own Tailscale node. +embed its own Tailscale node. When `collector.local_only` is `true`, tail-watch SHALL +NOT connect to tailscaled at all and SHALL ignore `--socket`. #### Scenario: Custom socket - **WHEN** tail-watch runs with `--socket /run/ts/tailscaled.sock` - **THEN** all LocalAPI calls use that socket +#### Scenario: Local-only makes no LocalAPI call + +- **WHEN** `collector.local_only: true` is set and no tailscaled runs on the host +- **THEN** tail-watch makes no LocalAPI call and runs its checks + ### Requirement: Startup preflight Before scheduling checks, tail-watch SHALL confirm that tailscaled is reachable and @@ -34,6 +40,11 @@ NOT be treated as proof of an installed host route. Deployment documentation SHALL require checking the effective route for each remote subnet and selecting `network: local` explicitly for locally monitored LANs. +When `collector.local_only` is `true`, preflight SHALL skip every tailscaled check +above, SHALL log the hostname and that it runs in local-only mode, and SHALL use the +hostname as the collector identity, so notifications carry `collector.name` and omit +`collector.tailscale_ip`. + #### Scenario: Subnet routes not accepted - **WHEN** a Linux collector has a remote subnet target but does not accept routes @@ -57,6 +68,18 @@ SHALL require checking the effective route for each remote subnet and selecting - **THEN** tail-watch starts and logs a warning that notifications will route through the exit node +#### Scenario: Local-only startup without tailscaled + +- **WHEN** `collector.local_only: true` is set and the LocalAPI socket does not exist +- **THEN** tail-watch starts, logs its hostname and local-only mode, and schedules + its checks + +#### Scenario: Local-only collector identity + +- **WHEN** a local-only collector on host `pg-ingress-0` sends `collector.started` +- **THEN** the payload has `collector.name` `pg-ingress-0` and no + `collector.tailscale_ip` + ### Requirement: HTTP endpoints tail-watch SHALL serve on the configured listen address: `/` as the prober status diff --git a/openspec/specs/proxygroup-sidecar/spec.md b/openspec/specs/proxygroup-sidecar/spec.md new file mode 100644 index 0000000..83beaff --- /dev/null +++ b/openspec/specs/proxygroup-sidecar/spec.md @@ -0,0 +1,132 @@ +# proxygroup-sidecar Specification + +## Purpose + +Runs tail-watch as an opt-in sidecar in Tailscale Kubernetes operator ProxyGroup pods, +so a broken path from the proxy to the workload is detected and alerted on even while +the operator and control report the Tailscale Service as advertised. + +## Requirements + +### Requirement: Opt-in sidecar selection + +The documented injection SHALL add the tail-watch sidecar only to pods that carry both +`tailscale.com/parent-resource-type: proxygroup`, which the operator sets, and +`tail-watch/inject: "true"`, which the admin sets through the ProxyGroup's ProxyClass +pod labels. Any other pod, including a ProxyGroup pod without the opt-in label, SHALL +be admitted unchanged. Removing the label from the ProxyClass SHALL stop injection +for pods created afterwards. + +#### Scenario: Opted-in ProxyGroup gets the sidecar + +- **WHEN** a ProxyGroup uses a ProxyClass that sets the pod label + `tail-watch/inject: "true"` and its pods are created +- **THEN** each proxy pod runs a `tail-watch` container next to the `tailscale` + container + +#### Scenario: Other pods are untouched + +- **WHEN** a ProxyGroup pod without the opt-in label, or any pod that is not a + ProxyGroup pod, is created +- **THEN** the pod is admitted exactly as submitted + +### Requirement: Per-ProxyGroup configuration + +The injected container SHALL run tail-watch in local-only mode with its config read +from the ConfigMap `tail-watch-` in the pod's namespace, key `tail-watch.yaml`, +where `` is the pod's `tailscale.com/parent-resource` label, mounted read-only. +Targets in that config SHALL name the cluster Services the ProxyGroup forwards to, +for example `web.apps.svc.cluster.local` with `tcp:` or `http:` checks. Secrets for +notification sinks SHALL come from a Kubernetes Secret through environment variables, +using the existing `*_env` keys, and SHALL NOT appear in the ConfigMap. + +#### Scenario: Config follows the ProxyGroup name + +- **WHEN** a pod of ProxyGroup `ingress-a` is injected +- **THEN** its `tail-watch` container reads `tail-watch.yaml` from ConfigMap + `tail-watch-ingress-a` + +#### Scenario: Missing ConfigMap + +- **WHEN** the ConfigMap for an opted-in ProxyGroup does not exist +- **THEN** the pod is still created, the `tail-watch` container reports the missing + config, and the `tailscale` container runs, serves traffic and keeps the Service + advertised as it would without the sidecar + +### Requirement: Proxy isolation + +Injection SHALL change only the pod: it SHALL add the `tail-watch` container and its +config volume, and SHALL NOT modify the `tailscale` container, the pod's other +settings, or the operator-owned StatefulSet. A failure of the injector SHALL NOT stop +proxy pods from being created; such a pod SHALL start without the sidecar. The +documented container SHALL set CPU and memory requests and limits and run as a +non-root user, so the admin can see and bound what opting in costs. + +#### Scenario: Sidecar failure keeps the Service advertised + +- **WHEN** the `tail-watch` container exits or restarts repeatedly +- **THEN** the `tailscale` container keeps serving and the operator keeps the + Service advertised from that pod + +#### Scenario: Injector unavailable + +- **WHEN** the injector of a method that relies on an admission webhook is down and + a ProxyGroup pod is created +- **THEN** the pod is created without the `tail-watch` container + +#### Scenario: Operator reconcile keeps the sidecar + +- **WHEN** the operator reconciles an opted-in ProxyGroup and its pods are recreated +- **THEN** the new pods have the `tail-watch` container again and the StatefulSet + spec does not contain it + +### Requirement: Documented injection methods + +The documentation SHALL describe, with the same opt-in label and ConfigMap contract, +four ways to inject the sidecar so the admin can choose: a MutatingAdmissionPolicy; +a Kyverno ClusterPolicy that mutates Pods on create with policy autogeneration for +pod controllers turned off; a custom mutating admission webhook, giving the +AdmissionReview contract, the JSON patch, the webhook configuration and TLS setup, and +example handler code; and freeze-and-patch, marked as evaluation only because the +operator reconciles nothing while it is scaled to zero. Each method SHALL state the +Kubernetes and component versions it needs and how its failure policy keeps proxy pods +starting. The repository SHALL ship example manifests for the MutatingAdmissionPolicy, +Kyverno and freeze-and-patch methods, and no injector image. + +#### Scenario: Admin picks Kyverno + +- **WHEN** an admin with Kyverno applies the documented ClusterPolicy and opts a + ProxyGroup in +- **THEN** the ProxyGroup's pods get the sidecar and the StatefulSet is not mutated + +#### Scenario: Admin picks a custom webhook + +- **WHEN** an admin builds an injector from the documented contract and example code +- **THEN** it produces the same pod as the other methods for the same opt-in label + and ConfigMap name + +### Requirement: Detect a broken proxy-to-workload hop + +With the sidecar running and the Service still advertised, a target for a cluster +Service SHALL go `DOWN` and send `target.down` when traffic from the proxy pod to the +workload fails, even while the Service's EndpointSlice endpoints are `Ready`. It SHALL +send `target.recovered` when the path works again. Detection SHALL rely only on the +sidecar's own checks, not on operator status, endpoint readiness or the LocalAPI. + +#### Scenario: NetworkPolicy blocks the proxy + +- **WHEN** a NetworkPolicy denies ingress from the operator's namespace to the + workload, its endpoints stay `Ready`, and the operator still reports the Service as + advertised +- **THEN** the sidecar's check for that Service fails and `target.down` is sent + +#### Scenario: Readiness probe on another port + +- **WHEN** the workload's readiness probe passes on one port while nothing listens on + the Service's `targetPort` +- **THEN** the sidecar's check for that Service fails and `target.down` is sent + +#### Scenario: Path restored + +- **WHEN** the blocking NetworkPolicy is removed +- **THEN** the target recovers and `target.recovered` is sent diff --git a/scripts/kind-stack.sh b/scripts/kind-stack.sh new file mode 100755 index 0000000..6b003e6 --- /dev/null +++ b/scripts/kind-stack.sh @@ -0,0 +1,131 @@ +#!/usr/bin/env bash +# Bring up (or tear down) the kind cluster for TestKindProxyGroup: the +# Tailscale operator pinned to the tailscale.com version in go.mod, the +# fixtures in cmd/tail-watch/testdata/kind, and a locally built tail-watch +# image loaded into the cluster. It does not inject the sidecar or run any +# scenario; the test does that with the doc blocks from +# docs/how-to/kubernetes-proxygroup.md. See +# docs/contributing/kind-proxygroup-test.md. Credentials come from the pulumi +# env in TAIL_WATCH_PULUMI_ENV (default tailscale-phase-2/tailscale): the +# operator's OAuth client as tailscale:operatorClientId and +# tailscale:operatorClientSecret, and tailscale:apiKey plus tailscale:tailnet +# for teardown. +# +# scripts/kind-stack.sh up create the cluster and reach a Ready ProxyGroup +# scripts/kind-stack.sh down delete the cluster, the Tailscale Service and +# the stack's devices +set -euo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +CLUSTER="${TAIL_WATCH_KIND_CLUSTER:-tail-watch-proxygroup}" +ENVNAME="${TAIL_WATCH_PULUMI_ENV:-tailscale-phase-2/tailscale}" +IMAGE="${TAIL_WATCH_KIND_IMAGE:-tail-watch:0.1.0}" +SERVICE_NAME="svc:tw-pg-web" +DEVICE_PREFIX="tw-pg" + +kube() { kubectl --context "kind-$CLUSTER" "$@"; } + +creds() { + pulumi env open "$ENVNAME" --format json | python3 -c ' +import json,sys +t=json.load(sys.stdin)["tailscale"] +for k in ("apiKey","tailnet","operatorClientId","operatorClientSecret"): + print(t.get(k,""))' +} + +load_creds() { + local out + out="$(creds)" + APIKEY="$(sed -n 1p <<<"$out")" + TAILNET="$(sed -n 2p <<<"$out")" + CLIENT_ID="$(sed -n 3p <<<"$out")" + CLIENT_SECRET="$(sed -n 4p <<<"$out")" +} + +# chart_version pins the operator chart to the tailscale.com module version in +# go.mod. Not every module release ships a chart (v1.102.5 has none), so pick +# the newest chart version that is not newer than the module version. +chart_version() { + local mod + mod="$(cd "$ROOT" && mise exec -- go list -m -f '{{.Version}}' tailscale.com | sed 's/^v//')" + helm search repo tailscale/tailscale-operator --versions --output json | + python3 -c ' +import json,sys +want=sys.argv[1] +def key(v): + return [int(p) for p in v.split(".")] +versions=[e["version"] for e in json.load(sys.stdin)] +ok=[v for v in versions if key(v)<=key(want)] +if not ok: + sys.exit("no tailscale-operator chart at or below "+want) +print(sorted(ok,key=key)[-1])' "$mod" +} + +api() { curl -fsS -K <(printf 'user = "%s:"\n' "$APIKEY") -H 'Content-Type: application/json' "$@"; } + +up() { + load_creds + if [ -z "$CLIENT_ID" ] || [ "$CLIENT_ID" = "None" ] || [ -z "$CLIENT_SECRET" ] || [ "$CLIENT_SECRET" = "None" ]; then + echo "pulumi env $ENVNAME has no tailscale:operatorClientId/operatorClientSecret;" >&2 + echo "add an OAuth client with Devices Core, Auth Keys and Services write scopes, tagged tag:k8s-operator" >&2 + exit 1 + fi + helm repo add tailscale https://pkgs.tailscale.com/helmcharts >/dev/null + helm repo update tailscale >/dev/null + local version + version="$(chart_version)" + if ! kind get clusters 2>/dev/null | grep -qx "$CLUSTER"; then + kind create cluster --name "$CLUSTER" --config "$ROOT/cmd/tail-watch/testdata/kind-proxygroup.yaml" --wait 120s + fi + # Secrets go through a values file, never the command line. + local values + values="$(mktemp)" + chmod 600 "$values" + trap 'rm -f "$values"' EXIT + cat >"$values" </dev/null + kind load docker-image "$IMAGE" --name "$CLUSTER" + + kube apply -f "$ROOT/cmd/tail-watch/testdata/kind/workload.yaml" + kube apply -f "$ROOT/cmd/tail-watch/testdata/kind/proxygroup.yaml" + # The sink Secret (tail-watch-ingress in namespace tailscale) is left to the + # test: the how-to's blocks apply the placeholder from + # examples/kubernetes/secret.yaml, and the test then points OPS_WEBHOOK_URL + # at the in-cluster alert-receiver. + kube -n apps wait --for=condition=Available deploy/web deploy/alert-receiver --timeout=240s + kube wait --for=condition=ProxyGroupReady proxygroup/ingress proxygroup/plain --timeout=600s + echo "kind cluster $CLUSTER is up; ProxyGroup ingress and plain are Ready" + echo "Run the test with: TAIL_WATCH_KIND=1 go test -run TestKindProxyGroup ./cmd/tail-watch" +} + +down() { + load_creds + kind delete cluster --name "$CLUSTER" || true + api -X DELETE "https://api.tailscale.com/api/v2/tailnet/$TAILNET/services/$SERVICE_NAME" >/dev/null 2>&1 && + echo "deleted Tailscale Service $SERVICE_NAME" || true + api "https://api.tailscale.com/api/v2/tailnet/$TAILNET/devices?fields=all" | python3 -c ' +import json,sys +p=sys.argv[1].lower() +for d in json.load(sys.stdin)["devices"]: + if d["hostname"].lower().startswith(p): print(d["id"])' "$DEVICE_PREFIX" | + while read -r id; do api -X DELETE "https://api.tailscale.com/api/v2/device/$id" >/dev/null && echo "deleted device $id"; done +} + +case "${1:-}" in +up) up ;; +down) down ;; +*) echo "usage: $0 up|down" >&2; exit 2 ;; +esac diff --git a/scripts/live-stack.sh b/scripts/live-stack.sh index 841ccf3..1dd7bbd 100755 --- a/scripts/live-stack.sh +++ b/scripts/live-stack.sh @@ -23,7 +23,7 @@ t=json.load(sys.stdin)["tailscale"] print(t["apiKey"]); print(t["tailnet"])' } -api() { curl -fsS -u "$APIKEY:" -H 'Content-Type: application/json' "$@"; } +api() { curl -fsS -K <(printf 'user = "%s:"\n' "$APIKEY") -H 'Content-Type: application/json' "$@"; } mint_key() { api -X POST "https://api.tailscale.com/api/v2/tailnet/$TAILNET/keys" \ diff --git a/tailnet-reachability-monitor-design.md b/tailnet-reachability-monitor-design.md index 0127ec0..f793f0a 100644 --- a/tailnet-reachability-monitor-design.md +++ b/tailnet-reachability-monitor-design.md @@ -112,6 +112,13 @@ file. - Don't run the collector on the subnet router itself. A router-hosted collector reaches the LAN directly and never crosses the tunnel, so it can't see `tunnel`-layer failures, and it goes down with the thing it is supposed to report on. +- In Kubernetes, break that rule deliberately for the operator's ProxyGroup proxy pods: + run the collector as a sidecar in the proxy pod. Its subject is hop 2, the path from + the proxy pod to the workload, which is only visible from the proxy's network + namespace. The don't-run-on-the-device rule is about tunnel visibility, and hop-2 + checks never cross the tunnel, so running on the proxy costs nothing there. Hop 1 + (tailnet client to proxy) stays with a regular tailnet collector probing the Service's + tailnet name. See `docs/how-to/kubernetes-proxygroup.md`. - Two collectors in different regions allow quorum. Only alert when both agree, which removes most false pages caused by the collector's own network.