Skip to content

Add site-to-site VPN solution (Crusoe ⇄ AWS/GCP, route-based IPsec + BGP) - #89

Open
smazigh wants to merge 20 commits into
mainfrom
site-to-site-vpn
Open

Add site-to-site VPN solution (Crusoe ⇄ AWS/GCP, route-based IPsec + BGP)#89
smazigh wants to merge 20 commits into
mainfrom
site-to-site-vpn

Conversation

@smazigh

@smazigh smazigh commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Site-to-Site VPN — Crusoe ⇄ AWS/GCP (route-based IPsec + BGP)

Adds a new solution: a hardened, redundant, route-based IPsec (IKEv2) VPN with dynamic BGP routing that terminates on Crusoe and connects a Crusoe VPC to a customer's AWS or GCP VPC. The Crusoe side is fully automated with Terraform from a single params file; the customer side ships as optional Terraform plus documented runbooks.

New top-level directory: site-to-site-vpn/. No existing files touched.

What's included

  • terraform/crusoe/ — the owned gateway module: 1 VM (ha_mode=single) or 2 (dual), static public IP(s), default-deny Crusoe firewall (UDP 500/4500 from peer IPs, SSH allow-listed), and a bash startup_script that renders strongSwan (swanctl, XFRM route-based tunnels), FRR (bgpd eBGP + prefix filters + ECMP), nftables, and kernel hardening. Idempotent.
  • terraform/gcp/, terraform/aws/ — optional customer-side modules (GCP HA VPN + Cloud Router; AWS S2S VPN on Transit Gateway). Existing AWS deployments are covered by a docs-only path.
  • k8s/cluster-egress/ — optional Helm chart: a vxlan overlay DaemonSet so CMK pods / other Crusoe VMs can egress through the gateway (works around the Crusoe VPC fabric only delivering own-destination frames).
  • scripts/ — live test suite (verify, mtu, failover, rekey, security, healthcheck) + shared lib.
  • tests/render/ — static template render harness (CI phase 0).
  • .github/workflows/ci.yml — fmt, validate x3, tflint, trivy config scan, render dry-run, gitleaks, tracked-secrets guard.
  • docs/ — architecture (with mermaid diagrams), runbook (deploy -> verify -> troubleshoot -> rotate -> teardown), customer GCP/AWS guides, crypto profiles, IP planning, cluster-egress, outgrowing-this; plus params/schema.md.

Validation

Run live against Crusoe (dx-lab / seif-projects) and GCP (crusoe-solutions-engineering):

Path Status
GCP peer, ha_mode=single Validated end-to-end (control + data plane, MTU/MSS, failover, rekey, reboot recovery, security)
ha_mode=dual Validated — tunnels split across 2 VMs; VM-failure survived on the survivor (bounded BGP reconvergence)
Cluster egress (CMK) to GCP Validated end-to-end incl. multi-node + node-churn self-heal
AWS peer Documented + module-validated; first live deployment should be guided

See site-to-site-vpn/tests/matrix.md for the run record and pinned versions.

Security

  • No secrets in the repo — PSKs supplied only via TF_VAR_tunnel_psks; CI enforces a secret-leak scan. PSK-in-state/instance-metadata exposure is documented with mitigations.
  • Hardened by default: IKEv2 only, strong crypto floor, default-deny host + network firewalls, SSH key-only/no-root, BGP prefix filters both directions, unattended security updates.

Notes / limitations

  • Day-2 gateway config changes replace the VM (new IP); the runbook documents a zero-drop in-place PSK rotation path.
  • The Crusoe VPC fabric forwards only own-destination frames (no VPC routes / source-dest-check toggle); routing other hosts' traffic through the gateway uses the cluster-egress overlay. Documented as a platform property.
  • WireGuard overlay transport and full per-pod source identity are noted as roadmap.

Out of scope

Physical/partner interconnect, WireGuard to managed cloud VPN gateways, and provisioning of the customer's intra-cloud topology beyond the landing VPC.

Seif Mazigh added 20 commits July 23, 2026 11:46
Seif Mazigh
site-to-site-vpn: add SPEC.md and implementation plan
b79b2c5
Seif Mazigh
site-to-site-vpn: scaffold repo layout, gitignore, license
122cc7a
Seif Mazigh
crusoe module: instances, firewall, overlap guard, bootstrap templates
2dbc318 
- variables with strict validations (ASN ranges, PSK charset, tunnel
  uniqueness, mss<=mtu-40, no world-open SSH)
- single/dual HA via tunnels_by_vm grouping with assignment preconditions
- firewall product-expansion (one CIDR per Crusoe rule)
- startup script renders swanctl/FRR/nftables/XFRM configs; healthcheck,
  journald retention, apt retry, SSH hardening
- MSS clamp ordered before conntrack accept; FRR no network import-check;
  prefix-list filtering both directions
Seif Mazigh
gcp/aws customer modules, render test harness, params schema
05da6ec 
- gcp: HA VPN gateway, external gateway (single/dual redundancy), Cloud
  Router, 2 IKEv2 tunnels, BGP peers on /30s; input validations
- aws: TGW with ECMP, CGW, S2S VPN pinned to IKEv2 AES256-GCM-16/SHA2-384/
  DH20; optional VPC return routes to TGW
- tests/render: renders swanctl/FRR/nftables templates from fixtures,
  21 stanza assertions, exits non-zero on failure
- params: example tfvars + full variable schema with GCP/AWS mappings
Seif Mazigh
test scripts, CI workflow, documentation set
2fa9c60 
- scripts: verify, mtu, failover, rekey, security suites + shared lib
  (single-string remote commands, exported env for bash -c asserts,
  tool guards, explicit TUNNEL_A_IF_ID for failover)
- healthcheck.sh standalone copy of the on-VM monitor
- ci: fmt/validate/tflint/trivy config scan/render dry-run/gitleaks/
  tracked-secrets guard (assumes dir is repo root)
- docs: architecture, runbook (with observability + rotation), customer
  GCP/AWS guides, crypto profiles, IP planning, outgrowing-this
- tests: phase runbook + tested-against matrix
- apt retry loop in bootstrap; README finalized with quickstart and
  deliberate SPEC deviations
Seif Mazigh
close integration-review gaps
d263622 
- render harness now covers all 7 templates (secrets, xfrm, handoff, full
  startup script) with bash -n on rendered shell; 34 assertions
- TUNNEL_A_NAME defaults to tunnel-a in failover/rekey tests (matches docs)
- matrix/docs wording: lock files gitignored, trivy is the CI scanner
Seif Mazigh
fixes from first live E2E run (Crusoe seif-projects ⇄ GCP, phases 1-5…
5b6b371 
… PASS)

- crusoe module: firewall destination needs /32 suffix (API normalizes,
  caused perpetual plan drift); bootstrap now restarts strongswan/frr
  instead of enable --now (apt pre-starts them with stock config; frr
  never launched bgpd, stale vici socket broke swanctl --load-all) and
  retries the config load
- scripts: pin VM-originated probe source to the VPC address (tunnel
  traffic otherwise sources from 169.254 and peers filter it); remove
  unsupported swanctl --rekey --timeout flag; failover asserts data-path
  (xfrm link) state instead of SA state since peers re-initiate
  instantly, and retries traffic across the BGP reconvergence window;
  fix literal \! escapes that broke negated asserts; TRANSFER_URL
  fallback for the MSS-clamp transfer proof; CALLER_ALLOWLISTED mode
  for security probes from an allow-listed source
- record live run results in tests/matrix.md
Seif Mazigh
docs: first-class path for existing AWS Site-to-Site VPN deployments
edabaf3 
customer-aws.md restructured into Path A (customer already runs AWS S2S
VPN: create/re-point CGW + VPN connection via CLI or console, pin IKEv2
crypto, extract outside IPs/inside CIDRs/PSKs with
describe-vpn-connections, route propagation checks) and Path B
(greenfield via the optional terraform/aws module, unchanged).
schema.md and README point existing deployments at Path A.
Seif Mazigh
fix healthcheck xfrm interface name parsing (false CRIT)
c170114 
ip -o link show type xfrm yields 'ipsecNNN@parent'; the device name is
the part before '@'. Strip it with cut -d@ -f1 so 'ip link show' resolves,
and match 'state (UP|UNKNOWN)' (xfrm interfaces sit in UNKNOWN with the UP
flag). Fresh redeploy validated the bootstrap fix: tunnels + BGP + data
plane came up with zero manual intervention (verify.sh 7/7).
Seif Mazigh
docs: validated cluster/multi-VM egress through the gateway VM
ea36efb 
Documents the Crusoe VPC constraint (fabric drops foreign-dst frames; no
VPC routes or src/dest-check toggle) and the proven workaround: vxlan
overlay host->gateway + open gateway firewall for the overlay transport +
SNAT to the gateway LAN IP. Validated live: CMK pod -> node vxlan ->
gateway VM -> IPsec -> GCP, 0% loss + 20MB transfer. Compares with
ipsec-tunnel-cmk.
Seif Mazigh
cluster-egress: automate vxlan overlay (gateway TF + node DaemonSet)
fc8cef2 
New cluster_egress variable (default off) makes the gateway forward other
Crusoe hosts' traffic despite the VPC fabric's dst-filtering:
- gateway: vxlan-ceg hub (learning) + nftables overlay input/forward/MSS-clamp
  + runtime SNAT to the gateway LAN IP (set from the VM's own IP at boot via a
  persisted systemd unit) + Crusoe firewall rule for the vxlan transport
- k8s/cluster-egress DaemonSet: per-node vxlan to the gateway, collision-free
  overlay IP from the node IP octets, routes remote CIDRs, overlay MSS clamp
- render harness covers enabled + disabled nftables and the full bootstrap

Validated live: terraform apply (gateway) + kubectl apply (DaemonSet) only ->
CMK pod -> node vxlan -> gateway -> IPsec -> GCP, 0% loss + 20MB HTTP 200.
Seif Mazigh
cluster-egress: flag surface + churn-resilient multi-node overlay
130f267 
- cluster_egress flags: snat_mode (gateway|node), overlay_transport
  (vxlan; wireguard reserved), vxlan_id/port, overlay_cidr. snat_mode=node
  drops SNAT and advertises the overlay via BGP (per-node source identity).
- Helm chart (k8s/cluster-egress) replaces the flat manifest; values fed
  from TF outputs; multi-gateway list with per-node hash spread.
- Churn resilience: deterministic node MAC 02:00:a9:fe:<O3>:<O4> encodes the
  overlay IP; gateway neighbor-reconcile loop maps learned FDB -> permanent
  neigh so return traffic resolves with no ARP flood (learning vxlan hubs
  can't flood without multicast). New/removed/re-IP'd nodes self-heal, no
  per-node gateway config.
- render harness covers snat_mode gateway+node branches.

Static-validated (fmt/validate/render/helm lint). Multi-node live
re-validation pending (local egress IP changed -> gateway SSH locked; needs
gateway rebuild + GCP reapply).
Seif Mazigh
docs: cluster-egress + churn resilience in README, add repo CLAUDE.md
e26e8ab 
- README: cluster/multi-VM egress section (resilience to node churn/IP
  changes, deterministic-MAC + gateway neigh-reconcile), doc-map entry,
  and SPEC §17 marked answered (Crusoe VPC drops foreign-dst; no route/
  src-dest-check primitive — overlay is the sanctioned path).
- CLAUDE.md: repo conventions, environment quirks (sandbox, 1Password
  signing, VM-replace-on-config-change, /32 SSH lockout), Crusoe platform
  facts, live status.
- matrix: 2026-07-24 cluster-egress run — multi-node (2 nodes) + churn
  self-heal PASS; gateway auto-neigh for all 10 nodes.

Multi-node + churn validated live.
Seif Mazigh
docs(aws): note Crusoe module's >=16-char PSK floor vs AWS custom PSKs
2f9f4b3
Seif Mazigh
security: document IKE-scan limitation (responder pinned to peer IPs)
24c122e 
Active IKEv1/weak-proposal probes from a non-peer source get no response
regardless of IKE version, so they can't prove IKEv1 rejection on their own;
the on-box 'version = 2 only' config assertion is authoritative. Verified
live: ike-scan IKEv1/IKEv2/weak-aggressive all 0 handshakes (source-pinned).
Seif Mazigh
docs: add validation-status table to README
e4aa96f 
Distinguishes live-validated (GCP single, cluster-egress incl. multi-node+
churn) from documented/module-validated (AWS, ha_mode=dual) and roadmap
(WireGuard, per-pod identity).
Seif Mazigh
matrix: record dual-VM run blocked by Crusoe environment
93a3bbd 
ha_mode=dual applied clean (2 VMs, GCP two-interface, rules correct) but both
VMs stayed non-functional 25+ min (no SSH, no IKE from peer), unchanged by
reset — a platform issue in eu-iceland1-a; module is validate-clean and the
single-VM path with identical bootstrap passed the same day.
Seif Mazigh
dual-VM HA validated live (fresh VMs)
260d923 
ha_mode=dual: vm0/tunnel-a + vm1/tunnel-b, GCP TWO_IPS_REDUNDANCY. Each VM
terminates its tunnel + BGP. VM-failure test: stopped vm1 mid-ping,
connection survived on vm0 (16.7% loss over 90s = BGP reconvergence, then
stable). Prior fresh pair was a transient Crusoe env glitch; clean re-apply
worked immediately. README status + matrix updated.
Seif Mazigh
make repo customer-ready; move build methodology to gitignored context/
4d5ea8c 
- context/ (gitignored): methodology + learnings + the implementation SPEC and
  plan moved out of the shipped tree — internal build notes only.
- docs: removed all SPEC cross-references and build/session narrative; docs read
  as neutral product documentation. cluster-egress doc reframed to a feature
  guide (flags, deploy, resilience, status/roadmap). matrix reads as a clean
  tested-against record. README 'Platform notes' replaces 'deviations from SPEC'.
- code: stripped narrative/'Plan bug fix' comments from scripts; kept genuine
  constraint comments. No logic changes (render harness byte-identical).
- CLAUDE.md trimmed to repo guidance + Crusoe platform facts.

Static gate green: fmt, validate x3, render (45 checks), shellcheck, helm lint,
ci yaml.
Seif Mazigh
docs: add mermaid diagrams (topology, dual-VM HA, cluster-egress over…
6b33107 
…lay)

GitHub-native mermaid replaces/augments the ASCII: architecture.md topology
+ dual-VM HA; crusoe-cluster-egress.md overlay packet path. Validated via
mermaid render.
youngjeong46
youngjeong46 reviewed Aug 19, 2026
View reviewed changes

@youngjeong46 youngjeong46 left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a link this this directory in the Main README? Should go under "Networking". You may also need to rebase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@youngjeong46 youngjeong46 youngjeong46 left review comments
@chinmaybaikar chinmaybaikar chinmaybaikar approved these changes
@datadoc24 datadoc24 Awaiting requested review from datadoc24 datadoc24 is a code owner

Assignees

@youngjeong46 youngjeong46

@smazigh smazigh

@chinmaybaikar chinmaybaikar

@martin-cala1 martin-cala1

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@smazigh @youngjeong46 @chinmaybaikar @martin-cala1