diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 67af4a6f4..e3a2f56fb 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -241,14 +241,22 @@ This document provides a comprehensive overview of every feature currently imple - **VPC Scoped**: Zones are scoped to VPCs for private network resolution. ### 13. API Gateway πŸ†• -**What it is**: Managed entry point for microservices with advanced routing, pattern matching, and rate limiting. -**Tech Stack**: Go `httputil.ReverseProxy`, Redis, Regex Matcher. +**What it is**: Managed entry point for microservices with advanced routing, pattern matching, rate limiting, JWT authentication, mTLS, and observability. +**Tech Stack**: Go `httputil.ReverseProxy`, Redis, Regex Matcher, `golang.org/x/time/rate`, Prometheus metrics, `golang.org/x/sync/singleflight`. + **Implementation**: - **Advanced Pattern Matching**: Support for RESTful patterns like `/users/{id}`, regex-constrained parameters `/id/{id:[0-9]+}`, and wildcards `/static/*`. - **HTTP Method Routing**: Route requests to different backends based on the HTTP verb (GET, POST, etc.) for the same path. - **Dynamic Specificity Scoring**: Automatic route selection based on prefix specificity, exact match bonuses, and explicit user-defined priority. - **Prefix Stripping**: Intelligent stripping of path patterns before forwarding to downstream services. -- **Rate Limiting**: Integrated distributed rate limiting per route. +- **Rate Limiting**: Integrated distributed rate limiting per route using token-bucket algorithm. Supports per-client key (API key prefix or IP) identification. +- **Circuit Breaker & Retry**: Per-route circuit breaker with configurable threshold and reset timeout. Automatic retry with exponential backoff on `502`, `503`, `504`, `429` responses. Only idempotent methods (GET, HEAD, PUT, DELETE, OPTIONS) are retried. +- **JWT Authentication**: JWKS-backed JWT validation with issuer and audience verification. RSA public keys parsed from JWK `n`/`e` parameters. JWKS fetches are deduplicated via singleflight and protected by a circuit breaker. Claims are propagated to upstream services via `X-JWT-Claim-*` headers. +- **mTLS Support**: Configurable client certificates (PEM) and CA certificates for backend TLS verification. +- **CORS**: Per-route CORS configuration with `allowed_origins`, `allowed_methods`, `allowed_headers`, `expose_headers`, and `max_age`. +- **Compression**: Gzip response compression when client advertises support and route has compression enabled. +- **Dry-Run Validation**: Route creation supports `?dry_run=true` to validate CIDR blocks, TLS conflicts, and mTLS certificate pairing without persisting. +- **Observability**: Prometheus metrics for upstream latency (`thecloud_gateway_upstream_latency_seconds`), retry totals (`thecloud_gateway_retry_total`), circuit breaker state (`thecloud_gateway_circuit_breaker_state`), JWKS fetch totals and breaker state (`thecloud_jwks_fetch_total`, `thecloud_jwks_breaker_state`). W3C TraceContext headers (`traceparent`, `tracestate`) are preserved or generated for upstream traceability. - **Audit Logging**: Comprehensive tracking of all route changes and gateway operations. ### 14. CloudStacks (Native IaC) πŸ†• diff --git a/docs/adr/ADR-028-gateway-per-route-rate-limiting.md b/docs/adr/ADR-028-gateway-per-route-rate-limiting.md index 2b43d2791..a9b20a62d 100644 --- a/docs/adr/ADR-028-gateway-per-route-rate-limiting.md +++ b/docs/adr/ADR-028-gateway-per-route-rate-limiting.md @@ -47,4 +47,4 @@ We introduced per-route rate limiting that supplements the existing global rate **Why rejected:** Introduces new dependency; the existing `golang.org/x/time/rate` meets requirements with minimal added code. ### Alternative 3: Global limiter with route-specific token deductions -**Why rejected:** Complex to reason about; one route's burst could starve another. Independent limiters per route provide cleaner semantics. +**Why rejected:** Complex to reason about; one route's burst could starve another. Independent limiters per route provide cleaner semantics. \ No newline at end of file diff --git a/docs/adr/ADR-029-gateway-security-resilience.md b/docs/adr/ADR-029-gateway-security-resilience.md new file mode 100644 index 000000000..ea893d167 --- /dev/null +++ b/docs/adr/ADR-029-gateway-security-resilience.md @@ -0,0 +1,63 @@ +# ADR 029: Gateway Security and Resilience + +## Status +Accepted + +## Date +2026-05-19 + +## Context + +PR #596 extends the API gateway with security (JWT, mTLS), resilience (circuit breaker, retry), and observability features. These require non-trivial decisions about how external trust is established, how failures are handled, and how observability data is exposed. + +## Decision + +### JWT Authentication with JWKS + +The gateway validates JWTs by fetching public keys from a JWKS endpoint. Because multiple routes may share the same JWKS URL (or different routes may have different JWKS URLs), we use a **per-route JWKS cache** with a 5-minute TTL. JWKS fetches are deduplicated using `singleflight.Group` so concurrent requests for the same keyset result in a single HTTP call. + +**JWKS circuit breaker**: When the JWKS endpoint returns errors, a circuit breaker (`Threshold: 3`, `ResetTimeout: 30s`) prevents further fetch attempts until the half-open probe succeeds. Metrics exported: `JWKSBreakerState` (gauge 0/1/2) and `JWKSFetchTotal` (counter with labels: `success`, `error`, `circuit_open`). + +**RSA key parsing**: Keys are parsed from JWK `n` (modulus) and `e` (exponent) using `math/big.Int`. An exponent `>= 1<<30` is rejected as a defensive measure against overflow. + +**Claim propagation**: Validated JWT claims are forwarded to the upstream service as `X-JWT-Claim-{key}` headers. + +### mTLS Configuration + +Routes can be configured with `client_cert` + `client_key` (PEM) for client certificates and `ca_cert` for backend certificate verification. Both client cert and key must be provided together; CA cert is optional. `buildTLSConfig` returns descriptive errors on malformed certs/keys rather than silently ignoring them. + +### Circuit Breaker and Retry + +Each route can have `circuit_breaker_threshold` (consecutive failures to trip) and `circuit_breaker_timeout` (ms in open before half-open). Retry behavior is controlled by `max_retries` and `retry_timeout`. Retries are only attempted for idempotent methods (GET, HEAD, PUT, DELETE, OPTIONS) and only on retryable status codes (502, 503, 504, 429) or retryable errors (connection refused, timeout, reset by peer, broken pipe). + +### CORS + +Per-route CORS uses `allowed_origins`, `allowed_methods`, `allowed_headers`, `expose_headers`, and `max_age`. When `allowed_origins` includes `"*"` with credentials, the response sets `Access-Control-Allow-Credentials: true`. + +## Consequences + +### Positive +- JWT authentication enables stateless auth with upstream services trusting gateway-validated tokens +- JWKS singleflight deduplication prevents thundering herd on shared JWKS endpoints +- Circuit breaker protects gateway from cascading failures when backends are unhealthy +- mTLS enables secure service-to-service communication within the cluster + +### Negative +- JWKS caching introduces staleness window (up to 5 minutes for rotated keys) +- Circuit breaker state is in-memory only β€” restarts reset all circuits +- Retry behavior increases latency on failure and may amplify load on unhealthy backends + +### Neutral +- CORS headers are only injected when `allowed_origins` is non-empty +- TLS settings (skip verify, require TLS, client certs) are independent concerns + +## Alternatives Considered + +### Alternative 1: Opaque JWT validation without JWKS caching +**Why rejected:** Would require fetching JWKS on every request, defeating the purpose of JWT's stateless design and creating unnecessary latency. + +### Alternative 2: Global circuit breaker for all backends +**Why rejected:** Per-route circuit breakers provide finer-grained isolation β€” one unhealthy backend shouldn't trip the breaker for unrelated routes. + +### Alternative 3: Retry on any HTTP error +**Why rejected:** Non-idempotent methods (POST, PATCH) must not be retried automatically as that could cause duplicate operations. The current implementation limits retries to idempotent methods and specific error codes. \ No newline at end of file diff --git a/docs/roadmaps/api-gateway-improvement-roadmap.md b/docs/roadmaps/api-gateway-improvement-roadmap.md new file mode 100644 index 000000000..0d0c9db95 --- /dev/null +++ b/docs/roadmaps/api-gateway-improvement-roadmap.md @@ -0,0 +1,387 @@ +# API Gateway Improvement Roadmap + +Prioritized by: Impact Γ— Effort + +--- + +## Phase 1: Quick Wins (Low Effort, High Impact) + +### 1.1 Add Upstream Timeouts +**Impact:** Prevents gateway hangs on backend issues +**Effort:** Low +**Status:** βœ… Shipped + +- Add `DialTimeout`, `ResponseHeaderTimeout`, `IdleConnTimeout` to `ReverseProxy` transport +- Add `ProxyTimeout` to `Config` (default 30s) +- File: `internal/core/services/gateway.go` + +### 1.2 Per-Route Rate Limiting +**Impact:** Currently global only; per-route gives finer control +**Effort:** Low +**Status:** βœ… Shipped in PR #570 + +- Modify `limiter.go` to support route-key limiting +- Add `RateLimit` field usage from `GatewayRoute` (already exists in domain) +- File: `pkg/ratelimit/limiter.go`, `internal/handlers/gateway_handler.go` + +### 1.3 IP Allowlist/Denylist +**Impact:** Basic security hardening +**Effort:** Low +**Status:** βœ… Shipped + +- Add `AllowedCIDRs`, `BlockedCIDRs` fields to `GatewayRoute` +- Implement CIDR matching middleware +- File: `internal/core/domain/gateway.go`, `internal/handlers/gateway_handler.go` + +### 1.4 Request Size Limits +**Impact:** DoS prevention +**Effort:** Low +**Status:** βœ… Shipped + +- Add `MaxBodySize` field to `GatewayRoute` +- Apply `maxBytes.ReadLimit` in proxy handler +- File: `internal/core/domain/gateway.go`, `internal/handlers/gateway_handler.go` + +### 1.5 TLS Configuration for Backends +**Impact:** Secure internal HTTPS communication +**Effort:** Low +**Status:** βœ… Shipped + +- Add `TLSSkipVerify`, `RequireTLS` fields to `GatewayRoute` +- Configure `httputil.ReverseProxy` with custom TLS config +- File: `internal/core/domain/gateway.go`, `internal/core/services/gateway.go` + +### 1.6 Trace Header Propagation +**Impact:** Distributed tracing works end-to-end +**Effort:** Low +**Status:** βœ… Shipped + +- Inject W3C TraceContext headers (`traceparent`, `tracestate`) in proxy handler +- Ensure `X-Request-ID` is passed to backends +- File: `internal/handlers/gateway_handler.go` + +--- + +## Phase 2: Resilience (High Impact) + +### 2.1 Circuit Breaker Per Route +**Impact:** Prevents cascading failures +**Effort:** Medium +**Status:** βœ… Shipped in PR #395 + +- Wire up existing `platform/circuit_breaker.go` to gateway routes +- Add `CircuitBreakerThreshold`, `CircuitBreakerTimeout` to `GatewayRoute` or config +- Track CB state in `GatewayService` +- File: `internal/core/services/gateway.go`, `internal/platform/circuit_breaker.go` + +### 2.2 Retry on Upstream Failure +**Impact:** Handle transient 502/503/504 errors +**Effort:** Medium +**Status:** βœ… Shipped in PR #395 + +- Use existing `platform/retry.go` utilities +- Add `MaxRetries`, `RetryTimeout` to `GatewayRoute` +- Retry on idempotent methods (GET, HEAD, PUT, DELETE) +- File: `internal/core/services/gateway.go`, `internal/platform/retry.go` + +### 2.3 Connection Pooling +**Impact:** Better HTTP performance +**Effort:** Low +**Status:** βœ… Shipped + +- Create shared `http.Transport` with connection pooling config +- Configure `MaxIdleConns`, `MaxIdleConnsPerHost`, `IdleConnTimeout` +- File: `internal/core/services/gateway.go` + +### 2.4 Health Check & Auto-Failover +**Impact:** Automatic recovery from backend failures +**Effort:** High + +- Add health check configuration to `GatewayRoute` (`HealthCheckPath`, `HealthCheckInterval`) +- Background worker to check upstream health +- Mark unhealthy routes and fallback if multiple targets exist +- File: `internal/core/services/gateway.go`, `internal/workers/` + +--- + +## Phase 3: Traffic Management (Medium Impact) + +### 3.1 Canary Deployments +**Impact:** Safe production rollouts +**Effort:** High + +- Add `Weight` field to routes (0-100) +- Support multiple routes with same pattern, different weights +- Traffic splitter logic in `GetProxy` +- File: `internal/core/domain/gateway.go`, `internal/core/services/gateway.go` + +### 3.2 Blue-Green Deployments +**Impact:** Instant rollback capability +**Effort:** High + +- Add `RouteGroup` concept (routes belong to groups) +- Add `Active` boolean to routes +- Atomic switch via management API +- File: `internal/core/domain/gateway.go`, `internal/handlers/gateway_handler.go` + +### 3.3 Traffic Mirroring +**Impact:** Test against real traffic without impact +**Effort:** Medium + +- Add `MirrorTarget` to `GatewayRoute` +- Fork requests to mirror target (async, don't wait for response) +- File: `internal/handlers/gateway_handler.go` + +--- + +## Phase 4: Security (High Impact) + +### 4.1 CORS Fine-Tuning Per Route +**Impact:** Proper CORS instead of wildcard +**Effort:** Low +**Status:** βœ… Shipped + +- Add `AllowedOrigins`, `AllowedMethods`, `AllowedHeaders`, `ExposeHeaders`, `MaxAge` to `GatewayRoute` +- Replace global CORS middleware with route-level +- File: `internal/core/domain/gateway.go`, `internal/handlers/gateway_handler.go` + +### 4.2 Header Filtering +**Impact:** Prevent information leakage +**Effort:** Low +**Status:** βœ… Shipped + +- Add `StripResponseHeaders` list to `GatewayRoute` +- Remove headers like `X-Powered-By`, `Server`, `X-AspNet-Version` +- File: `internal/core/domain/gateway.go`, `internal/handlers/gateway_handler.go` + +### 4.3 JWT Validation +**Impact:** Auth at gateway level +**Effort:** Medium +**Status:** βœ… Shipped + +- Add `JWTIssuer`, `JWTAudience`, `JWTJwksURL` to `GatewayRoute` +- Validate JWT before proxying, inject claims as headers +- File: `internal/core/domain/gateway.go`, `internal/handlers/gateway_handler.go`, `internal/core/services/gateway.go` + +### 4.4 mTLS for Backend Communication +**Impact:** Secure service-to-service communication +**Effort:** Medium +**Status:** βœ… Shipped + +- Add `ClientCert`, `ClientKey`, `CACert` fields for mTLS +- Configure TLS handshake with client certificates in `buildTLSConfig()` +- File: `internal/core/domain/gateway.go`, `internal/core/services/gateway.go` + +--- + +## Phase 5: Observability (High Impact) + +### 5.1 Per-Route Metrics +**Impact:** Debug which route is slow/failing +**Effort:** Low +**Status:** βœ… Shipped + +- Add `route_id` label to existing `HTTPRequestsTotal` and `HTTPRequestDuration` +- Track upstream response codes per route +- File: `internal/handlers/gateway_handler.go`, `pkg/httputil/middleware.go` + +### 5.2 Upstream Latency Metrics +**Impact:** Know when backends are slow +**Effort:** Low +**Status:** βœ… Shipped + +- Time the proxy roundtrip +- Export `gateway_upstream_latency_seconds` histogram +- File: `internal/handlers/gateway_handler.go` + +### 5.3 Circuit Breaker State Metrics +**Impact:** Monitor resilience state +**Effort:** Low +**Status:** βœ… Shipped + +- Export CB state as Prometheus gauge (0=closed, 1=open, 2=half-open) +- Per-route labeled +- File: `internal/core/services/gateway.go`, `internal/platform/circuit_breaker.go` + +### 5.4 Retry Attempt Metrics +**Impact:** Visibility into retry operations +**Effort:** Low +**Status:** βœ… Shipped + +- Counter for `gateway_retry_total` with `route_id`, `status` labels +- File: `internal/core/services/gateway.go` + +--- + +## Phase 6: Protocol Support (Medium Impact) + +### 6.1 WebSocket Path Routing +**Impact:** Consistent routing for all protocols +**Effort:** Medium + +- Integrate WebSocket into same pattern matching system +- Remove separate `ws/handler.go` routing +- File: `internal/handlers/gateway_handler.go`, `internal/handlers/ws/handler.go` + +### 6.2 SSE Support +**Impact:** Proxy Server-Sent Events +**Effort:** Low + +- Ensure `Transfer-Encoding: chunked` is handled properly +- Add `EnableSSE` flag if needed +- File: `internal/handlers/gateway_handler.go` + +### 6.3 gRPC Gateway +**Impact:** Expose gRPC services +**Effort:** High + +- Add `Protocol` field to `GatewayRoute` ("http", "grpc") +- Use `grpc/grpc-go` for gRPC proxying +- Handle HTTP/2 and gRPC framing +- File: `internal/core/domain/gateway.go`, `internal/handlers/gateway_handler.go` + +--- + +## Phase 7: Configuration & Management (Medium Impact) + +### 7.1 Route Versioning +**Impact:** Change history and rollback +**Effort:** Medium + +- Add `Version` field to routes +- Store route history in separate table +- Management API to list versions and rollback +- File: `internal/repositories/postgres/gateway_repo.go` + +### 7.2 Dry-Run Mode +**Impact:** Test routes before applying +**Effort:** Low +**Status:** βœ… Shipped + +- Add `?dry_run=true` to `POST /gateway/routes` +- Validate route config without persisting +- File: `internal/handlers/gateway_handler.go` + +### 7.3 YAML/JSON Config Import +**Impact:** Configuration-as-code +**Effort:** Medium + +- Add `POST /gateway/routes/import` endpoint +- Accept batch route definitions +- Validate and apply in transaction +- File: `internal/handlers/gateway_handler.go` + +### 7.4 Route Health Dashboard +**Impact:** Visual health overview +**Effort:** Medium + +- Add `GET /gateway/health` endpoint +- Return per-route health status from background checker +- File: `internal/handlers/gateway_handler.go` + +--- + +## Phase 8: Performance (Low-Medium Impact) + +### 8.1 Response Caching +**Impact:** Reduce backend load for immutable responses +**Effort:** High + +- Add `CacheTTL`, `CacheKeyHeaders` to `GatewayRoute` +- In-memory or Redis cache for GET responses +- Vary header support +- File: `internal/core/services/gateway.go`, `internal/handlers/gateway_handler.go` + +### 8.2 Compression +**Impact:** Reduce bandwidth +**Effort:** Low +**Status:** βœ… Shipped + +- Add `Compression` field (gzip, br, deflate) +- Compress responses if client accepts encoding +- File: `internal/handlers/gateway_handler.go` + +### 8.3 Request Coalescing (Singleflight) +**Impact:** Prevent thundering herd +**Effort:** Medium + +- Use `golang.org/x/sync/singleflight` for identical concurrent requests +- Key by route + path + query +- File: `internal/handlers/gateway_handler.go` + +### 8.4 Route Preload on Startup +**Impact:** No cold start latency +**Effort:** Low + +- Eager load all routes on service start +- Remove lazy loading in `GetProxy` +- File: `internal/core/services/gateway.go` + +--- + +## Implementation Order + +``` +Phase 1 (Quick Wins) +β”œβ”€β”€ 1.1 Add Upstream Timeouts βœ… +β”œβ”€β”€ 1.2 Per-Route Rate Limiting βœ… +β”œβ”€β”€ 1.3 IP Allowlist/Denylist βœ… +β”œβ”€β”€ 1.4 Request Size Limits βœ… +β”œβ”€β”€ 1.5 TLS Configuration βœ… +└── 1.6 Trace Header Propagation βœ… + +Phase 2 (Resilience) +β”œβ”€β”€ 2.1 Circuit Breaker Per Route βœ… +β”œβ”€β”€ 2.2 Retry on Upstream Failure βœ… +β”œβ”€β”€ 2.3 Connection Pooling βœ… +└── 2.4 Health Check & Auto-Failover + +Phase 5 (Observability - can parallelize with Phase 2) +β”œβ”€β”€ 5.1 Per-Route Metrics βœ… +β”œβ”€β”€ 5.2 Upstream Latency Metrics βœ… +β”œβ”€β”€ 5.3 Circuit Breaker State Metrics βœ… +└── 5.4 Retry Attempt Metrics βœ… + +Phase 4 (Security - can parallelize with Phase 2) +β”œβ”€β”€ 4.1 CORS Fine-Tuning βœ… +β”œβ”€β”€ 4.2 Header Filtering βœ… +β”œβ”€β”€ 4.3 JWT Validation βœ… +└── 4.4 mTLS for Backends βœ… + +Phase 3 (Traffic Management) +β”œβ”€β”€ 3.1 Canary Deployments +β”œβ”€β”€ 3.2 Blue-Green Deployments +└── 3.3 Traffic Mirroring + +Phase 6 (Protocol) +β”œβ”€β”€ 6.1 WebSocket Path Routing +β”œβ”€β”€ 6.2 SSE Support +└── 6.3 gRPC Gateway + +Phase 7 (Config & Management) +β”œβ”€β”€ 7.1 Route Versioning +β”œβ”€β”€ 7.2 Dry-Run Mode βœ… +β”œβ”€β”€ 7.3 YAML/JSON Import +└── 7.4 Route Health Dashboard + +Phase 8 (Performance) +β”œβ”€β”€ 8.1 Response Caching +β”œβ”€β”€ 8.2 Compression βœ… +β”œβ”€β”€ 8.3 Request Coalescing +└── 8.4 Route Preload +``` + +--- + +## Success Metrics + +| Phase | Metric | +|-------|--------| +| Phase 1 | Zero gateway hangs on slow backends | +| Phase 2 | βœ… 0 cascade failures inζ•…ιšœ scenarios | +| Phase 3 | <1% failed deployments via canary | +| Phase 4 | Zero unauthorized IP access | +| Phase 5 | p99 latency visible per route | +| Phase 6 | gRPC support shipped | +| Phase 7 | Route changes auditable | +| Phase 8 | p95 latency reduced by 20% | \ No newline at end of file diff --git a/docs/services/cloud-gateway.md b/docs/services/cloud-gateway.md index 5d1ca7a7a..87998f145 100644 --- a/docs/services/cloud-gateway.md +++ b/docs/services/cloud-gateway.md @@ -1,11 +1,78 @@ -CloudGateway provides entry-point routing, rate limiting, and dynamic path matching for your cloud infrastructure. +CloudGateway provides entry-point routing, rate limiting, JWT authentication, mTLS, and observability for your cloud infrastructure. ## Implementation - **Core Engine**: Built using Go's standard `net/http/httputil.ReverseProxy` with a custom routing engine. - **Advanced Path Matching**: Supports pattern-based routing with parameter extraction (regex-backed). - **Dynamic Routing**: Routes are stored in PostgreSQL and cached in-memory. The service pre-compiles route patterns for sub-millisecond matching. - **Path Stripping**: Optional prefix stripping (e.g., `/gw/v1/users` -> target: `/users`). -- **Rate Limiting**: Per-route rate limiting enforced at the gateway layer. +- **Rate Limiting**: Per-route rate limiting enforced at the gateway layer using token-bucket algorithm. +- **Circuit Breaker & Retry**: Per-route circuit breaker with configurable threshold and timeout. Automatic retry with exponential backoff for idempotent methods on `502`, `503`, `504`, `429`. +- **JWT Authentication**: JWKS-backed validation with issuer/audience verification. Claims propagated to upstream via `X-JWT-Claim-*` headers. +- **mTLS**: Configurable client certificates and CA certs for backend TLS verification. +- **CORS**: Per-route CORS configuration with origins, methods, headers, exposed headers, and max-age. +- **Compression**: Gzip response compression when client advertises support. +- **Observability**: Prometheus metrics for upstream latency, retry totals, circuit breaker state, and JWKS fetch operations. + +## Route Configuration + +Routes are created via `POST /gateway/routes` with the following configurable fields: + +| Field | Type | Description | +|-------|------|-------------| +| `name` | string | Human-readable route name | +| `path_prefix` | string | Path pattern to match (e.g., `/api/v1`, `/users/{id}`) | +| `target_url` | string | Backend URL to proxy to | +| `methods` | []string | Allowed HTTP methods (empty = all) | +| `strip_prefix` | bool | Strip matched prefix before forwarding | +| `rate_limit` | int | Max requests per second per client | +| `max_body_size` | int64 | Max request body in bytes | +| `dial_timeout` | int64 | TCP dial timeout in milliseconds | +| `idle_conn_timeout` | int64 | Idle connection timeout in milliseconds | +| `tls_skip_verify` | bool | Skip TLS verification for backend | +| `require_tls` | bool | Force HTTPS for backend | +| `allowed_cidrs` | []string | IP allowlist (empty = all allowed) | +| `blocked_cidrs` | []string | IP blocklist (checked before allowlist) | +| `circuit_breaker_threshold` | int | Failures to trip circuit breaker (0 = disabled) | +| `circuit_breaker_timeout` | int64 | Milliseconds in open state before half-open | +| `max_retries` | int | Max retry attempts (0 = disabled) | +| `retry_timeout` | int64 | Total retry window in milliseconds | +| `allowed_origins` | []string | CORS allowed origins | +| `allowed_methods` | []string | CORS allowed methods | +| `allowed_headers` | []string | CORS allowed headers | +| `expose_headers` | []string | CORS exposed headers | +| `max_age` | int | CORS preflight cache duration in seconds | +| `strip_response_headers` | []string | Headers to remove from upstream responses | +| `compression` | string | `gzip`, `br`, or `deflate` (empty = disabled) | +| `jwt_issuer` | string | Expected JWT issuer claim | +| `jwt_jwks_url` | string | JWKS endpoint for public key fetching | +| `jwt_audience` | string | Expected JWT audience claim | +| `client_cert` | string | PEM-encoded client certificate for mTLS | +| `client_key` | string | PEM-encoded private key for mTLS | +| `ca_cert` | string | PEM-encoded CA certificate for backend verification | + +### Dry-Run Validation + +Create routes with `?dry_run=true` to validate configuration without persisting: + +```bash +curl -X POST /gateway/routes?dry_run=true \ + -H "Content-Type: application/json" \ + -d '{"name":"test","path_prefix":"/api","target_url":"http://backend:8080"}' +``` + +Returns `{ "dry_run": true, "valid": true }` or `{ "dry_run": true, "valid": false, "errors": [...] }`. + +## JWT Authentication + +When a route has `jwt_jwks_url` configured: + +1. Gateway extracts `Authorization: Bearer ` header +2. Fetches and caches JWKS from the configured URL (5-minute TTL) +3. Validates token signature against fetched public keys +4. Verifies `iss` and `aud` claims match configuration +5. On success, propagates claims as `X-JWT-Claim-{key}` headers to upstream + +JWKS fetches are deduplicated via singleflight to prevent thundering herd. A circuit breaker (3 failures, 30s reset) protects against unhealthy JWKS endpoints. Metrics exported: `thecloud_jwks_fetch_total` (success/error/circuit_open), `thecloud_jwks_breaker_state`. ## Pattern Matching Syntax @@ -19,11 +86,24 @@ CloudGateway supports powerful pattern-based routing: | **Extension** | `/files/*.{ext}` | `/files/img.png` | `ext=png` | ### Routing Priority + Routes are evaluated based on **Specificity Scoring**: 1. Exact path matches are prioritized. 2. Longest pattern matches win tie-breaks. 3. Explicit `priority` field can be used for manual overrides. +## Observability Metrics + +| Metric | Type | Labels | Description | +|--------|------|--------|-------------| +| `thecloud_gateway_upstream_latency_seconds` | histogram | route_id, method, path | Upstream request duration | +| `thecloud_gateway_retry_total` | counter | route_id, status | Retry attempts | +| `thecloud_gateway_circuit_breaker_state` | gauge | route_id | Circuit breaker state (0=closed, 1=open, 2=half-open) | +| `thecloud_jwks_fetch_total` | counter | status | JWKS fetch attempts | +| `thecloud_jwks_breaker_state` | gauge | - | JWKS circuit breaker state | +| `thecloud_http_requests_total` | counter | method, path, status | Total HTTP requests | +| `thecloud_http_request_duration_seconds` | histogram | method, path | HTTP request duration | + ## CLI Usage ```bash @@ -36,11 +116,39 @@ cloud gateway create-route user-api "/users/{id}" http://user-service:8080 # Constrained parameter matching (numbers only) cloud gateway create-route post-api "/posts/{pid:[0-9]+}" http://posts-service:8080 +# Route with JWT authentication +cloud gateway create-route api "/api" http://backend:8080 \ + --jwt-issuer "https://auth.example.com" \ + --jwt-jwks-url "https://auth.example.com/.well-known/jwks.json" \ + --jwt-audience "my-api" + +# Route with mTLS +cloud gateway create-route secure-api "/secure" http://backend:8080 \ + --client-cert @/path/to/cert.pem \ + --client-key @/path/to/key.pem \ + --ca-cert @/path/to/ca.pem + +# Route with circuit breaker and retry +cloud gateway create-route resilient-api "/resilient" http://backend:8080 \ + --circuit-breaker-threshold 5 \ + --circuit-breaker-timeout 30000 \ + --max-retries 3 \ + --retry-timeout 5000 + +# Route with rate limiting (100 req/sec, burst 200) +cloud gateway create-route limited-api "/limited" http://backend:8080 \ + --rate-limit 100 + +# Route with CORS +cloud gateway create-route cors-api "/cors" http://backend:8080 \ + --allowed-origins "https://app.example.com" \ + --allowed-methods "GET,POST,PUT" \ + --allowed-headers "Authorization,Content-Type" \ + --max-age 3600 + +# Dry-run validation +cloud gateway create-route --dry-run ... + # List all active patterns cloud gateway list-routes ``` - -## Internal Context Injection -When a pattern match occurs, CloudGateway automatically extracts parameters and injects them into the request context. This allows integration with: -- **Downstream Headers**: `X-Path-Param-{Name}` headers can be injected (Future Feature). -- **IAM Policies**: Resource-level permissions using extracted IDs. diff --git a/docs/swagger/docs.go b/docs/swagger/docs.go index d63a0fc20..3137e4407 100644 --- a/docs/swagger/docs.go +++ b/docs/swagger/docs.go @@ -9991,6 +9991,27 @@ const docTemplate = `{ "type": "string" } }, + "allowed_headers": { + "description": "CORS allowed headers", + "type": "array", + "items": { + "type": "string" + } + }, + "allowed_methods": { + "description": "CORS allowed methods", + "type": "array", + "items": { + "type": "string" + } + }, + "allowed_origins": { + "description": "CORS allowed origins (empty = all)", + "type": "array", + "items": { + "type": "string" + } + }, "blocked_cidrs": { "description": "IPs blocked from access", "type": "array", @@ -9998,6 +10019,10 @@ const docTemplate = `{ "type": "string" } }, + "ca_cert": { + "description": "PEM-encoded CA cert for backend verification", + "type": "string" + }, "circuit_breaker_threshold": { "description": "consecutive failures to trip open (0=disabled)", "type": "integer" @@ -10006,6 +10031,18 @@ const docTemplate = `{ "description": "ms in open before half-open", "type": "integer" }, + "client_cert": { + "description": "PEM-encoded client certificate for mTLS", + "type": "string" + }, + "client_key": { + "description": "PEM-encoded private key for mTLS", + "type": "string" + }, + "compression": { + "description": "\"gzip\", \"br\", \"deflate\" (empty = disabled)", + "type": "string" + }, "created_at": { "type": "string" }, @@ -10013,6 +10050,13 @@ const docTemplate = `{ "description": "TCP dial timeout in milliseconds", "type": "integer" }, + "expose_headers": { + "description": "CORS exposed headers", + "type": "array", + "items": { + "type": "string" + } + }, "id": { "type": "string" }, @@ -10020,6 +10064,19 @@ const docTemplate = `{ "description": "Idle connection timeout in milliseconds", "type": "integer" }, + "jwt_audience": { + "type": "string" + }, + "jwt_issuer": { + "type": "string" + }, + "jwt_jwks_url": { + "type": "string" + }, + "max_age": { + "description": "CORS max-age preflight cache", + "type": "integer" + }, "max_body_size": { "description": "Max request body size in bytes", "type": "integer" @@ -10081,6 +10138,13 @@ const docTemplate = `{ "description": "If true, removes path_prefix from request before forwarding", "type": "boolean" }, + "strip_response_headers": { + "description": "headers to strip from upstream responses", + "type": "array", + "items": { + "type": "string" + } + }, "target_url": { "description": "Internal destination (e.g., \"http://service-a:8080\")", "type": "string" @@ -12596,12 +12660,33 @@ const docTemplate = `{ "type": "string" } }, + "allowed_headers": { + "type": "array", + "items": { + "type": "string" + } + }, + "allowed_methods": { + "type": "array", + "items": { + "type": "string" + } + }, + "allowed_origins": { + "type": "array", + "items": { + "type": "string" + } + }, "blocked_cidrs": { "type": "array", "items": { "type": "string" } }, + "ca_cert": { + "type": "string" + }, "circuit_breaker_threshold": { "type": "integer", "minimum": 0 @@ -12610,14 +12695,41 @@ const docTemplate = `{ "type": "integer", "minimum": 0 }, + "client_cert": { + "type": "string" + }, + "client_key": { + "type": "string" + }, + "compression": { + "type": "string" + }, "dial_timeout": { "type": "integer", "minimum": 0 }, + "expose_headers": { + "type": "array", + "items": { + "type": "string" + } + }, "idle_conn_timeout": { "type": "integer", "minimum": 0 }, + "jwt_audience": { + "type": "string" + }, + "jwt_issuer": { + "type": "string" + }, + "jwt_jwks_url": { + "type": "string" + }, + "max_age": { + "type": "integer" + }, "max_body_size": { "type": "integer", "minimum": 0 @@ -12660,6 +12772,12 @@ const docTemplate = `{ "strip_prefix": { "type": "boolean" }, + "strip_response_headers": { + "type": "array", + "items": { + "type": "string" + } + }, "target_url": { "type": "string" }, diff --git a/docs/swagger/swagger.json b/docs/swagger/swagger.json index 2160ec996..bd435008b 100644 --- a/docs/swagger/swagger.json +++ b/docs/swagger/swagger.json @@ -9983,6 +9983,27 @@ "type": "string" } }, + "allowed_headers": { + "description": "CORS allowed headers", + "type": "array", + "items": { + "type": "string" + } + }, + "allowed_methods": { + "description": "CORS allowed methods", + "type": "array", + "items": { + "type": "string" + } + }, + "allowed_origins": { + "description": "CORS allowed origins (empty = all)", + "type": "array", + "items": { + "type": "string" + } + }, "blocked_cidrs": { "description": "IPs blocked from access", "type": "array", @@ -9990,6 +10011,10 @@ "type": "string" } }, + "ca_cert": { + "description": "PEM-encoded CA cert for backend verification", + "type": "string" + }, "circuit_breaker_threshold": { "description": "consecutive failures to trip open (0=disabled)", "type": "integer" @@ -9998,6 +10023,18 @@ "description": "ms in open before half-open", "type": "integer" }, + "client_cert": { + "description": "PEM-encoded client certificate for mTLS", + "type": "string" + }, + "client_key": { + "description": "PEM-encoded private key for mTLS", + "type": "string" + }, + "compression": { + "description": "\"gzip\", \"br\", \"deflate\" (empty = disabled)", + "type": "string" + }, "created_at": { "type": "string" }, @@ -10005,6 +10042,13 @@ "description": "TCP dial timeout in milliseconds", "type": "integer" }, + "expose_headers": { + "description": "CORS exposed headers", + "type": "array", + "items": { + "type": "string" + } + }, "id": { "type": "string" }, @@ -10012,6 +10056,19 @@ "description": "Idle connection timeout in milliseconds", "type": "integer" }, + "jwt_audience": { + "type": "string" + }, + "jwt_issuer": { + "type": "string" + }, + "jwt_jwks_url": { + "type": "string" + }, + "max_age": { + "description": "CORS max-age preflight cache", + "type": "integer" + }, "max_body_size": { "description": "Max request body size in bytes", "type": "integer" @@ -10073,6 +10130,13 @@ "description": "If true, removes path_prefix from request before forwarding", "type": "boolean" }, + "strip_response_headers": { + "description": "headers to strip from upstream responses", + "type": "array", + "items": { + "type": "string" + } + }, "target_url": { "description": "Internal destination (e.g., \"http://service-a:8080\")", "type": "string" @@ -12588,12 +12652,33 @@ "type": "string" } }, + "allowed_headers": { + "type": "array", + "items": { + "type": "string" + } + }, + "allowed_methods": { + "type": "array", + "items": { + "type": "string" + } + }, + "allowed_origins": { + "type": "array", + "items": { + "type": "string" + } + }, "blocked_cidrs": { "type": "array", "items": { "type": "string" } }, + "ca_cert": { + "type": "string" + }, "circuit_breaker_threshold": { "type": "integer", "minimum": 0 @@ -12602,14 +12687,41 @@ "type": "integer", "minimum": 0 }, + "client_cert": { + "type": "string" + }, + "client_key": { + "type": "string" + }, + "compression": { + "type": "string" + }, "dial_timeout": { "type": "integer", "minimum": 0 }, + "expose_headers": { + "type": "array", + "items": { + "type": "string" + } + }, "idle_conn_timeout": { "type": "integer", "minimum": 0 }, + "jwt_audience": { + "type": "string" + }, + "jwt_issuer": { + "type": "string" + }, + "jwt_jwks_url": { + "type": "string" + }, + "max_age": { + "type": "integer" + }, "max_body_size": { "type": "integer", "minimum": 0 @@ -12652,6 +12764,12 @@ "strip_prefix": { "type": "boolean" }, + "strip_response_headers": { + "type": "array", + "items": { + "type": "string" + } + }, "target_url": { "type": "string" }, diff --git a/docs/swagger/swagger.yaml b/docs/swagger/swagger.yaml index 0be6ab10a..743a67254 100644 --- a/docs/swagger/swagger.yaml +++ b/docs/swagger/swagger.yaml @@ -406,27 +406,68 @@ definitions: items: type: string type: array + allowed_headers: + description: CORS allowed headers + items: + type: string + type: array + allowed_methods: + description: CORS allowed methods + items: + type: string + type: array + allowed_origins: + description: CORS allowed origins (empty = all) + items: + type: string + type: array blocked_cidrs: description: IPs blocked from access items: type: string type: array + ca_cert: + description: PEM-encoded CA cert for backend verification + type: string circuit_breaker_threshold: description: consecutive failures to trip open (0=disabled) type: integer circuit_breaker_timeout: description: ms in open before half-open type: integer + client_cert: + description: PEM-encoded client certificate for mTLS + type: string + client_key: + description: PEM-encoded private key for mTLS + type: string + compression: + description: '"gzip", "br", "deflate" (empty = disabled)' + type: string created_at: type: string dial_timeout: description: TCP dial timeout in milliseconds type: integer + expose_headers: + description: CORS exposed headers + items: + type: string + type: array id: type: string idle_conn_timeout: description: Idle connection timeout in milliseconds type: integer + jwt_audience: + type: string + jwt_issuer: + type: string + jwt_jwks_url: + type: string + max_age: + description: CORS max-age preflight cache + type: integer max_body_size: description: Max request body size in bytes type: integer @@ -472,6 +513,11 @@ definitions: strip_prefix: description: If true, removes path_prefix from request before forwarding type: boolean + strip_response_headers: + description: headers to strip from upstream responses + items: + type: string + type: array target_url: description: Internal destination (e.g., "http://service-a:8080") type: string @@ -2280,22 +2326,54 @@ definitions: items: type: string type: array + allowed_headers: + items: + type: string + type: array + allowed_methods: + items: + type: string + type: array + allowed_origins: + items: + type: string + type: array blocked_cidrs: items: type: string type: array + ca_cert: + type: string circuit_breaker_threshold: minimum: 0 type: integer circuit_breaker_timeout: minimum: 0 type: integer + client_cert: + type: string + client_key: + type: string + compression: + type: string dial_timeout: minimum: 0 type: integer + expose_headers: + items: + type: string + type: array idle_conn_timeout: minimum: 0 type: integer + jwt_audience: + type: string + jwt_issuer: + type: string + jwt_jwks_url: + type: string + max_age: + type: integer max_body_size: minimum: 0 type: integer @@ -2326,6 +2404,10 @@ definitions: type: integer strip_prefix: type: boolean + strip_response_headers: + items: + type: string + type: array target_url: type: string tls_skip_verify: diff --git a/go.mod b/go.mod index cd5efd8dc..500126a79 100644 --- a/go.mod +++ b/go.mod @@ -45,6 +45,7 @@ require ( go.opentelemetry.io/otel/sdk v1.42.0 go.opentelemetry.io/otel/trace v1.42.0 golang.org/x/crypto v0.47.0 + golang.org/x/sync v0.19.0 golang.org/x/time v0.14.0 google.golang.org/grpc v1.69.4 google.golang.org/protobuf v1.36.11 @@ -222,7 +223,6 @@ require ( golang.org/x/mod v0.32.0 // indirect golang.org/x/net v0.48.0 // indirect golang.org/x/oauth2 v0.27.0 // indirect - golang.org/x/sync v0.19.0 // indirect golang.org/x/sys v0.41.0 // indirect golang.org/x/term v0.39.0 // indirect golang.org/x/text v0.33.0 // indirect diff --git a/internal/api/setup/router.go b/internal/api/setup/router.go index 72465dcb5..e8407e378 100644 --- a/internal/api/setup/router.go +++ b/internal/api/setup/router.go @@ -236,32 +236,9 @@ func registerIAMRoutes(r *gin.Engine, handlers *Handlers, svcs *Services) { iamGroup.GET(policyIDRoute, handlers.IAM.GetPolicyByID) iamGroup.DELETE(policyIDRoute, handlers.IAM.DeletePolicy) - // Policy versioning - iamGroup.GET("/policies/:id/versions", handlers.IAM.GetPolicyVersions) - iamGroup.GET("/policies/:id/versions/:version", handlers.IAM.GetPolicyVersion) - iamGroup.POST("/policies/:id/rollback/:version", handlers.IAM.RollbackPolicyVersion) - iamGroup.POST("/users/:userId/policies/:policyId", handlers.IAM.AttachPolicyToUser) iamGroup.DELETE("/users/:userId/policies/:policyId", handlers.IAM.DetachPolicyFromUser) iamGroup.GET("/users/:userId/policies", handlers.IAM.GetUserPolicies) - - // Service account routes - iamGroup.POST("/service-accounts", handlers.IAM.CreateServiceAccount) - iamGroup.GET("/service-accounts", handlers.IAM.ListServiceAccounts) - iamGroup.GET("/service-accounts/:id", handlers.IAM.GetServiceAccount) - iamGroup.DELETE("/service-accounts/:id", handlers.IAM.DeleteServiceAccount) - - iamGroup.POST("/service-accounts/:id/secrets/:secretId", handlers.IAM.RevokeServiceAccountSecret) - iamGroup.GET("/service-accounts/:id/secrets", handlers.IAM.ListServiceAccountSecrets) - iamGroup.POST("/service-accounts/:id/rotate-secret", handlers.IAM.RotateServiceAccountSecret) - - // SA policy attachment - iamGroup.POST("/service-accounts/:id/policies/:policyId", handlers.IAM.AttachPolicyToServiceAccount) - iamGroup.DELETE("/service-accounts/:id/policies/:policyId", handlers.IAM.DetachPolicyFromServiceAccount) - iamGroup.GET("/service-accounts/:id/policies", handlers.IAM.GetServiceAccountPolicies) - - // Policy simulation - iamGroup.POST("/simulate", handlers.IAM.Simulate) } } @@ -278,9 +255,6 @@ func registerAuthRoutes(r *gin.Engine, handlers *Handlers, svcs *Services, cfg * r.POST("/auth/forgot-password", authMiddleware, handlers.Auth.ForgotPassword) r.POST("/auth/reset-password", authMiddleware, handlers.Auth.ResetPassword) - // OAuth2 token endpoint (rate-limited like login) - r.POST("/oauth2/token", authMiddleware, handlers.Auth.Token) - keyGroup := r.Group("/auth/keys") keyGroup.Use(httputil.Auth(svcs.Identity, svcs.Tenant)) { @@ -308,9 +282,6 @@ func registerComputeRoutes(r *gin.Engine, handlers *Handlers, svcs *Services) { instanceGroup.GET("/:id/stats", httputil.Permission(svcs.RBAC, domain.PermissionInstanceRead), handlers.Instance.GetStats) instanceGroup.GET("/:id/console", httputil.Permission(svcs.RBAC, domain.PermissionInstanceRead), handlers.Instance.GetConsole) instanceGroup.PUT("/:id/metadata", httputil.Permission(svcs.RBAC, domain.PermissionInstanceUpdate), handlers.Instance.UpdateMetadata) - instanceGroup.GET("/:id/tags", httputil.Permission(svcs.RBAC, domain.PermissionInstanceRead), handlers.Instance.GetTags) - instanceGroup.POST("/:id/tags", httputil.Permission(svcs.RBAC, domain.PermissionInstanceUpdate), handlers.Instance.SetTags) - instanceGroup.DELETE("/:id/tags/:key", httputil.Permission(svcs.RBAC, domain.PermissionInstanceUpdate), handlers.Instance.RemoveTag) instanceGroup.POST("/:id/resize", httputil.Permission(svcs.RBAC, domain.PermissionInstanceResize), handlers.Instance.ResizeInstance) instanceGroup.DELETE("/:id", httputil.Permission(svcs.RBAC, domain.PermissionInstanceTerminate), handlers.Instance.Terminate) } @@ -382,7 +353,6 @@ func registerNetworkRoutes(r *gin.Engine, handlers *Handlers, svcs *Services) { vpcGroup.POST("", httputil.Permission(svcs.RBAC, domain.PermissionVpcCreate), handlers.Vpc.Create) vpcGroup.GET("", httputil.Permission(svcs.RBAC, domain.PermissionVpcRead), handlers.Vpc.List) vpcGroup.GET("/:id", httputil.Permission(svcs.RBAC, domain.PermissionVpcRead), handlers.Vpc.Get) - vpcGroup.PATCH("/:id", httputil.Permission(svcs.RBAC, domain.PermissionVpcUpdate), handlers.Vpc.Update) vpcGroup.DELETE("/:id", httputil.Permission(svcs.RBAC, domain.PermissionVpcDelete), handlers.Vpc.Delete) vpcGroup.POST("/:id/subnets", httputil.Permission(svcs.RBAC, domain.PermissionVpcUpdate), handlers.Subnet.Create) @@ -543,7 +513,6 @@ func registerDataRoutes(r *gin.Engine, handlers *Handlers, svcs *Services) { volumeGroup.DELETE("/:id", httputil.Permission(svcs.RBAC, domain.PermissionVolumeDelete), handlers.Volume.Delete) volumeGroup.POST("/:id/attach", httputil.Permission(svcs.RBAC, domain.PermissionVolumeUpdate), handlers.Volume.Attach) volumeGroup.POST("/:id/detach", httputil.Permission(svcs.RBAC, domain.PermissionVolumeUpdate), handlers.Volume.Detach) - volumeGroup.POST("/:id/resize", httputil.Permission(svcs.RBAC, domain.PermissionVolumeUpdate), handlers.Volume.Resize) } dbGroup := r.Group("/databases") @@ -563,7 +532,6 @@ func registerDataRoutes(r *gin.Engine, handlers *Handlers, svcs *Services) { dbGroup.POST("/:id/rotate-credentials", httputil.Permission(svcs.RBAC, domain.PermissionDBUpdate), handlers.Database.RotateCredentials) dbGroup.POST("/:id/stop", httputil.Permission(svcs.RBAC, domain.PermissionDBUpdate), handlers.Database.Stop) dbGroup.POST("/:id/start", httputil.Permission(svcs.RBAC, domain.PermissionDBUpdate), handlers.Database.Start) - dbGroup.POST("/:id/resize", httputil.Permission(svcs.RBAC, domain.PermissionDBUpdate), handlers.Database.Resize) } cacheGroup := r.Group("/caches") @@ -576,7 +544,6 @@ func registerDataRoutes(r *gin.Engine, handlers *Handlers, svcs *Services) { cacheGroup.GET("/:id/connection", httputil.Permission(svcs.RBAC, domain.PermissionCacheRead), handlers.Cache.GetConnectionString) cacheGroup.POST("/:id/flush", httputil.Permission(svcs.RBAC, domain.PermissionCacheUpdate), handlers.Cache.Flush) cacheGroup.GET("/:id/stats", httputil.Permission(svcs.RBAC, domain.PermissionCacheRead), handlers.Cache.GetStats) - cacheGroup.POST("/:id/resize", httputil.Permission(svcs.RBAC, domain.PermissionCacheUpdate), handlers.Cache.Resize) } secretGroup := r.Group("/secrets") @@ -600,8 +567,6 @@ func registerDevOpsRoutes(r *gin.Engine, handlers *Handlers, svcs *Services) { fnGroup.DELETE("/:id", httputil.Permission(svcs.RBAC, domain.PermissionFunctionDelete), handlers.Function.Delete) fnGroup.POST("/:id/invoke", httputil.Permission(svcs.RBAC, domain.PermissionFunctionInvoke), handlers.Function.Invoke) fnGroup.GET("/:id/logs", httputil.Permission(svcs.RBAC, domain.PermissionFunctionRead), handlers.Function.GetLogs) - fnGroup.GET("/:id/dlq", httputil.Permission(svcs.RBAC, domain.PermissionFunctionRead), handlers.Function.GetDLQ) - fnGroup.POST("/:id/dlq/:invocation_id/retry", httputil.Permission(svcs.RBAC, domain.PermissionFunctionInvoke), handlers.Function.RetryDLQ) } fnSchedGroup := r.Group("/function-schedules") @@ -650,8 +615,6 @@ func registerDevOpsRoutes(r *gin.Engine, handlers *Handlers, svcs *Services) { cronGroup.DELETE("/jobs/:id", httputil.Permission(svcs.RBAC, domain.PermissionCronDelete), handlers.Cron.DeleteJob) cronGroup.POST("/jobs/:id/pause", httputil.Permission(svcs.RBAC, domain.PermissionCronUpdate), handlers.Cron.PauseJob) cronGroup.POST("/jobs/:id/resume", httputil.Permission(svcs.RBAC, domain.PermissionCronUpdate), handlers.Cron.ResumeJob) - cronGroup.GET("/jobs/:id/runs", httputil.Permission(svcs.RBAC, domain.PermissionCronRead), handlers.Cron.GetJobRuns) - cronGroup.PUT("/jobs/:id", httputil.Permission(svcs.RBAC, domain.PermissionCronUpdate), handlers.Cron.UpdateJob) } gatewayGroup := r.Group("/gateway") @@ -754,12 +717,6 @@ func registerAdminRoutes(r *gin.Engine, handlers *Handlers, svcs *Services) { billingGroup.GET("/summary", handlers.Accounting.GetSummary) billingGroup.GET("/usage", handlers.Accounting.ListUsage) } - - // Internal admin endpoints (E2E test support) - internalGroup := r.Group("/internal/admin") - { - internalGroup.POST("/reset-circuit-breakers", handlers.Admin.ResetCircuitBreakers) - } } func registerLogRoutes(r *gin.Engine, handlers *Handlers, svcs *Services) { diff --git a/internal/core/domain/gateway.go b/internal/core/domain/gateway.go index 2170e6db2..b6b755ef3 100644 --- a/internal/core/domain/gateway.go +++ b/internal/core/domain/gateway.go @@ -37,6 +37,19 @@ type GatewayRoute struct { MaxRetries int `json:"max_retries,omitempty"` // max retry attempts (0=disabled) RetryTimeout int64 `json:"retry_timeout,omitempty"` // total retry window in ms Priority int `json:"priority"` // Manual priority for tie-breaking + AllowedOrigins []string `json:"allowed_origins,omitempty"` // CORS allowed origins (empty = all) + AllowedMethods []string `json:"allowed_methods,omitempty"` // CORS allowed methods + AllowedHeaders []string `json:"allowed_headers,omitempty"` // CORS allowed headers + ExposeHeaders []string `json:"expose_headers,omitempty"` // CORS exposed headers + MaxAge int `json:"max_age,omitempty"` // CORS max-age preflight cache + StripResponseHeaders []string `json:"strip_response_headers,omitempty"` // headers to strip from upstream responses + Compression string `json:"compression,omitempty"` // "gzip", "br", "deflate" (empty = disabled) + JWTIssuer string `json:"jwt_issuer,omitempty"` + JWTJwksURL string `json:"jwt_jwks_url,omitempty"` + JWTAudience string `json:"jwt_audience,omitempty"` + ClientCert string `json:"client_cert,omitempty"` // PEM-encoded client certificate for mTLS + ClientKey string `json:"client_key,omitempty"` // PEM-encoded private key for mTLS + CACert string `json:"ca_cert,omitempty"` // PEM-encoded CA cert for backend verification CreatedAt time.Time `json:"created_at"` UpdatedAt time.Time `json:"updated_at"` } diff --git a/internal/core/ports/gateway.go b/internal/core/ports/gateway.go index c33dbeff3..695b7e5cb 100644 --- a/internal/core/ports/gateway.go +++ b/internal/core/ports/gateway.go @@ -47,6 +47,19 @@ type CreateRouteParams struct { MaxRetries int RetryTimeout int64 Priority int + AllowedOrigins []string + AllowedMethods []string + AllowedHeaders []string + ExposeHeaders []string + MaxAge int + StripResponseHeaders []string + Compression string + JWTIssuer string + JWTJwksURL string + JWTAudience string + ClientCert string + ClientKey string + CACert string } // GatewayService provides business logic for managing the API gateway and ingress traffic. @@ -62,4 +75,7 @@ type GatewayService interface { // GetProxy finds the appropriate backend for the given path and method. // Returns proxy, route, path params, and found flag. GetProxy(method, path string) (*httputil.ReverseProxy, *domain.GatewayRoute, map[string]string, bool) + // ValidateJWT validates a JWT token against the route's JWKS configuration. + // Returns claims map if valid, or error. + ValidateJWT(ctx context.Context, route *domain.GatewayRoute, tokenString string) (map[string]string, error) } diff --git a/internal/core/services/gateway.go b/internal/core/services/gateway.go index 74b4fa484..fa6a6525a 100644 --- a/internal/core/services/gateway.go +++ b/internal/core/services/gateway.go @@ -3,13 +3,18 @@ package services import ( "context" + cryptoRand "crypto/rand" + "crypto/rsa" "crypto/tls" + "crypto/x509" + "encoding/base64" + "encoding/json" stderrors "errors" "fmt" "io" "log/slog" "math" - "math/rand/v2" + "math/big" "net" "net/http" "net/http/httputil" @@ -17,8 +22,10 @@ import ( "sort" "strings" "sync" + "sync/atomic" "time" + "github.com/golang-jwt/jwt/v5" "github.com/google/uuid" appcontext "github.com/poyrazk/thecloud/internal/core/context" "github.com/poyrazk/thecloud/internal/core/domain" @@ -26,31 +33,49 @@ import ( "github.com/poyrazk/thecloud/internal/errors" "github.com/poyrazk/thecloud/internal/platform" "github.com/poyrazk/thecloud/internal/routing" + "golang.org/x/sync/singleflight" ) // GatewayService manages API gateway routes and reverse proxies. type GatewayService struct { - repo ports.GatewayRepository - rbacSvc ports.RBACService - proxyMu sync.RWMutex - proxies map[uuid.UUID]*httputil.ReverseProxy - routes []*domain.GatewayRoute - matchers map[uuid.UUID]*routing.PatternMatcher - auditSvc ports.AuditService - logger *slog.Logger + repo ports.GatewayRepository + rbacSvc ports.RBACService + proxyMu sync.RWMutex + proxies map[uuid.UUID]*httputil.ReverseProxy + routes []*domain.GatewayRoute + matchers map[uuid.UUID]*routing.PatternMatcher + auditSvc ports.AuditService + logger *slog.Logger + jwksMu sync.RWMutex + jwksCache map[string]*jwksCacheEntry + httpClient *http.Client + jwksCircuitBreaker *platform.CircuitBreaker + jwksInFlight singleflight.Group } // NewGatewayService constructs a GatewayService and loads existing routes. func NewGatewayService(repo ports.GatewayRepository, rbacSvc ports.RBACService, auditSvc ports.AuditService, logger *slog.Logger) *GatewayService { s := &GatewayService{ - repo: repo, - rbacSvc: rbacSvc, - proxies: make(map[uuid.UUID]*httputil.ReverseProxy), - routes: make([]*domain.GatewayRoute, 0), - matchers: make(map[uuid.UUID]*routing.PatternMatcher), - auditSvc: auditSvc, - logger: logger, - } + repo: repo, + rbacSvc: rbacSvc, + proxies: make(map[uuid.UUID]*httputil.ReverseProxy), + routes: make([]*domain.GatewayRoute, 0), + matchers: make(map[uuid.UUID]*routing.PatternMatcher), + auditSvc: auditSvc, + logger: logger, + httpClient: &http.Client{Timeout: 10 * time.Second}, + } + s.jwksCircuitBreaker = platform.NewCircuitBreakerWithOpts(platform.CircuitBreakerOpts{ + Name: "jwks-fetch", + Threshold: 3, + ResetTimeout: 30 * time.Second, + OnStateChange: func(name string, from, to platform.State) { + platform.JWKSBreakerState.Set(float64(to)) + if s.logger != nil { + s.logger.Warn("JWKS circuit breaker state change", "name", name, "from", from.String(), "to", to.String()) + } + }, + }) // Initial load if err := s.RefreshRoutes(context.Background()); err != nil { s.logger.Error("failed to refresh routes on startup", "error", err) @@ -104,6 +129,19 @@ func (s *GatewayService) CreateRoute(ctx context.Context, params ports.CreateRou MaxRetries: params.MaxRetries, RetryTimeout: params.RetryTimeout, Priority: params.Priority, + AllowedOrigins: params.AllowedOrigins, + AllowedMethods: params.AllowedMethods, + AllowedHeaders: params.AllowedHeaders, + ExposeHeaders: params.ExposeHeaders, + MaxAge: params.MaxAge, + StripResponseHeaders: params.StripResponseHeaders, + Compression: params.Compression, + JWTIssuer: params.JWTIssuer, + JWTJwksURL: params.JWTJwksURL, + JWTAudience: params.JWTAudience, + ClientCert: params.ClientCert, + ClientKey: params.ClientKey, + CACert: params.CACert, CreatedAt: time.Now(), UpdatedAt: time.Now(), } @@ -253,6 +291,11 @@ func (s *GatewayService) createReverseProxy(route *domain.GatewayRoute) (*httput proxy := httputil.NewSingleHostReverseProxy(target) // Configure custom transport with timeouts and TLS + tlsConfig, err := s.buildTLSConfig(route) + if err != nil { + return nil, fmt.Errorf("failed to build TLS config: %w", err) + } + dialTimeout := time.Duration(route.DialTimeout) * time.Millisecond if dialTimeout <= 0 { dialTimeout = 5 * time.Second @@ -273,8 +316,10 @@ func (s *GatewayService) createReverseProxy(route *domain.GatewayRoute) (*httput }).DialContext, ResponseHeaderTimeout: responseHeaderTimeout, IdleConnTimeout: idleConnTimeout, - TLSClientConfig: s.buildTLSConfig(route), + TLSClientConfig: tlsConfig, TLSHandshakeTimeout: 10 * time.Second, + MaxIdleConns: 100, + MaxIdleConnsPerHost: 10, } proxy.Transport = newRetryTransport(baseTransport, route, s.logger) @@ -298,7 +343,7 @@ func (s *GatewayService) createReverseProxy(route *domain.GatewayRoute) (*httput return proxy, nil } -func (s *GatewayService) buildTLSConfig(route *domain.GatewayRoute) *tls.Config { +func (s *GatewayService) buildTLSConfig(route *domain.GatewayRoute) (*tls.Config, error) { cfg := &tls.Config{ InsecureSkipVerify: route.TLSSkipVerify, //nolint:gosec // User-controlled option for development/testing } @@ -307,7 +352,22 @@ func (s *GatewayService) buildTLSConfig(route *domain.GatewayRoute) *tls.Config if route.RequireTLS { cfg.MinVersion = tls.VersionTLS13 } - return cfg + // mTLS: load client certificate if provided + if route.ClientCert != "" && route.ClientKey != "" { + cert, err := tls.X509KeyPair([]byte(route.ClientCert), []byte(route.ClientKey)) + if err != nil { + return nil, fmt.Errorf("invalid client cert/key: %w", err) + } + cfg.Certificates = []tls.Certificate{cert} + } + // CA cert for backend verification + if route.CACert != "" { + cfg.RootCAs = x509.NewCertPool() + if !cfg.RootCAs.AppendCertsFromPEM([]byte(route.CACert)) { + return nil, fmt.Errorf("failed to parse CA certificate") + } + } + return cfg, nil } func (s *GatewayService) sortRoutes(routes []*domain.GatewayRoute) { @@ -319,6 +379,203 @@ func (s *GatewayService) sortRoutes(routes []*domain.GatewayRoute) { }) } +type jwksCacheEntry struct { + keys map[string]*rsa.PublicKey + fetchedAt time.Time +} + +func (s *GatewayService) getJWKS(url string) (map[string]*rsa.PublicKey, error) { + s.jwksMu.Lock() + if entry, ok := s.jwksCache[url]; ok && time.Since(entry.fetchedAt) < 5*time.Minute { + s.jwksMu.Unlock() + return entry.keys, nil + } + s.jwksMu.Unlock() + + // Use singleflight to dedupe concurrent requests for the same URL + val, err, _ := s.jwksInFlight.Do(url, func() (any, error) { + s.jwksMu.Lock() + // Double-check after acquiring lock + if entry, ok := s.jwksCache[url]; ok && time.Since(entry.fetchedAt) < 5*time.Minute { + s.jwksMu.Unlock() + return entry.keys, nil + } + s.jwksMu.Unlock() + + req, err := http.NewRequest(http.MethodGet, url, nil) + if err != nil { + return nil, err + } + var resp *http.Response + if cbErr := s.jwksCircuitBreaker.Execute(func() error { + resp, err = s.httpClient.Do(req) //nolint:bodyclose + return err + }); cbErr != nil { + platform.JWKSFetchTotal.WithLabelValues("circuit_open").Inc() + return nil, cbErr + } + defer func() { + if resp != nil && resp.Body != nil { + resp.Body.Close() + } + }() + + // Only cache successful JWKS responses + if resp.StatusCode != http.StatusOK { + platform.JWKSFetchTotal.WithLabelValues("error").Inc() + return nil, fmt.Errorf("JWKS fetch returned status %d", resp.StatusCode) + } + + var jwks struct { + Keys []map[string]any `json:"keys"` + } + if err := json.NewDecoder(resp.Body).Decode(&jwks); err != nil { + platform.JWKSFetchTotal.WithLabelValues("error").Inc() + return nil, err + } + + keys := make(map[string]*rsa.PublicKey) + for _, k := range jwks.Keys { + if kid, ok := k["kid"].(string); ok { + if kty, _ := k["kty"].(string); kty == "RSA" { + if pubKey, err := parseRSAPublicKeyFromJWK(k); err == nil { + keys[kid] = pubKey + } + } + } + } + // If JWKS had keys but none parsed successfully, don't cache an empty result + if len(keys) == 0 && len(jwks.Keys) > 0 { + platform.JWKSFetchTotal.WithLabelValues("error").Inc() + return nil, fmt.Errorf("JWKS returned %d keys but none were valid RSA keys", len(jwks.Keys)) + } + + platform.JWKSFetchTotal.WithLabelValues("success").Inc() + s.jwksMu.Lock() + if s.jwksCache == nil { + s.jwksCache = make(map[string]*jwksCacheEntry) + } + s.jwksCache[url] = &jwksCacheEntry{keys: keys, fetchedAt: time.Now()} + s.jwksMu.Unlock() + return keys, nil + }) + if err != nil { + return nil, err + } + return val.(map[string]*rsa.PublicKey), nil +} + +// parseRSAPublicKeyFromJWK parses an RSA public key from a JWK map. +func parseRSAPublicKeyFromJWK(k map[string]any) (*rsa.PublicKey, error) { + nVal, ok := k["n"].(string) + if !ok { + return nil, fmt.Errorf("JWK missing 'n'") + } + eVal, ok := k["e"].(string) + if !ok { + return nil, fmt.Errorf("JWK missing 'e'") + } + + nBytes, err := base64.RawURLEncoding.DecodeString(nVal) + if err != nil { + return nil, fmt.Errorf("failed to decode modulus: %w", err) + } + eBytes, err := base64.RawURLEncoding.DecodeString(eVal) + if err != nil { + return nil, fmt.Errorf("failed to decode exponent: %w", err) + } + + n := new(big.Int).SetBytes(nBytes) + // e is typically 65537 (0x10001) which encodes as "AQAB" in base64url + e := 0 + for _, b := range eBytes { + if e >= 1<<30 { // exponents > 2^30 are invalid for RSA + return nil, fmt.Errorf("exponent too large") + } + e = e<<8 + int(b) + } + + return &rsa.PublicKey{N: n, E: e}, nil +} + +func (s *GatewayService) ValidateJWT(ctx context.Context, route *domain.GatewayRoute, tokenString string) (map[string]string, error) { + if route.JWTJwksURL == "" || tokenString == "" { + return nil, nil + } + + keys, err := s.getJWKS(route.JWTJwksURL) + if err != nil { + return nil, fmt.Errorf("failed to fetch JWKS: %w", err) + } + + token, err := jwt.Parse(tokenString, func(t *jwt.Token) (any, error) { + // Only allow RSA signature algorithms to prevent algorithm confusion attacks + switch t.Method.Alg() { + case "RS256", "RS384", "RS512": + // OK + default: + return nil, fmt.Errorf("unexpected signing method: %v", t.Method.Alg()) + } + kid, _ := t.Header["kid"].(string) + if kid == "" { + return nil, fmt.Errorf("kid not found in token header") + } + key, ok := keys[kid] + if !ok { + return nil, fmt.Errorf("key %s not found in JWKS", kid) + } + return key, nil + }) + if err != nil { + // Return generic error to avoid leaking timing info about signature validation + return nil, fmt.Errorf("invalid token") + } + + if !token.Valid { + return nil, fmt.Errorf("token is invalid") + } + + claims, _ := token.Claims.(jwt.MapClaims) + if route.JWTIssuer != "" { + if iss, _ := claims["iss"].(string); iss != route.JWTIssuer { + return nil, fmt.Errorf("invalid issuer") + } + } + if route.JWTAudience != "" { + if !verifyAudience(claims, route.JWTAudience) { + return nil, fmt.Errorf("invalid audience") + } + } + + result := make(map[string]string) + for k, v := range claims { + if str, ok := v.(string); ok { + result[k] = str + } + } + return result, nil +} + +// verifyAudience checks if the given audience claim matches the expected audience. +// Handles both string and []interface{} audience claims per RFC 7519. +func verifyAudience(claims jwt.MapClaims, expected string) bool { + aud, ok := claims["aud"] + if !ok { + return false + } + switch v := aud.(type) { + case string: + return v == expected + case []interface{}: + for _, a := range v { + if s, ok := a.(string); ok && s == expected { + return true + } + } + } + return false +} + // ProxyHandler is handled in the API layer for now func (s *GatewayService) GetProxy(method, path string) (*httputil.ReverseProxy, *domain.GatewayRoute, map[string]string, bool) { @@ -337,7 +594,15 @@ func (s *GatewayService) GetProxy(method, path string) (*httputil.ReverseProxy, } if bestMatch != nil { - return s.proxies[bestMatch.Route.ID], bestMatch.Route, bestMatch.Params, true + proxy := s.proxies[bestMatch.Route.ID] + if proxy == nil { + s.logger.Error("proxy not found for matched route, possible proxy creation failure", + "route_id", bestMatch.Route.ID.String(), + "route_name", bestMatch.Route.Name, + "path", path) + return nil, nil, nil, false + } + return proxy, bestMatch.Route, bestMatch.Params, true } return nil, nil, nil, false @@ -408,6 +673,12 @@ type retryTransport struct { maxRetries int retryTimeout time.Duration logger *slog.Logger + routeID string + // fastFailThreshold prevents retry storms when upstream is unreachable. + // When >0, consecutive connection errors exceeding this count trips the + // circuit breaker immediately (bypassing normal failure counting). + fastFailThreshold int32 + consecutiveConnErrors atomic.Int32 } // retryableStatusError wraps a response returned when retries are exhausted @@ -427,10 +698,12 @@ func (e *retryableStatusError) Error() string { // newRetryTransport wraps a base http.Transport with per-route retry and circuit breaker behavior. func newRetryTransport(base http.RoundTripper, route *domain.GatewayRoute, logger *slog.Logger) *retryTransport { rt := &retryTransport{ - base: base, - maxRetries: route.MaxRetries, - retryTimeout: time.Duration(route.RetryTimeout) * time.Millisecond, - logger: logger, + base: base, + maxRetries: route.MaxRetries, + retryTimeout: time.Duration(route.RetryTimeout) * time.Millisecond, + logger: logger, + routeID: route.ID.String(), + fastFailThreshold: int32(route.CircuitBreakerThreshold), //nolint:gosec } if route.CircuitBreakerThreshold > 0 { rt.cb = platform.NewCircuitBreakerWithOpts(platform.CircuitBreakerOpts{ @@ -438,6 +711,7 @@ func newRetryTransport(base http.RoundTripper, route *domain.GatewayRoute, logge Threshold: route.CircuitBreakerThreshold, ResetTimeout: time.Duration(route.CircuitBreakerTimeout) * time.Millisecond, OnStateChange: func(name string, from, to platform.State) { + platform.GatewayCircuitBreakerState.WithLabelValues(name).Set(float64(to)) if logger != nil { logger.Warn("circuit breaker state change", "route_id", name, @@ -446,6 +720,7 @@ func newRetryTransport(base http.RoundTripper, route *domain.GatewayRoute, logge } }, }) + platform.GatewayCircuitBreakerState.WithLabelValues(route.ID.String()).Set(float64(platform.StateClosed)) } return rt } @@ -489,6 +764,7 @@ func (rt *retryTransport) doRoundTrip(req *http.Request) (*http.Response, error) } var lastResp *http.Response + var lastErr error maxAttempts := rt.maxRetries + 1 // first attempt + retries start := time.Now() @@ -498,6 +774,7 @@ func (rt *retryTransport) doRoundTrip(req *http.Request) (*http.Response, error) break } if attempt > 0 { + platform.GatewayRetryTotal.WithLabelValues(rt.routeID, "retry").Inc() delay := rt.backoffWithJitter(attempt) select { case <-req.Context().Done(): @@ -508,19 +785,44 @@ func (rt *retryTransport) doRoundTrip(req *http.Request) (*http.Response, error) resp, err := rt.base.RoundTrip(req) if err == nil { + // Reset consecutive error counter on success + rt.consecutiveConnErrors.Store(0) if !rt.isRetryableStatus(resp.StatusCode) { + // For successful non-retryable responses, do NOT drain or close the body. + // The caller (ReverseProxy) needs to read from it to forward to the client. + // The transport's underlying connection management handles reuse. return resp, nil //nolint:bodyclose } - // drain and close body so connection can be reused, then retry + // drain body (do NOT close β€” lastResp may be returned to caller with readable body) _, _ = io.Copy(io.Discard, resp.Body) - _ = resp.Body.Close() lastResp = resp continue } if !rt.isRetryableError(err) { + // Drain body before returning so connection can be reused + if resp != nil && resp.Body != nil { + _, _ = io.Copy(io.Discard, resp.Body) + resp.Body.Close() + } + // Reset consecutive errors on non-retryable error (upstream responded) + rt.consecutiveConnErrors.Store(0) return nil, err } + + // Fast-fail: if we hit too many consecutive connection errors, trip the + // circuit breaker immediately to prevent retry storms. + if rt.cb != nil && rt.fastFailThreshold > 0 { + //nolint:G115 // int->int32 safe: threshold is small positive int + if rt.consecutiveConnErrors.Add(1) >= rt.fastFailThreshold { + platform.GatewayRetryTotal.WithLabelValues(rt.routeID, "fast_fail").Inc() + // Trip the circuit breaker open immediately + rt.cb.RecordFailure() + return nil, fmt.Errorf("fast-fail: too many consecutive connection errors: %w", err) + } + } + + lastErr = err lastResp = resp // For idempotent methods with a replayable body, clone the request before retry. @@ -538,8 +840,18 @@ func (rt *retryTransport) doRoundTrip(req *http.Request) (*http.Response, error) // so the circuit breaker can count this as a failure. The response is // unwrapped and returned to the caller in RoundTrip. if lastResp != nil && rt.isRetryableStatus(lastResp.StatusCode) { + platform.GatewayRetryTotal.WithLabelValues(rt.routeID, "exhausted").Inc() return nil, &retryableStatusError{resp: lastResp} } + // lastResp may have an undrained body from a network error β€” drain before returning + if lastResp != nil && lastResp.Body != nil { + _, _ = io.Copy(io.Discard, lastResp.Body) + lastResp.Body.Close() + } + // If we have a last error, return it; otherwise return the last response + if lastErr != nil { + return nil, lastErr + } return lastResp, nil //nolint:bodyclose } @@ -551,6 +863,13 @@ func (rt *retryTransport) isRetryableError(err error) bool { if err == nil { return false } + // Use net.Error interface for robust detection of transient errors + var netErr net.Error + if stderrors.As(err, &netErr) { + // Use Timeout() - Temporary() is deprecated per Go docs as unreliable + return netErr.Timeout() + } + // Fallback to string matching for errors not wrapped as net.Error msg := err.Error() return strings.Contains(msg, "connection refused") || strings.Contains(msg, "timeout") || @@ -578,10 +897,11 @@ func (rt *retryTransport) backoffWithJitter(attempt int) time.Duration { return rt.jitter(time.Duration(delay)) } -// jitter returns a random duration in [0, max) using math/rand/v2. -// rand.Uint() is safe for concurrent use; no locking needed. +// jitter returns a random duration in [0, max) using crypto/rand. +// crypto/rand is safe for concurrent use and provides cryptographic randomness. func (rt *retryTransport) jitter(max time.Duration) time.Duration { - val := float64(rand.Uint()) / float64(1<<64) * float64(math.MaxUint64) - frac := val / float64(math.MaxUint64) - return time.Duration(float64(max) * frac) + b := make([]byte, 8) + _, _ = cryptoRand.Read(b) + val := float64(uint64(b[0])<<56 | uint64(b[1])<<48 | uint64(b[2])<<40 | uint64(b[3])<<32 | uint64(b[4])<<24 | uint64(b[5])<<16 | uint64(b[6])<<8 | uint64(b[7])) + return time.Duration(float64(max) * (val / float64(1<<64))) } diff --git a/internal/core/services/gateway_retry_test.go b/internal/core/services/gateway_retry_test.go index a085ef325..35af7cc63 100644 --- a/internal/core/services/gateway_retry_test.go +++ b/internal/core/services/gateway_retry_test.go @@ -268,10 +268,13 @@ func TestRetryTransport_GivesUpOnConnectionErrorAfterMaxRetries(t *testing.T) { req, _ := http.NewRequest("GET", "/", nil) resp, err := transport.RoundTrip(req) //nolint:bodyclose - // When all attempts return errors, doRoundTrip returns nil, nil - require.NoError(t, err) + // When all attempts return connection errors, we retry until retries + // are exhausted then return the last error (which allows circuit breaker + // to properly count the failure) + require.Error(t, err, "expected error after exhausting retries") assert.Nil(t, resp) assert.Equal(t, 3, m.calls, "3 attempts: first + 2 retries") + assert.Contains(t, err.Error(), "connection refused") } func TestRetryTransport_SucceedsOnFirstAttempt(t *testing.T) { diff --git a/internal/core/services/gateway_test.go b/internal/core/services/gateway_test.go index 036625bbb..16e21c802 100644 --- a/internal/core/services/gateway_test.go +++ b/internal/core/services/gateway_test.go @@ -91,7 +91,7 @@ func TestGatewayServiceGetProxy(t *testing.T) { _, _ = svc.CreateRoute(ctx, ports.CreateRouteParams{Name: "api", Pattern: "/api", Target: "http://localhost:8080"}) - proxy, paramsMap, ok := svc.GetProxy("GET", "/api/users") + proxy, _, paramsMap, ok := svc.GetProxy("GET", "/api/users") assert.True(t, ok) assert.NotNil(t, proxy) assert.Nil(t, paramsMap) @@ -102,7 +102,7 @@ func TestGatewayServiceGetProxyPattern(t *testing.T) { _, _ = svc.CreateRoute(ctx, ports.CreateRouteParams{Name: "users", Pattern: "/users/{id}", Target: "http://localhost:8080"}) - proxy, paramsMap, ok := svc.GetProxy("GET", "/users/123") + proxy, _, paramsMap, ok := svc.GetProxy("GET", "/users/123") assert.True(t, ok) assert.NotNil(t, proxy) assert.Equal(t, "123", paramsMap["id"]) diff --git a/internal/handlers/gateway_handler.go b/internal/handlers/gateway_handler.go index da22bdb5a..8a926edbd 100644 --- a/internal/handlers/gateway_handler.go +++ b/internal/handlers/gateway_handler.go @@ -2,6 +2,8 @@ package httphandlers import ( + "bufio" + "compress/gzip" "crypto/rand" "encoding/hex" "fmt" @@ -9,13 +11,16 @@ import ( "log/slog" "net" "net/http" + "strconv" "strings" + "time" "github.com/gin-gonic/gin" "github.com/google/uuid" "github.com/poyrazk/thecloud/internal/core/domain" "github.com/poyrazk/thecloud/internal/core/ports" "github.com/poyrazk/thecloud/internal/errors" + "github.com/poyrazk/thecloud/internal/platform" "github.com/poyrazk/thecloud/pkg/httputil" "github.com/poyrazk/thecloud/pkg/ratelimit" "golang.org/x/time/rate" @@ -42,6 +47,19 @@ type CreateRouteRequest struct { CircuitBreakerTimeout int64 `json:"circuit_breaker_timeout" binding:"gte=0"` MaxRetries int `json:"max_retries" binding:"gte=0"` RetryTimeout int64 `json:"retry_timeout" binding:"gte=0"` + AllowedOrigins []string `json:"allowed_origins"` + AllowedMethods []string `json:"allowed_methods"` + AllowedHeaders []string `json:"allowed_headers"` + ExposeHeaders []string `json:"expose_headers"` + MaxAge int `json:"max_age"` + StripResponseHeaders []string `json:"strip_response_headers"` + Compression string `json:"compression"` + JWTIssuer string `json:"jwt_issuer"` + JWTJwksURL string `json:"jwt_jwks_url"` + JWTAudience string `json:"jwt_audience"` + ClientCert string `json:"client_cert"` + ClientKey string `json:"client_key"` + CACert string `json:"ca_cert"` } // GatewayHandler handles API gateway HTTP endpoints. @@ -86,6 +104,18 @@ func (h *GatewayHandler) CreateRoute(c *gin.Context) { return } + // Validate mTLS certificate pairing + if (req.ClientCert != "") != (req.ClientKey != "") { + httputil.Error(c, errors.New(errors.InvalidInput, "both client_cert and client_key must be provided together")) + return + } + + // Dry-run mode: validate without persisting + if c.Request.URL.Query().Get("dry_run") == "true" { + h.validateDryRun(c, req) + return + } + params := ports.CreateRouteParams{ Name: req.Name, Pattern: req.PathPrefix, @@ -106,6 +136,19 @@ func (h *GatewayHandler) CreateRoute(c *gin.Context) { CircuitBreakerTimeout: req.CircuitBreakerTimeout, MaxRetries: req.MaxRetries, RetryTimeout: req.RetryTimeout, + AllowedOrigins: req.AllowedOrigins, + AllowedMethods: req.AllowedMethods, + AllowedHeaders: req.AllowedHeaders, + ExposeHeaders: req.ExposeHeaders, + MaxAge: req.MaxAge, + StripResponseHeaders: req.StripResponseHeaders, + Compression: req.Compression, + JWTIssuer: req.JWTIssuer, + JWTJwksURL: req.JWTJwksURL, + JWTAudience: req.JWTAudience, + ClientCert: req.ClientCert, + ClientKey: req.ClientKey, + CACert: req.CACert, } route, err := h.svc.CreateRoute(c.Request.Context(), params) @@ -161,6 +204,80 @@ func (h *GatewayHandler) DeleteRoute(c *gin.Context) { httputil.Success(c, http.StatusOK, gin.H{"message": "Route deleted"}) } +// handleJWT validates JWT if configured and injects claims into upstream request. +// Returns true if request should be aborted (auth failed). +func (h *GatewayHandler) handleJWT(c *gin.Context, route *domain.GatewayRoute) bool { + if route == nil || route.JWTJwksURL == "" { + return false + } + authHeader := c.GetHeader("Authorization") + if authHeader == "" || !strings.HasPrefix(authHeader, "Bearer ") { + c.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"error": "missing authorization header"}) + return true + } + tokenString := strings.TrimPrefix(authHeader, "Bearer ") + if tokenString == "" { + c.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"error": "missing token"}) + return true + } + claims, err := h.svc.ValidateJWT(c.Request.Context(), route, tokenString) + if err != nil { + c.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"error": "invalid token"}) + return true + } + for k, v := range claims { + c.Request.Header.Set("X-JWT-Claim-"+k, v) + } + return false +} + +// handleBodySizeLimit applies request body size limit for the route. +// For non-chunked bodies, rejects oversized requests early. +// For chunked bodies, wraps body with limitedReader. +// Returns true if request should be aborted (body too large). +func (h *GatewayHandler) handleBodySizeLimit(c *gin.Context, route *domain.GatewayRoute) bool { + if route == nil || route.MaxBodySize <= 0 { + return false + } + if c.Request.ContentLength > route.MaxBodySize { + httputil.Error(c, errors.New(errors.ObjectTooLarge, "request body too large")) + return true + } + if c.Request.ContentLength < 0 { + c.Request.Body = &limitedReader{ + ReadCloser: c.Request.Body, + limit: route.MaxBodySize, + } + } + return false +} + +// handleRateLimit applies per-route rate limiting. Returns true if request was rate-limited. +func (h *GatewayHandler) handleRateLimit(c *gin.Context, route *domain.GatewayRoute) bool { + if route == nil || route.RateLimit <= 0 || h.rateLimiter == nil { + return false + } + key := c.GetHeader("X-API-Key") + if key == "" { + key = c.ClientIP() + } else if len(key) > 5 { + key = "apikey:" + key[:5] + } + burst := route.RateLimit * 2 + limiter := h.rateLimiter.GetRouteLimiter(route.ID, key, rate.Limit(route.RateLimit), burst) + if !limiter.Allow() { + if h.logger != nil { + h.logger.Warn("per-route rate limit exceeded", + slog.String("key", key), + slog.String("path", c.Request.URL.Path), + slog.String("route_id", route.ID.String())) + } + c.AbortWithStatusJSON(http.StatusTooManyRequests, gin.H{"error": "rate limit exceeded"}) + return true + } + return false +} + func (h *GatewayHandler) Proxy(c *gin.Context) { path := c.Param("proxy") // Expecting routes like /gw/*proxy if !strings.HasPrefix(path, "/") { @@ -178,19 +295,15 @@ func (h *GatewayHandler) Proxy(c *gin.Context) { return } - // Apply request size limit - reject oversized requests before proxying - if route != nil && route.MaxBodySize > 0 { - if c.Request.ContentLength > route.MaxBodySize { - httputil.Error(c, errors.New(errors.ObjectTooLarge, "request body too large")) - return - } - // For chunked bodies, pre-read and enforce limit - if c.Request.ContentLength < 0 { - c.Request.Body = &limitedReader{ - ReadCloser: c.Request.Body, - limit: route.MaxBodySize, - } - } + // JWT validation if configured + if h.handleJWT(c, route) { + return + } + + // Apply request size limit + h.handleBodySizeLimit(c, route) + if c.Writer.Written() { + return } // Inject parameters into request context for downstream services if needed @@ -203,31 +316,142 @@ func (h *GatewayHandler) Proxy(c *gin.Context) { // Inject trace headers h.injectTraceHeaders(c) - // Apply per-route rate limiting if configured - if route != nil && route.RateLimit > 0 && h.rateLimiter != nil { - key := c.GetHeader("X-API-Key") - if key == "" { - key = c.ClientIP() - } else if len(key) > 5 { - key = "apikey:" + key[:5] + // Apply per-route rate limiting + if h.handleRateLimit(c, route) { + return + } + + // Record upstream latency metric + routeID := "" + if route != nil { + routeID = route.ID.String() + } + upstreamStart := time.Now() + + // Wrap response writer to capture status code for CORS headers and optionally compress + var wrapper http.ResponseWriter = &responseWrapper{ResponseWriter: c.Writer, status: http.StatusOK} + var needsClose bool + + // Enable compression if configured and client accepts it + if route != nil && route.Compression != "" && strings.Contains(c.GetHeader("Accept-Encoding"), route.Compression) { + wrapper = newCompressWriter(c.Writer, route.Compression) + needsClose = true + } + + proxy.ServeHTTP(wrapper, c.Request) + + // Flush gzip writer if compression was enabled + if needsClose { + if cw, ok := wrapper.(*compressWriter); ok { + cw.Close() } - burst := route.RateLimit * 2 - limiter := h.rateLimiter.GetRouteLimiter(route.ID, key, rate.Limit(route.RateLimit), burst) - if !limiter.Allow() { - if h.logger != nil { - h.logger.Warn("per-route rate limit exceeded", - slog.String("key", key), - slog.String("path", c.Request.URL.Path), - slog.String("route_id", route.ID.String())) - } - c.AbortWithStatusJSON(http.StatusTooManyRequests, gin.H{ - "error": "rate limit exceeded", - }) - return + } + + // Apply route-level CORS headers if configured + if route != nil && len(route.AllowedOrigins) > 0 { + h.injectCORSHeaders(c, route) + } + + // Strip configured response headers + if route != nil && len(route.StripResponseHeaders) > 0 { + for _, h := range route.StripResponseHeaders { + c.Header(h, "") + } + } + + platform.GatewayUpstreamLatency.WithLabelValues(routeID, c.Request.Method, path).Observe(time.Since(upstreamStart).Seconds()) +} + +type responseWrapper struct { + http.ResponseWriter + status int +} + +func (rw *responseWrapper) WriteHeader(status int) { + rw.status = status + rw.ResponseWriter.WriteHeader(status) +} + +// compressWriter wraps http.ResponseWriter to provide gzip compression. +type compressWriter struct { + http.ResponseWriter + status int + compression string + gz *gzip.Writer +} + +func newCompressWriter(w http.ResponseWriter, compression string) *compressWriter { + return &compressWriter{ + ResponseWriter: w, + status: http.StatusOK, + compression: compression, + } +} + +func (cw *compressWriter) WriteHeader(status int) { + cw.Header().Set("Content-Encoding", cw.compression) + cw.ResponseWriter.WriteHeader(status) +} + +func (cw *compressWriter) Write(p []byte) (int, error) { + if cw.gz == nil { + cw.gz = gzip.NewWriter(cw.ResponseWriter) + } + return cw.gz.Write(p) +} + +func (cw *compressWriter) Close() error { + if cw.gz != nil { + _ = cw.gz.Close() + cw.gz = nil + } + return nil +} + +// Hijack implements http.Hijacker to support websocket and streaming scenarios. +func (cw *compressWriter) Hijack() (net.Conn, *bufio.ReadWriter, error) { + if hj, ok := cw.ResponseWriter.(http.Hijacker); ok { + return hj.Hijack() + } + return nil, nil, fmt.Errorf("embedded ResponseWriter does not implement http.Hijacker") +} + +func (h *GatewayHandler) validateDryRun(c *gin.Context, req CreateRouteRequest) { + var validationErrors []string + + // Pattern validation + if strings.Contains(req.PathPrefix, "{") || strings.Contains(req.PathPrefix, "*") { + // Validate using routing pattern compilation + _ = req.PathPrefix // pattern validation placeholder + } + + // CIDR validation + for _, cidr := range req.AllowedCIDRs { + if _, _, err := net.ParseCIDR(cidr); err != nil { + validationErrors = append(validationErrors, fmt.Sprintf("invalid allowed CIDR %q", cidr)) + } + } + for _, cidr := range req.BlockedCIDRs { + if _, _, err := net.ParseCIDR(cidr); err != nil { + validationErrors = append(validationErrors, fmt.Sprintf("invalid blocked CIDR %q", cidr)) } } - proxy.ServeHTTP(c.Writer, c.Request) + // TLS conflict + if req.RequireTLS && req.TLSSkipVerify { + validationErrors = append(validationErrors, "cannot set both require_tls and tls_skip_verify") + } + + // mTLS pairing + if (req.ClientCert != "") != (req.ClientKey != "") { + validationErrors = append(validationErrors, "both client_cert and client_key must be provided together") + } + + if len(validationErrors) > 0 { + httputil.Success(c, http.StatusOK, gin.H{"dry_run": true, "valid": false, "errors": validationErrors}) + return + } + httputil.Success(c, http.StatusOK, gin.H{"dry_run": true, "valid": true, "message": "Route configuration is valid"}) } func (h *GatewayHandler) injectTraceHeaders(c *gin.Context) { @@ -310,6 +534,42 @@ func (h *GatewayHandler) checkCIDR(c *gin.Context, route *domain.GatewayRoute) b return true } +func (h *GatewayHandler) injectCORSHeaders(c *gin.Context, route *domain.GatewayRoute) { + origin := c.GetHeader("Origin") + if origin == "" { + return + } + + // Check if origin is allowed + allowed := false + for _, o := range route.AllowedOrigins { + if o == "*" || o == origin { + allowed = true + break + } + } + if !allowed { + return + } + + c.Header("Access-Control-Allow-Origin", origin) + if origin != "*" { + c.Header("Access-Control-Allow-Credentials", "true") + } + if len(route.AllowedMethods) > 0 { + c.Header("Access-Control-Allow-Methods", strings.Join(route.AllowedMethods, ", ")) + } + if len(route.AllowedHeaders) > 0 { + c.Header("Access-Control-Allow-Headers", strings.Join(route.AllowedHeaders, ", ")) + } + if len(route.ExposeHeaders) > 0 { + c.Header("Access-Control-Expose-Headers", strings.Join(route.ExposeHeaders, ", ")) + } + if route.MaxAge > 0 { + c.Header("Access-Control-Max-Age", strconv.Itoa(route.MaxAge)) + } +} + // limitedReader wraps an io.ReadCloser and enforces a byte limit. type limitedReader struct { io.ReadCloser diff --git a/internal/handlers/gateway_handler_test.go b/internal/handlers/gateway_handler_test.go index 9a3098ce8..a0092bed0 100644 --- a/internal/handlers/gateway_handler_test.go +++ b/internal/handlers/gateway_handler_test.go @@ -4,10 +4,13 @@ import ( "bytes" "context" "encoding/json" + "fmt" + "net" "net/http" "net/http/httptest" "net/http/httputil" "net/url" + "strings" "testing" "github.com/gin-gonic/gin" @@ -15,7 +18,6 @@ import ( "github.com/poyrazk/thecloud/internal/core/domain" "github.com/poyrazk/thecloud/internal/core/ports" "github.com/poyrazk/thecloud/internal/errors" - "github.com/poyrazk/thecloud/pkg/ratelimit" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/mock" "github.com/stretchr/testify/require" @@ -86,11 +88,19 @@ func (m *mockGatewayService) RefreshRoutes(ctx context.Context) error { return args.Error(0) } +func (m *mockGatewayService) ValidateJWT(ctx context.Context, route *domain.GatewayRoute, tokenString string) (map[string]string, error) { + args := m.Called(ctx, route, tokenString) + if args.Get(0) == nil { + return nil, args.Error(1) + } + r0, _ := args.Get(0).(map[string]string) + return r0, args.Error(1) +} + func setupGatewayHandlerTest(_ *testing.T) (*mockGatewayService, *GatewayHandler, *gin.Engine) { gin.SetMode(gin.TestMode) svc := new(mockGatewayService) - rateLimiter := ratelimit.NewIPRateLimiter(100, 200, nil) - handler := NewGatewayHandler(svc, rateLimiter, nil) + handler := NewGatewayHandler(svc, nil, nil) r := gin.New() return svc, handler, r } @@ -258,6 +268,192 @@ func TestGatewayHandlerProxyWithSlash(t *testing.T) { assert.Equal(t, http.StatusOK, w.Code) } +func TestGatewayHandlerProxyJWTEmptyToken(t *testing.T) { + gin.SetMode(gin.TestMode) + mockSvc := new(mockGatewayService) + handler := NewGatewayHandler(mockSvc, nil, nil) + r := gin.New() + r.Any(gwProxyPath, handler.Proxy) + + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, wreq *http.Request) { + w.WriteHeader(http.StatusOK) + })) + defer ts.Close() + targetURL, _ := url.Parse(ts.URL) + proxy := httputil.NewSingleHostReverseProxy(targetURL) + + route := &domain.GatewayRoute{ + ID: uuid.New(), + Name: "jwt-test", + JWTJwksURL: "https://auth.example.com/.well-known/jwks.json", + AllowedIPNets: []*net.IPNet{}, + } + mockSvc.On("GetProxy", "GET", "/api").Return(proxy, route, map[string]string{}, true).Once() + + // Request without Authorization header + req, err := http.NewRequest("GET", gwAPITestPath, nil) + require.NoError(t, err) + req.RemoteAddr = "10.0.0.1:12345" + w := httptest.NewRecorder() + r.ServeHTTP(w, req) + + assert.Equal(t, http.StatusUnauthorized, w.Code) + mockSvc.AssertExpectations(t) +} + +func TestGatewayHandlerProxyJWTMissingBearer(t *testing.T) { + gin.SetMode(gin.TestMode) + mockSvc := new(mockGatewayService) + handler := NewGatewayHandler(mockSvc, nil, nil) + r := gin.New() + r.Any(gwProxyPath, handler.Proxy) + + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, wreq *http.Request) { + w.WriteHeader(http.StatusOK) + })) + defer ts.Close() + targetURL, _ := url.Parse(ts.URL) + proxy := httputil.NewSingleHostReverseProxy(targetURL) + + route := &domain.GatewayRoute{ + ID: uuid.New(), + Name: "jwt-test", + JWTJwksURL: "https://auth.example.com/.well-known/jwks.json", + AllowedIPNets: []*net.IPNet{}, + } + mockSvc.On("GetProxy", "GET", "/api").Return(proxy, route, map[string]string{}, true).Once() + + // Request with Authorization header but no Bearer prefix + req, err := http.NewRequest("GET", gwAPITestPath, nil) + require.NoError(t, err) + req.Header.Set("Authorization", "Basic dXNlcm5hbWU6cGFzc3dvcmQ=") + req.RemoteAddr = "10.0.0.1:12345" + w := httptest.NewRecorder() + r.ServeHTTP(w, req) + + assert.Equal(t, http.StatusUnauthorized, w.Code) + mockSvc.AssertExpectations(t) +} + +func TestGatewayHandlerProxyJWTValidToken(t *testing.T) { + gin.SetMode(gin.TestMode) + mockSvc := new(mockGatewayService) + handler := NewGatewayHandler(mockSvc, nil, nil) + r := gin.New() + r.Any(gwProxyPath, handler.Proxy) + + // Use a capturing proxy that records upstream headers + var upstreamReqHeaders http.Header + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, wreq *http.Request) { + upstreamReqHeaders = wreq.Header.Clone() + w.WriteHeader(http.StatusOK) + })) + defer ts.Close() + targetURL, _ := url.Parse(ts.URL) + proxy := httputil.NewSingleHostReverseProxy(targetURL) + + route := &domain.GatewayRoute{ + ID: uuid.New(), + Name: "jwt-test", + JWTJwksURL: "https://auth.example.com/.well-known/jwks.json", + AllowedIPNets: []*net.IPNet{}, + } + mockSvc.On("GetProxy", "GET", "/api").Return(proxy, route, map[string]string{}, true).Once() + mockSvc.On("ValidateJWT", mock.Anything, route, "valid-token").Return( + map[string]string{"sub": "user123", "iss": "test-issuer"}, nil).Once() + + req, err := http.NewRequest("GET", gwAPITestPath, nil) + require.NoError(t, err) + req.Header.Set("Authorization", "Bearer valid-token") + req.RemoteAddr = "10.0.0.1:12345" + w := &closeNotifierRecorder{httptest.NewRecorder()} + r.ServeHTTP(w, req) + + assert.Equal(t, http.StatusOK, w.Code) + assert.Equal(t, "user123", upstreamReqHeaders.Get("X-JWT-Claim-sub")) + assert.Equal(t, "test-issuer", upstreamReqHeaders.Get("X-JWT-Claim-iss")) + mockSvc.AssertExpectations(t) +} + +func TestGatewayHandlerProxyJWTInvalidTokenServiceError(t *testing.T) { + gin.SetMode(gin.TestMode) + mockSvc := new(mockGatewayService) + handler := NewGatewayHandler(mockSvc, nil, nil) + r := gin.New() + r.Any(gwProxyPath, handler.Proxy) + + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, wreq *http.Request) { + w.WriteHeader(http.StatusOK) + })) + defer ts.Close() + targetURL, _ := url.Parse(ts.URL) + proxy := httputil.NewSingleHostReverseProxy(targetURL) + + route := &domain.GatewayRoute{ + ID: uuid.New(), + Name: "jwt-test", + JWTJwksURL: "https://auth.example.com/.well-known/jwks.json", + AllowedIPNets: []*net.IPNet{}, + } + mockSvc.On("GetProxy", "GET", "/api").Return(proxy, route, map[string]string{}, true).Once() + mockSvc.On("ValidateJWT", mock.Anything, route, "malformed-token").Return( + nil, fmt.Errorf("invalid token")).Once() + + req, err := http.NewRequest("GET", gwAPITestPath, nil) + require.NoError(t, err) + req.Header.Set("Authorization", "Bearer malformed-token") + req.RemoteAddr = "10.0.0.1:12345" + w := &closeNotifierRecorder{httptest.NewRecorder()} + r.ServeHTTP(w, req) + + assert.Equal(t, http.StatusUnauthorized, w.Code) + mockSvc.AssertExpectations(t) +} + +func TestGatewayHandlerProxyJWTClaimsPropagation(t *testing.T) { + gin.SetMode(gin.TestMode) + mockSvc := new(mockGatewayService) + handler := NewGatewayHandler(mockSvc, nil, nil) + r := gin.New() + r.Any(gwProxyPath, handler.Proxy) + + var upstreamReqHeaders http.Header + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, wreq *http.Request) { + upstreamReqHeaders = wreq.Header.Clone() + w.WriteHeader(http.StatusOK) + })) + defer ts.Close() + targetURL, _ := url.Parse(ts.URL) + proxy := httputil.NewSingleHostReverseProxy(targetURL) + + route := &domain.GatewayRoute{ + ID: uuid.New(), + Name: "jwt-test", + JWTJwksURL: "https://auth.example.com/.well-known/jwks.json", + AllowedIPNets: []*net.IPNet{}, + } + mockSvc.On("GetProxy", "GET", "/api").Return(proxy, route, map[string]string{}, true).Once() + mockSvc.On("ValidateJWT", mock.Anything, route, "multi-claim-token").Return( + map[string]string{ + "sub": "user1", + "role": "admin", + "email": "u@example.com", + }, nil).Once() + + req, err := http.NewRequest("GET", gwAPITestPath, nil) + require.NoError(t, err) + req.Header.Set("Authorization", "Bearer multi-claim-token") + req.RemoteAddr = "10.0.0.1:12345" + w := &closeNotifierRecorder{httptest.NewRecorder()} + r.ServeHTTP(w, req) + + assert.Equal(t, http.StatusOK, w.Code) + assert.Equal(t, "user1", upstreamReqHeaders.Get("X-JWT-Claim-sub")) + assert.Equal(t, "admin", upstreamReqHeaders.Get("X-JWT-Claim-role")) + assert.Equal(t, "u@example.com", upstreamReqHeaders.Get("X-JWT-Claim-email")) + mockSvc.AssertExpectations(t) +} + func TestGatewayHandlerCreateError(t *testing.T) { t.Parallel() t.Run("InvalidJSON", func(t *testing.T) { @@ -352,11 +548,277 @@ func TestGatewayHandlerProxyParamWithoutSlash(t *testing.T) { mockSvc.AssertExpectations(t) } -func TestGatewayHandlerProxyRateLimitExceeded(t *testing.T) { - // TODO: Test requires mock-friendly rate limiter to be deterministic. - // The real limiter refills tokens over time (rate=10 β†’ 1 token/100ms), - // so by the time the proxied request runs, tokens may have refilled. - // Will be addressed when a mock-able RateLimiter interface is introduced. +func TestGatewayHandlerInjectCORSHeaders(t *testing.T) { + gin.SetMode(gin.TestMode) + svc := new(mockGatewayService) + handler := NewGatewayHandler(svc, nil, nil) + r := gin.New() + r.Any(gwProxyPath, handler.Proxy) + + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, wreq *http.Request) { + w.WriteHeader(http.StatusOK) + })) + defer ts.Close() + targetURL, _ := url.Parse(ts.URL) + proxy := httputil.NewSingleHostReverseProxy(targetURL) + + tests := []struct { + name string + origin string + allowedOrigins []string + allowedMethods []string + allowedHeaders []string + exposeHeaders []string + maxAge int + expectCORS bool + expectMethods bool + expectAllowMethods string + }{ + { + name: "wildcard origin - allowed", + origin: "http://example.com", + allowedOrigins: []string{"*"}, + expectCORS: true, + }, + { + name: "exact origin match", + origin: "http://example.com", + allowedOrigins: []string{"http://example.com", "http://test.com"}, + expectCORS: true, + }, + { + name: "origin not in allowlist", + origin: "http://evil.com", + allowedOrigins: []string{"http://example.com"}, + expectCORS: false, + }, + { + name: "with methods and headers", + origin: "http://example.com", + allowedOrigins: []string{"http://example.com"}, + allowedMethods: []string{"GET", "POST"}, + allowedHeaders: []string{"Authorization", "Content-Type"}, + exposeHeaders: []string{"X-Custom-Header"}, + maxAge: 3600, + expectCORS: true, + expectMethods: true, + expectAllowMethods: "GET, POST", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + route := &domain.GatewayRoute{ + ID: uuid.New(), + Name: "cors-test", + Compression: "", + AllowedOrigins: tc.allowedOrigins, + AllowedMethods: tc.allowedMethods, + AllowedHeaders: tc.allowedHeaders, + ExposeHeaders: tc.exposeHeaders, + MaxAge: tc.maxAge, + AllowedIPNets: []*net.IPNet{}, + } + svc.On("GetProxy", "GET", "/api").Return(proxy, route, map[string]string{}, true).Once() + + req, err := http.NewRequest("GET", gwAPITestPath, nil) + require.NoError(t, err) + req.Header.Set("Origin", tc.origin) + req.RemoteAddr = "10.0.0.1:12345" + w := &closeNotifierRecorder{httptest.NewRecorder()} + r.ServeHTTP(w, req) + + assert.Equal(t, http.StatusOK, w.Code) + if tc.expectCORS { + assert.NotEmpty(t, w.Header().Get("Access-Control-Allow-Origin")) + if tc.expectMethods { + assert.Equal(t, tc.expectAllowMethods, w.Header().Get("Access-Control-Allow-Methods")) + assert.Contains(t, w.Header().Get("Access-Control-Allow-Headers"), "Authorization") + assert.Equal(t, "X-Custom-Header", w.Header().Get("Access-Control-Expose-Headers")) + assert.Equal(t, "3600", w.Header().Get("Access-Control-Max-Age")) + } + } else { + assert.Empty(t, w.Header().Get("Access-Control-Allow-Origin")) + } + svc.AssertExpectations(t) + }) + } +} + +func TestGatewayHandlerInjectCORSHeadersPreflight(t *testing.T) { + gin.SetMode(gin.TestMode) + svc := new(mockGatewayService) + handler := NewGatewayHandler(svc, nil, nil) + r := gin.New() + r.Any(gwProxyPath, handler.Proxy) + + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, wreq *http.Request) { + w.WriteHeader(http.StatusOK) + })) + defer ts.Close() + targetURL, _ := url.Parse(ts.URL) + proxy := httputil.NewSingleHostReverseProxy(targetURL) + + route := &domain.GatewayRoute{ + ID: uuid.New(), + Name: "cors-preflight", + AllowedOrigins: []string{"http://example.com"}, + AllowedMethods: []string{"GET", "POST", "OPTIONS"}, + AllowedHeaders: []string{"Authorization", "Content-Type"}, + ExposeHeaders: []string{"X-Custom-Header"}, + MaxAge: 3600, + AllowedIPNets: []*net.IPNet{}, + } + svc.On("GetProxy", "OPTIONS", "/api").Return(proxy, route, map[string]string{}, true).Once() + + req, err := http.NewRequest("OPTIONS", gwAPITestPath, nil) + require.NoError(t, err) + req.Header.Set("Origin", "http://example.com") + req.Header.Set("Access-Control-Request-Method", "POST") + req.Header.Set("Access-Control-Request-Headers", "Authorization, Content-Type") + req.RemoteAddr = "10.0.0.1:12345" + w := &closeNotifierRecorder{httptest.NewRecorder()} + r.ServeHTTP(w, req) + + assert.Equal(t, http.StatusOK, w.Code) + assert.Equal(t, "http://example.com", w.Header().Get("Access-Control-Allow-Origin")) + assert.Equal(t, "true", w.Header().Get("Access-Control-Allow-Credentials")) + assert.Equal(t, "GET, POST, OPTIONS", w.Header().Get("Access-Control-Allow-Methods")) + assert.Contains(t, w.Header().Get("Access-Control-Allow-Headers"), "Authorization") + assert.Contains(t, w.Header().Get("Access-Control-Allow-Headers"), "Content-Type") + assert.Equal(t, "X-Custom-Header", w.Header().Get("Access-Control-Expose-Headers")) + assert.Equal(t, "3600", w.Header().Get("Access-Control-Max-Age")) + svc.AssertExpectations(t) +} + +func TestGatewayHandlerProxyCompression(t *testing.T) { + gin.SetMode(gin.TestMode) + svc := new(mockGatewayService) + handler := NewGatewayHandler(svc, nil, nil) + r := gin.New() + r.Any(gwProxyPath, handler.Proxy) + + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, wreq *http.Request) { + w.WriteHeader(http.StatusOK) + _, _ = w.Write([]byte("proxied response")) + })) + defer ts.Close() + targetURL, _ := url.Parse(ts.URL) + proxy := httputil.NewSingleHostReverseProxy(targetURL) + + tests := []struct { + name string + compression string + acceptEncoding string + expectEncoding string + }{ + { + name: "gzip compression enabled", + compression: "gzip", + acceptEncoding: "gzip", + expectEncoding: "gzip", + }, + { + name: "no compression - route disabled", + compression: "", + acceptEncoding: "gzip", + expectEncoding: "", + }, + { + name: "client does not accept gzip", + compression: "gzip", + acceptEncoding: "", + expectEncoding: "", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + route := &domain.GatewayRoute{ + ID: uuid.New(), + Name: "compression-test", + Compression: tc.compression, + AllowedIPNets: []*net.IPNet{}, + } + svc.On("GetProxy", "GET", "/api").Return(proxy, route, map[string]string{}, true).Once() + + req, err := http.NewRequest("GET", gwAPITestPath, nil) + require.NoError(t, err) + req.Header.Set("Accept-Encoding", tc.acceptEncoding) + req.RemoteAddr = "10.0.0.1:12345" + w := &closeNotifierRecorder{httptest.NewRecorder()} + r.ServeHTTP(w, req) + + assert.Equal(t, http.StatusOK, w.Code) + assert.Equal(t, tc.expectEncoding, w.Header().Get("Content-Encoding")) + svc.AssertExpectations(t) + }) + } +} + +func TestGatewayHandlerInjectTraceHeadersWithInbound(t *testing.T) { + t.Parallel() + svc := new(mockGatewayService) + handler := NewGatewayHandler(svc, nil, nil) + r := gin.New() + r.Any(gwProxyPath, handler.Proxy) + + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, wreq *http.Request) { + w.WriteHeader(http.StatusOK) + })) + defer ts.Close() + targetURL, _ := url.Parse(ts.URL) + + route := &domain.GatewayRoute{ID: uuid.New(), AllowedIPNets: []*net.IPNet{}} + svc.On("GetProxy", "GET", "/api").Return( + httputil.NewSingleHostReverseProxy(targetURL), route, map[string]string{}, true).Once() + + req, _ := http.NewRequest("GET", gwAPITestPath, nil) + req.Header.Set("traceparent", "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01") + req.Header.Set("tracestate", "congo=t61rcWkgMzE") + req.RemoteAddr = "10.0.0.1:12345" + w := &closeNotifierRecorder{httptest.NewRecorder()} + r.ServeHTTP(w, req) + + // Inbound traceparent should be preserved (not replaced with generated) + assert.Equal(t, "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01", w.Header().Get("traceparent")) + assert.Equal(t, "congo=t61rcWkgMzE", w.Header().Get("tracestate")) + svc.AssertExpectations(t) +} + +func TestGatewayHandlerProxyCompressionGzipFlushed(t *testing.T) { + t.Parallel() + svc := new(mockGatewayService) + handler := NewGatewayHandler(svc, nil, nil) + gin.SetMode(gin.TestMode) + r := gin.New() + r.Any(gwProxyPath, handler.Proxy) + + var gzipFlushed bool + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, wreq *http.Request) { + // Read body to confirm gzip was flushed + body := make([]byte, 100) + n, _ := wreq.Body.Read(body) + gzipFlushed = n > 0 + w.WriteHeader(http.StatusOK) + })) + defer ts.Close() + targetURL, _ := url.Parse(ts.URL) + + route := &domain.GatewayRoute{ID: uuid.New(), Compression: "gzip", AllowedIPNets: []*net.IPNet{}} + svc.On("GetProxy", "GET", "/api").Return( + httputil.NewSingleHostReverseProxy(targetURL), route, map[string]string{}, true).Once() + + req, _ := http.NewRequest("GET", gwAPITestPath, strings.NewReader("hello world this is some test data")) + req.Header.Set("Accept-Encoding", "gzip") + req.RemoteAddr = "10.0.0.1:12345" + w := &closeNotifierRecorder{httptest.NewRecorder()} + r.ServeHTTP(w, req) + + assert.Equal(t, http.StatusOK, w.Code) + assert.Equal(t, "gzip", w.Header().Get("Content-Encoding")) + assert.True(t, gzipFlushed, "gzip data should be flushed to upstream") + svc.AssertExpectations(t) } func TestGatewayHandlerDeleteError(t *testing.T) { @@ -382,3 +844,74 @@ func TestGatewayHandlerDeleteError(t *testing.T) { svc.AssertExpectations(t) }) } + +func TestGatewayHandlerCreateRouteDryRun(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + body map[string]interface{} + wantValid bool + wantErrCount int + errContains []string + }{ + { + name: "valid CIDR - passes", + body: map[string]interface{}{ + "name": "dry-run-test", + "path_prefix": "/api/v1", + "target_url": "http://example.com", + "allowed_cidrs": []string{"10.0.0.0/8"}, + }, + wantValid: true, + }, + { + name: "invalid CIDR - fails", + body: map[string]interface{}{ + "name": "dry-run-test", + "path_prefix": "/api/v1", + "target_url": "http://example.com", + "allowed_cidrs": []string{"not-a-cidr"}, + }, + wantValid: false, + errContains: []string{"invalid allowed CIDR"}, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + _, handler, r := setupGatewayHandlerTest(t) + r.POST(routesPath, handler.CreateRoute) + + body, err := json.Marshal(tc.body) + require.NoError(t, err) + req, err := http.NewRequest("POST", routesPath+"?dry_run=true", bytes.NewBuffer(body)) + require.NoError(t, err) + w := httptest.NewRecorder() + r.ServeHTTP(w, req) + + assert.Equal(t, http.StatusOK, w.Code) + var resp map[string]interface{} + err = json.Unmarshal(w.Body.Bytes(), &resp) + require.NoError(t, err) + data, ok := resp["data"].(map[string]interface{}) + require.True(t, ok, "response data should be a map") + assert.Equal(t, true, data["dry_run"]) + assert.Equal(t, tc.wantValid, data["valid"]) + if !tc.wantValid && len(tc.errContains) > 0 { + errs, ok := data["errors"].([]interface{}) + require.True(t, ok) + for _, exp := range tc.errContains { + found := false + for _, e := range errs { + if strings.Contains(e.(string), exp) { + found = true + break + } + } + assert.True(t, found, "expected error containing %q", exp) + } + } + }) + } +} diff --git a/internal/platform/circuit_breaker.go b/internal/platform/circuit_breaker.go index e710e885f..54c16cd92 100644 --- a/internal/platform/circuit_breaker.go +++ b/internal/platform/circuit_breaker.go @@ -237,6 +237,33 @@ func (cb *CircuitBreaker) Reset() { } } +// RecordFailure immediately records a failure and trips the circuit breaker +// if it is currently closed. This is used for fast-fail scenarios where +// we want to trip the CB immediately on specific errors. +func (cb *CircuitBreaker) RecordFailure() { + cb.mu.Lock() + var cbFunc StateChangeFunc + var name string + var from, to State + var changed bool + + cb.halfOpenInFlight = false + cb.failureCount++ + cb.lastFailure = time.Now() + + switch cb.state { + case StateClosed: + cbFunc, name, from, to, changed = cb.transitionLocked(StateOpen) + case StateHalfOpen: + cbFunc, name, from, to, changed = cb.transitionLocked(StateOpen) + } + cb.mu.Unlock() + + if changed && cbFunc != nil { + cbFunc(name, from, to) + } +} + // GetState returns the current state of the circuit breaker. func (cb *CircuitBreaker) GetState() State { cb.mu.Lock() diff --git a/internal/platform/metrics.go b/internal/platform/metrics.go index 039dfc670..860176636 100644 --- a/internal/platform/metrics.go +++ b/internal/platform/metrics.go @@ -48,6 +48,31 @@ var ( Buckets: prometheus.DefBuckets, }, []string{"method", "path"}) + // Gateway metrics + GatewayUpstreamLatency = promauto.NewHistogramVec(prometheus.HistogramOpts{ + Name: "thecloud_gateway_upstream_latency_seconds", + Help: "Duration of upstream (proxied) HTTP requests in seconds", + Buckets: prometheus.DefBuckets, + }, []string{"route_id", "method", "path"}) + GatewayRetryTotal = promauto.NewCounterVec(prometheus.CounterOpts{ + Name: "thecloud_gateway_retry_total", + Help: "Total number of gateway retry attempts", + }, []string{"route_id", "status"}) + GatewayCircuitBreakerState = promauto.NewGaugeVec(prometheus.GaugeOpts{ + Name: "thecloud_gateway_circuit_breaker_state", + Help: "State of gateway circuit breaker (0=closed, 1=open, 2=half-open)", + }, []string{"route_id"}) + + JWKSFetchTotal = promauto.NewCounterVec(prometheus.CounterOpts{ + Name: "thecloud_jwks_fetch_total", + Help: "Total number of JWKS fetch attempts", + }, []string{"status"}) // status: success, error, circuit_open + + JWKSBreakerState = promauto.NewGauge(prometheus.GaugeOpts{ + Name: "thecloud_jwks_breaker_state", + Help: "State of JWKS fetch circuit breaker (0=closed, 1=open, 2=half-open)", + }) + // Compute metrics InstancesTotal = promauto.NewGaugeVec(prometheus.GaugeOpts{ Name: "thecloud_instances_total",