feat(sandbox): introduce middleware layer for L7 proxy request transformations

[jhjaggars]

PR #1638 (SigV4 credential re-signing) adds AWS-specific fields (credential_signing, signing_service) threaded through the entire L7 proxy stack — policy YAML, proto, OPA data, L7EndpointConfig, RelayRequestOptions, and inline branching in relay_http_request_with_options_guarded. This works, but the approach doesn't scale: every future service-specific behavior (Azure token exchange, GCP IAM, rate limiting, custom header injection) would require adding more fields to the same structs and more if branches to the same relay function.
The review discussion on #1638 (comment) noted that a middleware pattern would fit more cleanly, aligning with the broader provider profile vision in #896.
A middleware is a named, configurable transform that intercepts an HTTP request between L7 policy evaluation and upstream forwarding. Middlewares operate on parsed HTTP requests — they see headers, body framing, and the resolved credential context, and can modify headers, buffer/re-sign bodies, or inject new headers before the request goes upstream.
Replace per-endpoint credential_signing and signing_service with a middleware list and a per-policy middleware_config map:

network_policies:
  aws_bedrock:
    middleware_config:
      sigv4:
        signing_service: bedrock
    endpoints:
      - host: bedrock-runtime.us-east-1.amazonaws.com
        port: 443
        protocol: rest
        access: full
        middleware:
          - sigv4

• middleware: ordered list of middleware names on the endpoint. Each name must resolve to an entry in middleware_config (or be a built-in needing no config). Validated at policy load time.
• middleware_config: per-policy map of middleware name to config object. Schema is middleware-specific; validated by the middleware's own validation function at load time.
  This keeps endpoint definitions clean — a single middleware: [sigv4] — while service-specific knobs live in a separate section that only grows when new middlewares are added.

// crates/openshell-sandbox/src/l7/middleware.rs
pub trait L7Middleware: Send + Sync + fmt::Debug {
    fn name(&self) -> &str;
    fn process_request<'a>(
        &'a self,
        ctx: &'a MiddlewareContext<'_>,
        req: &'a L7Request,
        headers: &'a [u8],
        client: &'a mut (dyn AsyncRead + Unpin + Send),
    ) -> Pin<Box<dyn Future<Output = Result<MiddlewareAction>> + Send + 'a>>;
}
pub enum MiddlewareAction {
    Forward { data: Vec<u8> },
    Passthrough,
}

In relay_http_request_with_options_guarded, replace the SigV4 inline block with a middleware chain:

for mw in &options.middleware {
    match mw.process_request(&mw_ctx, &req, &headers, client).await? {
        MiddlewareAction::Forward { data } => {
            upstream.write_all(&data).await?;
            forwarded = true;
            break;
        }
        MiddlewareAction::Passthrough => continue,
    }
}

Move existing SigV4 logic from the inline block in rest.rs into SigV4Middleware. The sigv4.rs utility module stays as-is — pure signing functions used by the middleware.

• L7EndpointConfig and RelayRequestOptions drop credential_signing/signing_service/host and gain middleware: Vec<Arc<dyn L7Middleware>>
• Middleware instances are constructed at policy load time in parse_l7_config
• Proto: repeated string middleware + middleware_config map on NetworkEndpoint
• Backward compat: credential_signing: sigv4 expands to middleware: [sigv4] during policy loading

1. Keep adding per-endpoint fields (current approach in feat(sandbox): proxy-side AWS SigV4 credential signing for CONNECT tunnels #1638) — works short-term but every new service-specific behavior adds fields and branches to the core relay path. The policy schema grows unboundedly.
2. Provider-level middleware only (tie to Enhanced Provider Management #896 provider profiles) — cleaner for providers that know their endpoints, but some middleware (rate limiting, custom headers) is per-endpoint, not per-provider. The endpoint-level middleware list handles both cases.
3. Tower-style middleware — more generic but heavier. The L7 proxy already has its own request/response lifecycle (parsed HTTP over raw streams, not hyper Request/Response objects). A simple trait matching the existing relay flow is more practical.
   Explored the L7 proxy architecture across these files:

• l7/mod.rs: L7EndpointConfig struct, parse_l7_config, CredentialSigning enum, policy validation. The PR adds credential_signing and signing_service fields here, parsed from OPA regorus values.
• l7/rest.rs: relay_http_request_with_options_guarded is the core relay function. The PR adds ~175 lines of inline SigV4 branching here (detect payload mode, handle Expect: 100-continue, buffer body, sign, forward). This is the function that would gain the middleware loop.
• l7/relay.rs: relay_rest and relay_with_route_selection thread RelayRequestOptions (including the new SigV4 fields) into the rest.rs relay function.
• l7/provider.rs: L7Provider trait — parse/relay/deny. Middleware sits between parse+evaluate and relay, not inside the provider.
• proxy.rs: CONNECT tunnel handling, TLS termination, routes to relay_with_inspection. Threads CredentialSigning::None through options in ~6 locations.
• sigv4.rs: Pure utility functions (extract region, strip AWS headers, apply signing). No policy coupling — stays as-is.
• openshell-policy/src/lib.rs: NetworkEndpointDef with serde, proto conversion, policy validation. PR adds credential_signing/signing_service string fields.
• proto/sandbox.proto: NetworkEndpoint message. PR adds fields 19-20.
  The PR review comments explicitly discuss the awkwardness of AWS-specific fields in the generic policy schema and reference Enhanced Provider Management #896 as the broader solution space.
• Depends on: feat(sandbox): proxy-side AWS SigV4 credential signing for CONNECT tunnels #1638 (SigV4 signing — provides the logic to be refactored into middleware)
• Related: Enhanced Provider Management #896 (Provider Profiles — would eventually declare middleware per provider type)
• Does NOT change: OPA/Rego evaluation, L7Provider trait, SecretResolver, sigv4.rs utilities
• L7Middleware trait defined with process_request returning MiddlewareAction
• SigV4Middleware implements the trait, moves logic from rest.rs inline block
• relay_http_request_with_options_guarded uses middleware chain instead of inline SigV4 branching
• Policy schema supports middleware list on endpoints and middleware_config map on policies
• Policy validation rejects unknown middleware names and missing config
• Backward compat: credential_signing: sigv4 policies expand to middleware form during loading
• Proto updated with middleware fields
• Existing SigV4 tests pass through the new middleware path
• mise run pre-commit and mise run test pass

[johntmyers]

Could middleware_config be its own block parallel with network_policies? If an endpoint can reference a named middleware then in the case where multiple endpoints need to use the same middleware, the mw config doesn't need to be redefined on each endpoint.

[jhjaggars]

Do you mean something like this?:

middleware_configs:
   sigv4_bedrock:
       middleware: sigv4
       signing_service: bedrock
network_policies:
  aws_bedrock:
    endpoints:
      - host: bedrock-runtime.us-east-1.amazonaws.com
        port: 443
        protocol: rest
        access: full
        middleware:
          - sigv4_bedrock

if so, then yes and I like that much more.