feat(sandbox): proxy-side AWS SigV4 credential signing for CONNECT tunnels by jhjaggars · Pull Request #1638 · NVIDIA/OpenShell

jhjaggars · 2026-05-29T19:43:11Z

Add proxy-side AWS SigV4 credential re-signing for CONNECT tunnel requests. The proxy auto-detects the correct signing mode from the client SDK's x-amz-content-sha256 header, supporting Bedrock (signed body), S3 uploads (streaming unsigned), and S3 reads/deletes (unsigned payload) — all with a single credential_signing: sigv4 policy setting.

Client `x-amz-content-sha256`	Proxy behavior	Services
Hex SHA-256 hash	Buffer body, hash, sign	Bedrock, most AWS
`STREAMING-UNSIGNED-PAYLOAD-TRAILER`	Sign headers, stream body	S3 PutObject, upload_fileobj
`STREAMING-AWS4-HMAC-SHA256-PAYLOAD`	Sign headers, stream body	S3 PutObject (plain HTTP)
`UNSIGNED-PAYLOAD`	Sign headers, no buffering	S3 over HTTPS
Any other `STREAMING-*`	Sign headers, stream body	Future SDK variants
Absent	Content-Length heuristic	Non-AWS SDKs
All `STREAMING-*` variants are re-signed as `STREAMING-UNSIGNED-PAYLOAD-TRAILER`. The proxy cannot reproduce per-chunk signatures, but AWS accepts unsigned streaming payloads over HTTPS. No AWS SDK sends per-chunk signed streaming over HTTPS by default.
Two explicit overrides: `sigv4:body` (always hash body) and `sigv4:no_body` (always unsigned).
Closes #1631
Related: #1576, #1568

sigv4.rs: SigV4 signing via aws-sigv4 — signed body and headers-only modes. Region extraction supports standard, dualstack, FIPS, FIPS+dualstack, virtual-hosted, GovCloud, and China partition hostnames. Rejects service names (e.g. s3) as regions via looks_like_region() heuristic.
l7/rest.rs: Auto-detects payload mode from client headers, branches into signed-body or streaming path. All STREAMING-* variants are handled as unsigned streaming. Handles Expect: 100-continue within SigV4 path only. Fails closed for chunked bodies in signed-body path.
l7/mod.rs: CredentialSigning enum (SigV4, SigV4Body, SigV4NoBody). Rejects unknown values and missing signing_service at sandbox startup.
Proto/Policy/OPA: credential_signing (field 19) and signing_service (field 20) on NetworkEndpoint, plumbed through serde, proto, Rego, and provider profile DTO.
Policy validation: Rejects unknown credential_signing values and missing signing_service at load time.
Provider profiles: EndpointProfile preserves both fields through round-trip.
OCSF: NetworkActivity event emitted for each SigV4 re-signing decision.
Architecture docs: Updated architecture/sandbox.md with all signing modes.
mise run pre-commit passes
Unit tests added/updated (region extraction, signing output, policy validation, L7 config parsing)
Integration test against real Bedrock and S3 (cargo test -p openshell-sandbox --test sigv4_real_aws -- --ignored)
End-to-end from inside Podman sandbox: Bedrock InvokeModel, S3 PUT/GET/DELETE, S3 streaming upload — all passing
Follows Conventional Commits
Commits are signed off (DCO)

copy-pr-bot · 2026-05-29T19:43:15Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-05-29T19:43:21Z

All contributors have signed the DCO ✍️ ✅
_{Posted by the DCO Assistant Lite bot.}

jhjaggars · 2026-05-29T19:47:42Z

I have read the DCO document and I hereby sign the DCO.

johntmyers · 2026-06-01T14:58:20Z

+    #[serde(default, skip_serializing_if = "String::is_empty")]
+    credential_signing: String,
+    #[serde(default, skip_serializing_if = "String::is_empty")]
+    signing_service: String,


Are there practical extensions to the values for these fields? Is there ever a non-AWS use case? Just trying to understand how AWS-specific these settings are vs the names of the settings that imply they could map to other service provider use cases.

These feel pretty specific to AWS. It's awkward, but I didn't see an obviously better way to have a special case like this today. credential_signing might be used in other services, but the signing_service almost certainly wouldn't.

I could imagine a cleaner way of expressing these things aligning with your issue here: #896. It does seem like a middleware pattern would fit nicely as well. I have opened an issue to propose a possible middleware system: #1694

russellb · 2026-06-01T20:23:31Z

just for visibility - I started working on AWS STS credential support (#1576), using access to S3 as a test case.

This depends on sigv4 support, so I pulled that in to my branch, or at least a version of it from the end of last week.

main...russellb:OpenShell:feat/1576-aws-sts-with-sigv4

…nnels Add proxy-side AWS SigV4 re-signing so sandbox clients can reach AWS services (Bedrock) through the CONNECT tunnel using placeholder credentials. The proxy strips the invalid signature, resolves real credentials from the SecretResolver, re-signs with the aws-sigv4 crate, and forwards. Configuration is policy-driven via two new fields (credential_signing, signing_service). Policy YAML example: credential_signing: sigv4 signing_service: bedrock Implementation: - sigv4.rs: strip_aws_headers removes old auth headers before the fail-closed placeholder scan; apply_sigv4_to_request re-signs using the aws-sigv4 SDK with PayloadChecksumKind::XAmzSha256 enabled. Returns Result instead of panicking. Non-signed headers (Accept, User-Agent, etc.) are preserved in the output. - rest.rs: SigV4 path buffers body (capped at MAX_REWRITE_BODY_BYTES) for signing, then forwards the re-signed request upstream. - Proto: credential_signing (field 19), signing_service (field 20) on NetworkEndpoint. - Policy/OPA: plumbed through serde, proto conversion, and Rego data. - Supports AWS session tokens (STS temporary credentials). - Integration test against real Bedrock (ignored, requires AWS creds). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Reject policies where credential_signing is set but signing_service is empty during validate_sandbox_policy() instead of failing at connection time. The runtime check in rest.rs is kept as defense-in-depth.

Extend the SigV4 proxy re-signing to auto-detect the correct payload signing mode from the client SDK's x-amz-content-sha256 header: - Hex hash → buffer body and include hash in signature (Bedrock) - STREAMING-UNSIGNED-PAYLOAD-TRAILER → sign headers only, stream body through for aws-chunked uploads (S3 PutObject, upload_fileobj) - UNSIGNED-PAYLOAD → sign headers only, no body buffering (S3 over HTTPS) - Absent → fall back to Content-Length heuristic This eliminates the need for body buffering on S3 uploads and adds support for chunked transfer encoding that the previous implementation could not handle. New credential_signing policy values: - sigv4 — auto-detect from client headers (recommended) - sigv4:body — always buffer and hash the body - sigv4:no_body — always use UNSIGNED-PAYLOAD Also adds Expect: 100-continue handling in the REST L7 relay so clients like boto3's S3 PutObject receive the interim 100 response before sending the body. Validated end-to-end from inside a Podman sandbox against real AWS: Bedrock InvokeModel, S3 PUT/GET/DELETE, and streaming upload_fileobj.

Critical: - Scope Expect: 100-continue handling to SigV4 paths only. Previously it fired for all L7-proxied requests, violating RFC 7231 §5.1.1 and risking double 100 responses on non-SigV4 traffic. Warnings: - Reject unknown credential_signing values at policy validation time. A typo like "sigv4_typo" now produces a clear PolicyViolation instead of silently falling back to no signing. - Support dualstack, FIPS, virtual-hosted, and China partition hostnames in extract_aws_region (e.g. s3.dualstack.us-west-2.amazonaws.com, s3.cn-north-1.amazonaws.com.cn). - Emit OCSF NetworkActivity event for SigV4 re-signing decisions instead of debug! tracing, per AGENTS.md structured logging guidelines. - Update architecture/sandbox.md to document all three signing modes (signed body, streaming unsigned trailer, unsigned payload) and the auto-detection mechanism.

…OCSF nit - Fix extract_aws_region for FIPS+dualstack combo hostnames like s3-fips.dualstack.us-west-2.amazonaws.com (scans past all "dualstack" labels instead of just one). - Add tests for FIPS+dualstack and GovCloud region extraction. - Add unit test for UnknownCredentialSigning policy validation (e.g. "sigv4_typo" produces the expected violation). - Use ActivityId::Traffic instead of ActivityId::Other for the SigV4 OCSF event — more descriptive for a signing operation on existing traffic flow.

…TO, startup validation Critical: - Reject STREAMING-AWS4-HMAC-SHA256-PAYLOAD in detect_payload_mode() instead of silently treating it as SignBody (per-chunk signing is not supported). Returns a clear error directing the user to sigv4:no_body. - Add defense-in-depth guard in the SignBody path: fail closed if the request uses chunked transfer encoding, preventing body-less forwards. Warnings: - Wire credential_signing and signing_service through EndpointProfile DTO in openshell-providers. Both endpoint_to_proto() and endpoint_from_proto() now preserve the fields during round-trip. - Reject unknown credential_signing values at sandbox L7 config parse time (returns None, disabling L7 for the endpoint) instead of silently downgrading to CredentialSigning::None. Also reject SigV4 modes with empty signing_service at startup rather than deferring the error to request time.

Boto3 connects to global S3 endpoints like bucket.s3.amazonaws.com (no region in the hostname). The previous extract_aws_region returned "s3" for this pattern because it took the label at parts[len-3] without checking if it was actually a region. Add looks_like_region() which requires a hyphen followed by a digit (e.g., us-east-1). Service names like "s3" or "bedrock-runtime" are rejected, causing the fallback to us-east-1. Refs NVIDIA#1576

Boto3 put_object sends x-amz-content-sha256 with the value STREAMING-AWS4-HMAC-SHA256-PAYLOAD, which was rejected by detect_payload_mode() because per-chunk signing is not supported. Treat all streaming- variants as StreamingUnsignedTrailer: re-sign headers only and stream the body through. The proxy cannot reproduce per-chunk signatures, but AWS accepts unsigned streaming payloads over HTTPS. Refs NVIDIA#1576

jhjaggars requested review from a team, derekwaynecarr, maxamillion and mrunalp as code owners May 29, 2026 19:43

jhjaggars marked this pull request as draft May 29, 2026 19:45

johntmyers reviewed Jun 1, 2026

View reviewed changes

Comment thread crates/openshell-sandbox/src/l7/rest.rs

johntmyers reviewed Jun 1, 2026

View reviewed changes

jhjaggars force-pushed the sigv4-credential-signing branch from bec3afc to e2f78da Compare June 1, 2026 15:49

jhjaggars force-pushed the sigv4-credential-signing branch from 1ee1ee5 to 84c5c3d Compare June 1, 2026 21:40

jhjaggars marked this pull request as ready for review June 1, 2026 21:41

russellb reviewed Jun 1, 2026

View reviewed changes

Comment thread crates/openshell-sandbox/src/l7/rest.rs Outdated

jhjaggars force-pushed the sigv4-credential-signing branch from 84c5c3d to 54e0f39 Compare June 1, 2026 23:08

jhjaggars added 2 commits June 1, 2026 19:39

fix(policy): validate signing_service at policy load time

7141a81

Reject policies where credential_signing is set but signing_service is empty during validate_sandbox_policy() instead of failing at connection time. The runtime check in rest.rs is kept as defense-in-depth.

jhjaggars force-pushed the sigv4-credential-signing branch from 54e0f39 to 7141a81 Compare June 1, 2026 23:40

jhjaggars added 3 commits June 2, 2026 12:30

jhjaggars requested a review from russellb June 2, 2026 17:55

jhjaggars and others added 3 commits June 2, 2026 14:25

jhjaggars mentioned this pull request Jun 2, 2026

feat(sandbox): introduce middleware layer for L7 proxy request transformations #1694

Open

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(sandbox): proxy-side AWS SigV4 credential signing for CONNECT tunnels#1638

feat(sandbox): proxy-side AWS SigV4 credential signing for CONNECT tunnels#1638
jhjaggars wants to merge 8 commits into
NVIDIA:mainfrom
jhjaggars:sigv4-credential-signing

jhjaggars commented May 29, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented May 29, 2026

Uh oh!

github-actions Bot commented May 29, 2026 •

edited

Loading

Uh oh!

jhjaggars commented May 29, 2026

Uh oh!

Uh oh!

johntmyers Jun 1, 2026

Uh oh!

jhjaggars Jun 1, 2026

Uh oh!

jhjaggars Jun 1, 2026 •

edited

Loading

Uh oh!

russellb commented Jun 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jhjaggars commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

copy-pr-bot Bot commented May 29, 2026

Uh oh!

github-actions Bot commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jhjaggars commented May 29, 2026

Uh oh!

Uh oh!

johntmyers Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

jhjaggars Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

jhjaggars Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

russellb commented Jun 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jhjaggars commented May 29, 2026 •

edited

Loading

github-actions Bot commented May 29, 2026 •

edited

Loading

jhjaggars Jun 1, 2026 •

edited

Loading