Exclude accept-encoding and hop-by-hop headers from aws_sigv4 signature

[regan-karlewicz]

Summary

Hello! This change came out of a production incident we had last night where our signed Cloudflare R2 requests began to fail with 403 responses after a deployment.

We tracked the bug to the introduction of ezstd in our code base for a separate feature:

1. When ezstd is installed, req adds zstd to the Accept-Encoding header provided by Req.Steps.compressed/1
2. Accept-Encoding header is included in the signature, and included in the authorization field SignedHeaders
3. Cloudflare R2 rewrites this header to Accept-Encoding: gzip on the edge, breaking the signature

After looking into other libraries that implement this auth mechanism e.g. boto3, we discovered that they have a list of headers they exclude from the signature.

• RFC 2616 Sect 13.5.1 "hop-by-hop" headers
• x-amzn-trace-id which can be rewritten by AWS ALBs
• Other library-specific headers that don't apply in the context of Req

Changes

• Add a list of headers to exclude when running the aws_sigv4 step:
  • RFC 2616 hop-by-hop headers
  • x-amzn-trace-id -- can be rewritten by AWS infrastructure
  • authorization -- aws_sigv4 creates this so it is not part of the canonical request
  • accept-encoding -- excluded because it is commonly rewritten by edge proxies
• Update unit tests

Alternatives Considered

I had considered moving the Req.Steps.compressed/1 step after Req.Steps.put_aws_sigv4, rejected since user-supplied accept-encoding would still get signed and hit the same bug. Additionally this change is more brittle since re-ordering this step could have other effects I'm not considering. Not reordering the steps keeps a clean mental model of "signature as the last step".

Steps to Reproduce

To reproduce the bug is straightforward:

1. Create a mix project with ezstd and req installed

  defp deps do
    [
      {:req, "~> 0.5.18"},
      {:ezstd, "~> 1.2"}
    ]
  end

2. Make a request to a Cloudflare R2 bucket using aws_sigv4, observe 403.

Req.head(
  "https://#{@account_id}.r2.cloudflarestorage.com/#{@bucket}/#{@key}",
  aws_sigv4: [
    access_key_id: @access_key_id,
    secret_access_key: @secret_access_key,
    service: :s3,
    region: "auto"
  ]
)
{:ok,
 %Req.Response{
   status: 403,
   headers: %{...},
   body: "",
   trailers: %{},
   assigns: %{},
   private: %{}
 }}

3. Include compressed: false on the request and observe 200

Req.head(
  "https://#{@account_id}.r2.cloudflarestorage.com/#{@bucket}/#{@key}",
  compressed: false,
  aws_sigv4: [
    access_key_id: @access_key_id,
    secret_access_key: @secret_access_key,
    service: :s3,
    region: "auto"
  ]
)
{:ok,
 %Req.Response{
   status: 200,
   headers: %{...},
   body: "",
   trailers: %{},
   assigns: %{},
   private: %{}
 }}

4. Remove ezstd, re-install deps, and perform the same request without compressed: false. Observe 200 response.

Req.head(
  "https://#{@account_id}.r2.cloudflarestorage.com/#{@bucket}/#{@key}",
  aws_sigv4: [
    access_key_id: @access_key_id,
    secret_access_key: @secret_access_key,
    service: :s3,
    region: "auto"
  ]
)
{:ok,
 %Req.Response{
   status: 200,
   headers: %{...},
   body: "",
   trailers: %{},
   assigns: %{},
   private: %{}
 }}

[wojtekmach]

Thank you!