Skip to content

S3 Status metadata key gets duplicated due to case sensitive handling #2242

@erincandescent

Description

@erincandescent

Bug Report

I was debugging an issue with signature validation and realised that Docs is sending multiple Status metadata headers when doing the S3 CopyObject after an upload

Specifically I saw this in the logs:

2026-04-23 17:52:21,505 botocore.endpoint DEBUG Making request for OperationModel(name=CopyObject) with params: {'url_path': '/b8406d6f-2dcd-4087-b0bd-afa49010ea0d/attachments/a0328413-a283-4361-88da-35a230982498.png', 'query_string': {}, 'method': 'PUT', 'headers': {'x-amz-copy-source': 'suite-docs-486cea00-c2fd-439a-b5f8-db02268d9789/b8406d6f-2dcd-4087-b0bd-afa49010ea0d/attachments/a0328413-a283-4361-88da-35a230982498.png', 'Content-Type': 'image/png', 'x-amz-meta-Owner': 'd3a2a5c3-c78e-4ff8-bf92-0b6c1b90e4b8', 'x-amz-meta-Status': 'processing', 'x-amz-meta-status': <DocumentAttachmentStatus.READY: 'ready'>, 'x-amz-metadata-directive': 'REPLACE', 'User-Agent': 'Boto3/1.42.59 md/Botocore#1.42.94 ua/2.1 os/linux#6.6.49-0-lts md/arch#x86_64 lang/python#3.13.3 md/pyimpl#CPython m/a,e,c,N,D cfg/retry-mode#legacy Botocore/1.42.94 Resource'}, 'body': b'', 'auth_path': '/suite-docs-486cea00-c2fd-439a-b5f8-db02268d9789/b8406d6f-2dcd-4087-b0bd-afa49010ea0d/attachments/a0328413-a283-4361-88da-35a230982498.png', 'url': 'https://s3.alioth.systems/suite-docs-486cea00-c2fd-439a-b5f8-db02268d9789/b8406d6f-2dcd-4087-b0bd-afa49010ea0d/attachments/a0328413-a283-4361-88da-35a230982498.png', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f5beac63890>, 'has_streaming_input': False, 'auth_type': None, 'unsigned_payload': None, 'auth_options': ['aws.auth#sigv4'], 's3_redirect': {'redirected': False, 'bucket': 'suite-docs-486cea00-c2fd-439a-b5f8-db02268d9789', 'params': {'Bucket': 'suite-docs-486cea00-c2fd-439a-b5f8-db02268d9789', 'CopySource': 'suite-docs-486cea00-c2fd-439a-b5f8-db02268d9789/b8406d6f-2dcd-4087-b0bd-afa49010ea0d/attachments/a0328413-a283-4361-88da-35a230982498.png', 'Key': 'b8406d6f-2dcd-4087-b0bd-afa49010ea0d/attachments/a0328413-a283-4361-88da-35a230982498.png', 'ContentType': 'image/png', 'Metadata': {'Owner': 'd3a2a5c3-c78e-4ff8-bf92-0b6c1b90e4b8', 'Status': 'processing', 'status': <DocumentAttachmentStatus.READY: 'ready'>}, 'MetadataDirective': 'REPLACE'}}, 'input_params': {'Bucket': 'suite-docs-486cea00-c2fd-439a-b5f8-db02268d9789', 'Key': 'b8406d6f-2dcd-4087-b0bd-afa49010ea0d/attachments/a0328413-a283-4361-88da-35a230982498.png'}, 'signing': {}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'us-east-1'}]}}}

Specifically see the metadata map:

 'Metadata': {
    'Owner': 'd3a2a5c3-c78e-4ff8-bf92-0b6c1b90e4b8', 
    'Status': 'processing', 
    'status': <DocumentAttachmentStatus.READY: 'ready'>
}

I think this may be the (a) root cause of #1189. Something inside boto3 or urllib3 appears to behave differently when X-Amz-Meta-Status and X-Amz-Meta-status are both present between HTTP 1 and HTTP 2. In our case this appears to eventually tickle a Ceph RadosGW bug, where it loses one of the duplicates and this invalidates the AWS request signature.

Problematic behavior
We shouldn't end up with two status headers

Expected behavior/code
We should emit a single status header

Steps to Reproduce

  1. Setup Docs with an S3 implementation/proxy which returns metadata not in lowercase
  2. Try to upload a file
  3. Watch as two headers pop out!

Environment

  • Docs version: 4.8.6
  • Ceph RadosGW 19.2.3 as S3 implementation, bug manifests when Traefik is used as proxy between the two

Possible Solution
Case sensitive map? Delete any meta header that case insensitively matches status? 🤷‍♀️

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions