Bug Report
I was debugging an issue with signature validation and realised that Docs is sending multiple Status metadata headers when doing the S3 CopyObject after an upload
Specifically I saw this in the logs:
2026-04-23 17:52:21,505 botocore.endpoint DEBUG Making request for OperationModel(name=CopyObject) with params: {'url_path': '/b8406d6f-2dcd-4087-b0bd-afa49010ea0d/attachments/a0328413-a283-4361-88da-35a230982498.png', 'query_string': {}, 'method': 'PUT', 'headers': {'x-amz-copy-source': 'suite-docs-486cea00-c2fd-439a-b5f8-db02268d9789/b8406d6f-2dcd-4087-b0bd-afa49010ea0d/attachments/a0328413-a283-4361-88da-35a230982498.png', 'Content-Type': 'image/png', 'x-amz-meta-Owner': 'd3a2a5c3-c78e-4ff8-bf92-0b6c1b90e4b8', 'x-amz-meta-Status': 'processing', 'x-amz-meta-status': <DocumentAttachmentStatus.READY: 'ready'>, 'x-amz-metadata-directive': 'REPLACE', 'User-Agent': 'Boto3/1.42.59 md/Botocore#1.42.94 ua/2.1 os/linux#6.6.49-0-lts md/arch#x86_64 lang/python#3.13.3 md/pyimpl#CPython m/a,e,c,N,D cfg/retry-mode#legacy Botocore/1.42.94 Resource'}, 'body': b'', 'auth_path': '/suite-docs-486cea00-c2fd-439a-b5f8-db02268d9789/b8406d6f-2dcd-4087-b0bd-afa49010ea0d/attachments/a0328413-a283-4361-88da-35a230982498.png', 'url': 'https://s3.alioth.systems/suite-docs-486cea00-c2fd-439a-b5f8-db02268d9789/b8406d6f-2dcd-4087-b0bd-afa49010ea0d/attachments/a0328413-a283-4361-88da-35a230982498.png', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f5beac63890>, 'has_streaming_input': False, 'auth_type': None, 'unsigned_payload': None, 'auth_options': ['aws.auth#sigv4'], 's3_redirect': {'redirected': False, 'bucket': 'suite-docs-486cea00-c2fd-439a-b5f8-db02268d9789', 'params': {'Bucket': 'suite-docs-486cea00-c2fd-439a-b5f8-db02268d9789', 'CopySource': 'suite-docs-486cea00-c2fd-439a-b5f8-db02268d9789/b8406d6f-2dcd-4087-b0bd-afa49010ea0d/attachments/a0328413-a283-4361-88da-35a230982498.png', 'Key': 'b8406d6f-2dcd-4087-b0bd-afa49010ea0d/attachments/a0328413-a283-4361-88da-35a230982498.png', 'ContentType': 'image/png', 'Metadata': {'Owner': 'd3a2a5c3-c78e-4ff8-bf92-0b6c1b90e4b8', 'Status': 'processing', 'status': <DocumentAttachmentStatus.READY: 'ready'>}, 'MetadataDirective': 'REPLACE'}}, 'input_params': {'Bucket': 'suite-docs-486cea00-c2fd-439a-b5f8-db02268d9789', 'Key': 'b8406d6f-2dcd-4087-b0bd-afa49010ea0d/attachments/a0328413-a283-4361-88da-35a230982498.png'}, 'signing': {}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'us-east-1'}]}}}
Specifically see the metadata map:
'Metadata': {
'Owner': 'd3a2a5c3-c78e-4ff8-bf92-0b6c1b90e4b8',
'Status': 'processing',
'status': <DocumentAttachmentStatus.READY: 'ready'>
}
I think this may be the (a) root cause of #1189. Something inside boto3 or urllib3 appears to behave differently when X-Amz-Meta-Status and X-Amz-Meta-status are both present between HTTP 1 and HTTP 2. In our case this appears to eventually tickle a Ceph RadosGW bug, where it loses one of the duplicates and this invalidates the AWS request signature.
Problematic behavior
We shouldn't end up with two status headers
Expected behavior/code
We should emit a single status header
Steps to Reproduce
- Setup Docs with an S3 implementation/proxy which returns metadata not in lowercase
- Try to upload a file
- Watch as two headers pop out!
Environment
- Docs version: 4.8.6
- Ceph RadosGW 19.2.3 as S3 implementation, bug manifests when Traefik is used as proxy between the two
Possible Solution
Case sensitive map? Delete any meta header that case insensitively matches status? 🤷♀️
Bug Report
I was debugging an issue with signature validation and realised that Docs is sending multiple Status metadata headers when doing the S3 CopyObject after an upload
Specifically I saw this in the logs:
Specifically see the metadata map:
I think this may be the (a) root cause of #1189. Something inside boto3 or urllib3 appears to behave differently when
X-Amz-Meta-StatusandX-Amz-Meta-statusare both present between HTTP 1 and HTTP 2. In our case this appears to eventually tickle a Ceph RadosGW bug, where it loses one of the duplicates and this invalidates the AWS request signature.Problematic behavior
We shouldn't end up with two status headers
Expected behavior/code
We should emit a single status header
Steps to Reproduce
Environment
Possible Solution
Case sensitive map? Delete any meta header that case insensitively matches
status? 🤷♀️