Skip to content

Decide request-parameter coverage, error translation, and S3 retry codes #1342

Open

Description

@laughingman7743

Part of #1327 (design problems 6 and 7 of the audit report).

Summary

Decide and document which request parameters reach which requests, how S3 errors are translated, and which layer retries S3 errors.

Current state (766d79d)

  1. s3_additional_kwargs reaches open, pipe_file and put_file only (documented), not touch, cat_file, cp_file/copy/mv, setxattr or rm: users migrating from s3fs who put SSE or SSE-C settings there get different results from touch or cp than from pipe, and cat(p) fails where open(p).read() works. Per call, rm_file ignores parameters that rm forwards, mkdir drops everything but acl and region_name, and touch(path, Body=b"x") writes data although it creates an empty object.
  2. Append mode updates the request with appended.to_api_repr(), which overrides the caller's ContentType, StorageClass and Metadata and forwards SSE and object-lock fields of the existing object (s3.py:2803); documented as keeping the old object's metadata.
  3. Error translation: only mkdir turns ParamValidationError into ValueError; cat_file and clear_multipart_uploads re-parse e.__cause__ because the translated OSError carries no S3 code; exists() maps 403 to True for buckets and raises PermissionError for keys; copy_object_annotation reads its payload without closing it or translating errors.
  4. Retries: the default RetryConfig retries ThrottlingException and TooManyRequestsException, which are Athena's codes. S3's SlowDown, ServiceUnavailable, InternalError and RequestTimeout are retried only by botocore, so stop_retries_on and unstoppable_retries govern no S3 retry unless the user adds S3 codes to a policy that a connection shares with Athena. docs/filesystem.md says only that requests are "sent with the retry policy". Retrying in both layers is acceptable; the question is which codes PyAthena's layer retries for S3 and how that is documented.

Proposal (for decision)

  1. Document, or extend, the coverage of s3_additional_kwargs, and make rm_file, mkdir and touch consistent with it.
  2. Let explicit caller parameters win over the appended object's in append mode, or document the precedence.
  3. Carry the S3 error code on translated errors, add core options such as missing_ok for aborts and an InvalidRange mode for get_object, and pick one ParamValidationError policy.
  4. Give S3 its own default retry codes, separate from Athena's, or document that botocore owns S3 retries.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions