Skip to content

feat: set user_agent_extra for S3 server-side request attribution#169

Open
goanpeca wants to merge 1 commit into
treeverse:mainfrom
goanpeca:feat/user-agent-extra
Open

feat: set user_agent_extra for S3 server-side request attribution#169
goanpeca wants to merge 1 commit into
treeverse:mainfrom
goanpeca:feat/user-agent-extra

Conversation

@goanpeca
Copy link
Copy Markdown

@goanpeca goanpeca commented May 21, 2026

DVC traffic on S3 server-side logs is currently indistinguishable from any other aiobotocore caller: Config.user_agent_extra is unset, so the wire UA only contains aiobotocore/X botocore/Y.

This sets user_agent_extra = "dvc-s3/<version>" on the boto3 Config built in S3FileSystem._prepare_credentials. The version is resolved lazily on first use (cached) via importlib.metadata.version("dvc-s3"), falling back to "dvc-s3/dev" for source / frozen installs (RFC 9110 product/version form preserved).

A user_agent_extra passed to the S3FileSystem constructor is appended to the dvc-s3 suffix rather than replacing it, so the wire UA always carries the dvc-s3 identifier: with no override it is dvc-s3/<version>, and with user_agent_extra="downstream/1.2.3" it becomes dvc-s3/<version> downstream/1.2.3. The value is stripped of surrounding whitespace and rejected (ConfigError) if it contains control characters, to avoid malformed or injected User-Agent headers.

Note: this wires user_agent_extra at the S3FileSystem constructor / programmatic level. Reaching it through dvc remote modify <name> user_agent_extra ... additionally requires adding the key to DVC's S3 remote schema in iterative/dvc, which is a separate follow-up.

Tests in dvc_s3/tests/test_s3.py cover the default, the appended user value, whitespace stripping, control-character rejection, and the real on-the-wire User-Agent (via moto).

Copilot AI review requested due to automatic review settings May 21, 2026 21:12
@CLAassistant
Copy link
Copy Markdown

CLAassistant commented May 21, 2026

CLA assistant check
All committers have signed the CLA.

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds S3 User-Agent attribution so traffic originating from dvc-s3 is identifiable in S3/B2 server-side logs by setting botocore’s Config.user_agent_extra to a dvc-s3/<version> token (with a safe fallback) while still honoring per-remote overrides.

Changes:

  • Resolve a default user_agent_extra at module import time via importlib.metadata.version("dvc-s3"), falling back to "dvc-s3" when metadata is unavailable.
  • Pass user_agent_extra into config_kwargs in _prepare_credentials, allowing config overrides via user_agent_extra.
  • Add tests covering the default and override behaviors.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
dvc_s3/__init__.py Introduces and applies a default user_agent_extra for botocore Config, with override support.
dvc_s3/tests/test_s3.py Adds assertions for default user_agent_extra behavior and explicit override handling.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread dvc_s3/__init__.py Outdated
@goanpeca goanpeca force-pushed the feat/user-agent-extra branch from f08ad5c to 1056219 Compare May 21, 2026 23:36
@goanpeca goanpeca requested a review from Copilot June 3, 2026 23:04
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

Comment thread dvc_s3/__init__.py Outdated
Comment thread dvc_s3/__init__.py Outdated
Comment thread dvc_s3/__init__.py
@goanpeca goanpeca force-pushed the feat/user-agent-extra branch from 1056219 to 8a90334 Compare June 3, 2026 23:20
@goanpeca goanpeca requested a review from Copilot June 3, 2026 23:22
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

Comment thread dvc_s3/__init__.py
Comment thread dvc_s3/tests/test_s3.py Outdated
Comment thread dvc_s3/tests/test_s3.py Outdated
@goanpeca goanpeca force-pushed the feat/user-agent-extra branch from 8a90334 to 23a33c2 Compare June 3, 2026 23:41
@goanpeca goanpeca requested a review from Copilot June 3, 2026 23:49
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

Comment thread dvc_s3/tests/test_s3.py
Comment thread dvc_s3/__init__.py
@goanpeca goanpeca force-pushed the feat/user-agent-extra branch from 23a33c2 to c60d0ec Compare June 3, 2026 23:59
@goanpeca goanpeca requested a review from Copilot June 4, 2026 00:02
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Comment thread dvc_s3/tests/test_s3.py
@goanpeca goanpeca force-pushed the feat/user-agent-extra branch from c60d0ec to 95d4233 Compare June 4, 2026 00:06
@goanpeca goanpeca requested a review from Copilot June 4, 2026 00:07
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

Comment thread dvc_s3/__init__.py
Comment thread dvc_s3/tests/test_s3.py Outdated
Comment thread dvc_s3/__init__.py Outdated
Comment thread dvc_s3/__init__.py Outdated
@goanpeca goanpeca force-pushed the feat/user-agent-extra branch from 95d4233 to 04fc5c7 Compare June 4, 2026 00:18
@goanpeca goanpeca requested a review from Copilot June 4, 2026 00:19
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

Comment thread dvc_s3/tests/test_s3.py Outdated
Comment thread dvc_s3/__init__.py
@goanpeca goanpeca force-pushed the feat/user-agent-extra branch from 04fc5c7 to 54032ef Compare June 4, 2026 00:26
@goanpeca goanpeca requested a review from Copilot June 4, 2026 00:28
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

Comment thread dvc_s3/__init__.py
Comment thread dvc_s3/__init__.py
@goanpeca goanpeca requested a review from Copilot June 4, 2026 00:31
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

Comment thread dvc_s3/tests/test_s3.py Outdated
Comment thread dvc_s3/__init__.py Outdated
@goanpeca goanpeca force-pushed the feat/user-agent-extra branch from 54032ef to 7533832 Compare June 4, 2026 00:37
@goanpeca goanpeca requested a review from Copilot June 4, 2026 00:38
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

Comment thread dvc_s3/__init__.py
Comment thread dvc_s3/tests/test_s3.py Outdated
@goanpeca goanpeca force-pushed the feat/user-agent-extra branch from 7533832 to d32ea12 Compare June 4, 2026 00:44
@goanpeca goanpeca requested a review from Copilot June 4, 2026 00:44
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

@goanpeca goanpeca requested a review from Copilot June 4, 2026 00:46
@goanpeca
Copy link
Copy Markdown
Author

goanpeca commented Jun 4, 2026

Hi @skshetry 🙂

All Copilot threads resolved.

Happy to tweak anything!

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Comment thread dvc_s3/__init__.py
Comment thread dvc_s3/tests/test_s3.py
@goanpeca goanpeca force-pushed the feat/user-agent-extra branch from d32ea12 to 5d75f8c Compare June 4, 2026 00:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants