feat: set user_agent_extra for S3 server-side request attribution#169
Open
goanpeca wants to merge 1 commit into
Open
feat: set user_agent_extra for S3 server-side request attribution#169goanpeca wants to merge 1 commit into
goanpeca wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds S3 User-Agent attribution so traffic originating from dvc-s3 is identifiable in S3/B2 server-side logs by setting botocore’s Config.user_agent_extra to a dvc-s3/<version> token (with a safe fallback) while still honoring per-remote overrides.
Changes:
- Resolve a default
user_agent_extraat module import time viaimportlib.metadata.version("dvc-s3"), falling back to"dvc-s3"when metadata is unavailable. - Pass
user_agent_extraintoconfig_kwargsin_prepare_credentials, allowing config overrides viauser_agent_extra. - Add tests covering the default and override behaviors.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
dvc_s3/__init__.py |
Introduces and applies a default user_agent_extra for botocore Config, with override support. |
dvc_s3/tests/test_s3.py |
Adds assertions for default user_agent_extra behavior and explicit override handling. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
f08ad5c to
1056219
Compare
1056219 to
8a90334
Compare
8a90334 to
23a33c2
Compare
23a33c2 to
c60d0ec
Compare
c60d0ec to
95d4233
Compare
95d4233 to
04fc5c7
Compare
04fc5c7 to
54032ef
Compare
54032ef to
7533832
Compare
7533832 to
d32ea12
Compare
Author
|
Hi @skshetry 🙂 All Copilot threads resolved. Happy to tweak anything! |
d32ea12 to
5d75f8c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DVC traffic on S3 server-side logs is currently indistinguishable from any other
aiobotocorecaller:Config.user_agent_extrais unset, so the wire UA only containsaiobotocore/X botocore/Y.This sets
user_agent_extra = "dvc-s3/<version>"on the boto3Configbuilt inS3FileSystem._prepare_credentials. The version is resolved lazily on first use (cached) viaimportlib.metadata.version("dvc-s3"), falling back to"dvc-s3/dev"for source / frozen installs (RFC 9110 product/version form preserved).A
user_agent_extrapassed to theS3FileSystemconstructor is appended to the dvc-s3 suffix rather than replacing it, so the wire UA always carries the dvc-s3 identifier: with no override it isdvc-s3/<version>, and withuser_agent_extra="downstream/1.2.3"it becomesdvc-s3/<version> downstream/1.2.3. The value is stripped of surrounding whitespace and rejected (ConfigError) if it contains control characters, to avoid malformed or injectedUser-Agentheaders.Note: this wires
user_agent_extraat theS3FileSystemconstructor / programmatic level. Reaching it throughdvc remote modify <name> user_agent_extra ...additionally requires adding the key to DVC's S3 remote schema initerative/dvc, which is a separate follow-up.Tests in
dvc_s3/tests/test_s3.pycover the default, the appended user value, whitespace stripping, control-character rejection, and the real on-the-wireUser-Agent(via moto).