Remove the python-dateutil dependency#3715
Conversation
* release-1.38.35: Bumping version to 1.38.35 Update partitions file Update endpoints model Update to latest models Add support for decimal.Decimal to the JSON serializer (boto#3487)
* release-1.38.36: Bumping version to 1.38.36 Update to latest models
* release-1.38.37: Bumping version to 1.38.37 Update to latest models Bump github/codeql-action from 3.28.1 to 3.29.0
* release-1.38.38: Bumping version to 1.38.38 Update to latest models Merge customizations for SecurityHub
* release-1.38.39: Bumping version to 1.38.39 Update endpoints model Update to latest models Add missing imagebuilder paginators (boto#3493)
* release-1.38.40: Bumping version to 1.38.40 Update to latest models Format strings in tests Format botocore files for imports and string formatting fixes Update dependency versions and ruff config
* release-1.38.41: Bumping version to 1.38.41 Update endpoints model Update to latest models
* release-1.38.42: Bumping version to 1.38.42 Update to latest models
* release-1.38.43: Bumping version to 1.38.43 Update to latest models
* release-1.38.44: Bumping version to 1.38.44 Update to latest models
* release-1.38.45: Bumping version to 1.38.45 Update endpoints model Update to latest models Merge customizations for KeyspacesStreams
* release-1.38.46: Bumping version to 1.38.46 Update endpoints model Update to latest models
* release-1.39.0: Bumping version to 1.39.0 Update endpoints model Update to latest models Merge customizations for Bedrock Add query compat protocol tests (boto#3500)
* release-1.39.2: Bumping version to 1.39.2 Update to latest models
* release-1.39.3: Bumping version to 1.39.3 Update endpoints model Update to latest models
* release-1.39.4: Bumping version to 1.39.4 Update endpoints model Update to latest models
* release-1.39.5: Bumping version to 1.39.5 Update endpoints model Update to latest models Add changes to parse all output members from the response body even when the same shape is reused for input. (boto#3509) Disable elastic beanstalk smoketests (boto#3519) Add test to ensure request checksum is calculated once (boto#3516) Replace bespoke NullHandler with Python's impl (boto#3510)
* release-1.39.6: Bumping version to 1.39.6 Update endpoints model Update to latest models Consolidate STS settings (boto#3521)
* release-1.39.7: Bumping version to 1.39.7 Update to latest models Query compatibility bugfix (boto#3515)
* release-1.39.8: Bumping version to 1.39.8 Update endpoints model Update to latest models
* release-1.39.10: Bumping version to 1.39.10 Update endpoints model Update to latest models Merge customizations for CloudFront
* release-1.39.11: Bumping version to 1.39.11 Update endpoints model Update to latest models
* release-1.39.12: Bumping version to 1.39.12 Update endpoints model Update to latest models Restrict bedrock bearer auth by trait (boto#3526)
* release-1.39.13: Bumping version to 1.39.13 Update endpoints model Update to latest models Merge customizations for DataZone
* release-1.39.14: Bumping version to 1.39.14 Update to latest models Merge customizations for SocialMessaging
* release-1.39.15: Bumping version to 1.39.15 Update endpoints model Update to latest models Add SSOTokenLoader expiration check (boto#3441)
* release-1.39.16: Bumping version to 1.39.16 Update endpoints model Update to latest models
* release-1.39.17: Bumping version to 1.39.17 Update endpoints model Update to latest models
* release-1.43.14: Bumping version to 1.43.14 Update endpoints model Update to latest models Improving caching of S3 endpoints by omitting unused parameters (boto#3692)
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #3715 +/- ##
===========================================
- Coverage 92.59% 92.57% -0.03%
===========================================
Files 68 68
Lines 15634 15801 +167
===========================================
+ Hits 14477 14627 +150
- Misses 1157 1174 +17 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
* release-1.43.15: Bumping version to 1.43.15 Update to latest models Temporarily disable route53 smoke tests (boto#3714)
* release-1.43.16: Bumping version to 1.43.16 Update to latest models
* release-1.43.17: Bumping version to 1.43.17 Update endpoints model Update to latest models Merge customizations for resiliencehubv2
* release-1.43.18: Bumping version to 1.43.18 Update endpoints model Update to latest models Merge customizations for QuickSight Add a changelog for retries update (boto#3717)
* release-1.43.19: Bumping version to 1.43.19 Update to latest models
* release-1.43.20: Bumping version to 1.43.20 Update to latest models
* release-1.43.21: Bumping version to 1.43.21 Update endpoints model Update to latest models
* release-1.43.22: Bumping version to 1.43.22 Update to latest models Merge customizations for SocialMessaging
* release-1.43.23: Bumping version to 1.43.23 Update to latest models
* release-1.43.24: Bumping version to 1.43.24 Update endpoints model Update to latest models Merge customizations for QuickSight
* release-1.43.25: Bumping version to 1.43.25 Update to latest models
* release-1.43.26: Bumping version to 1.43.26 Update endpoints model Update to latest models Cleanup dead urllib3 import specification in setup.cfg (boto#3723)
* release-1.43.27: Bumping version to 1.43.27 Update to latest models Bump github/codeql-action in the github-actions group (boto#3720)
* release-1.43.28: Bumping version to 1.43.28 Update to latest models
* release-1.43.29: Bumping version to 1.43.29 Update endpoints model Update to latest models Bump https://github.com/astral-sh/ruff-pre-commit (boto#3726)
* release-1.43.30: Bumping version to 1.43.30 Update to latest models
0c5b530 to
d9fc203
Compare
Use a helper from the standard library to set up local timezones, reducing reliance on `dateutil` library Signed-off-by: Mike Fiedler <miketheman@gmail.com>
Signed-off-by: Mike Fiedler <miketheman@gmail.com>
Mechanical change, removed test shim to improve clarity. Signed-off-by: Mike Fiedler <miketheman@gmail.com>
Swap dateutil.tz.tzutc for datetime.timezone.utc in credentials, tokens, and utils. One use in utils.py stays for now — it's inside a dateutil.parser.parse call and will be handled with a parser refactor. Refs: https://docs.python.org/3/library/datetime.html#datetime.timezone.utc Signed-off-by: Mike Fiedler <miketheman@gmail.com>
Replace mock.patch('datetime.datetime', spec=True) with a small
_FrozenDatetime subclass that overrides only now(). Subclassing keeps
fromisoformat / strptime / fromtimestamp callable through inheritance.
Same pattern as tests/unit/test_utils.py's MockDatetime fixture.
Signed-off-by: Mike Fiedler <miketheman@gmail.com>
Replace usages of dateutil.parser in any tests with stdlib equivalent behaviors. Leave a note about Python 3.11 support change. Signed-off-by: Mike Fiedler <miketheman@gmail.com>
Replace dateutil.parser.parse with datetime.fromisoformat in credentials, tokens, and utils. A shared _normalize_iso8601_for_python_310 helper in botocore.compat handles the ISO 8601 variants Python 3.10's strict fromisoformat doesn't accept: Z/UTC/GMT zone suffixes, compact basic form, space date/time separator, tz offsets without a colon, and fractional seconds with anything other than 3 or 6 digits. 3.11+ accepts these natively; the helper and its callers are marked for removal when 3.10 support ends. _parse_timestamp_with_tzinfo's string fallback also goes through email.utils.parsedate_to_datetime and a couple of strptime formats for legacy HTTP Expires shapes. parse_timestamp now raises ValueError for non-string input instead of leaking AttributeError. test_timestamp_example's tm_isdst expectation moves from 0 to -1. stdlib's datetime.timezone.utc returns None from dst() where dateutil's tzutc returned timedelta(0). The rendered datetime() in shared-example docs picks up the new value; the output is illustrative, not runnable. Refs: - https://docs.python.org/3/library/datetime.html#datetime.datetime.fromisoformat - https://docs.python.org/3/library/email.utils.html#email.utils.parsedate_to_datetime Signed-off-by: Mike Fiedler <miketheman@gmail.com>
No imports, external API preserved. Signed-off-by: Mike Fiedler <miketheman@gmail.com>
d9fc203 to
aead776
Compare
SamRemis
left a comment
There was a problem hiding this comment.
[Note]
This did pass our all of our test suites when ran locally, including integ tests against live services
| return email.utils.parsedate_to_datetime(value) | ||
| except (TypeError, ValueError): | ||
| pass | ||
| for fmt in ("%m/%d/%Y", "%d %b %Y"): |
There was a problem hiding this comment.
[Question]
This catches a few non-Smithy shapes but rejects others that dateutil handled. For example, it accepts 06/25/2026 but not 2026/06/25, and 15 Jan 2026 but not January 15, 2026. Why these two formats specifically?
| @@ -28,7 +28,6 @@ | |||
|
|
|||
| from botocore.vendored import six | |||
| from botocore.exceptions import MD5UnavailableError | |||
There was a problem hiding this comment.
[Discussion point]
Starting off with my biggest comment:
We have a specification for what accepted timestamp shapes are, so we should only ever need to accept those for parsing. This PR properly handles all officially supported datetime formats, so theoretically this is a fully safe change to make despite handling fewer formats than dateutil.
There are still a few places that I'm still concerned about breaking:
- AWS services that are incorrectly returning these values which botocore accepts
- Non-AWS third party services relying on botocore's overly permissive behavior
It's a bad thing that botocore is so permissive. We have seen AWS service teams test their service with other datetime formats against botocore and seen it work, then be surprised when the same datetime format broke the Java SDK. If we ever want to tighten up this behavior, we have to rip the bandaid off at some point or fully commit to supporting this forever.
One way we could make this a little more cautiously is to start by using this new code path first and emitting a warning when we have to fall back to the old one, then ripping out the legacy path in three months with a minor version bump. What do you think?
Description of changes:
Replaces every
dateutilusage with stdlib, then drops the requirement.Please review commit by commit. Each pair (test, then refactor) handles one piece on its own.
Why
dateutil.tzanddateutil.parsergave us timezone and string parsing.Python 3.10+ stdlib (
datetime,email.utils) handles every input shape we actually see.Removing dateutil also removes its transitive
sixdep - one fewer thing for packagersand security scanners to vet on every release.
(Note: botocore still carries a vendored version of
six- that is out of scope of this change, and may come later)Review order
Tests first, then refactor:
test: replace dateutil.tz.tzlocalrefactor: replace dateutil.tz.tzlocaltest: replace usages of dateutil.tz.tzutcrefactor: replace dateutil.tz.tzutc with stdlibtest: use consistent datetime fixtures in s3expresstest: replace dateutil.parser with stdlibrefactor: replace dateutil.parser with stdlibchore(deps): remove unused python-dateutilBehavior change
parse_timestampof an RFC 822 GMT string returns a datetime withtzinfo=datetime.timezone.utcinstead ofdateutil.tz.tzutc().Same instant, different
tm_isdst(-1 vs 0). One doc-renderer test was updated to match.Follow-ups
botocore.compat.get_tzinfo_optionsis now a one-line shim.The TODO in the function describes the cleanup; it removes a public
symbol, so it should be considered separately.
botocore.endpoint._calculate_ttlusesstrptime("%Z"), which isplatform-dependent.
email.utils.parsedate_to_datetimewould be abetter fit. Separate concern, may come at another time.
Issue #, if available: Resolves #3070
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.