Skip to content

Remove the python-dateutil dependency#3715

Open
miketheman wants to merge 2449 commits into
boto:developfrom
miketheman:remove-dateutil
Open

Remove the python-dateutil dependency#3715
miketheman wants to merge 2449 commits into
boto:developfrom
miketheman:remove-dateutil

Conversation

@miketheman

@miketheman miketheman commented May 25, 2026

Copy link
Copy Markdown
Contributor

Description of changes:

Replaces every dateutil usage with stdlib, then drops the requirement.
Please review commit by commit. Each pair (test, then refactor) handles one piece on its own.

Why

dateutil.tz and dateutil.parser gave us timezone and string parsing.
Python 3.10+ stdlib (datetime, email.utils) handles every input shape we actually see.
Removing dateutil also removes its transitive six dep - one fewer thing for packagers
and security scanners to vet on every release.

(Note: botocore still carries a vendored version of six - that is out of scope of this change, and may come later)

Review order

Tests first, then refactor:

  1. test: replace dateutil.tz.tzlocal
  2. refactor: replace dateutil.tz.tzlocal
  3. test: replace usages of dateutil.tz.tzutc
  4. refactor: replace dateutil.tz.tzutc with stdlib
  5. test: use consistent datetime fixtures in s3express
  6. test: replace dateutil.parser with stdlib
  7. refactor: replace dateutil.parser with stdlib
  8. chore(deps): remove unused python-dateutil

Behavior change

parse_timestamp of an RFC 822 GMT string returns a datetime with tzinfo=datetime.timezone.utc instead of dateutil.tz.tzutc().
Same instant, different tm_isdst (-1 vs 0). One doc-renderer test was updated to match.

Follow-ups

  • botocore.compat.get_tzinfo_options is now a one-line shim.
    The TODO in the function describes the cleanup; it removes a public
    symbol, so it should be considered separately.
  • botocore.endpoint._calculate_ttl uses strptime("%Z"), which is
    platform-dependent. email.utils.parsedate_to_datetime would be a
    better fit. Separate concern, may come at another time.

Issue #, if available: Resolves #3070

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

* release-1.38.35:
  Bumping version to 1.38.35
  Update partitions file
  Update endpoints model
  Update to latest models
  Add support for decimal.Decimal to the JSON serializer (boto#3487)
* release-1.38.36:
  Bumping version to 1.38.36
  Update to latest models
* release-1.38.37:
  Bumping version to 1.38.37
  Update to latest models
  Bump github/codeql-action from 3.28.1 to 3.29.0
* release-1.38.38:
  Bumping version to 1.38.38
  Update to latest models
  Merge customizations for SecurityHub
* release-1.38.39:
  Bumping version to 1.38.39
  Update endpoints model
  Update to latest models
  Add missing imagebuilder paginators (boto#3493)
* release-1.38.40:
  Bumping version to 1.38.40
  Update to latest models
  Format strings in tests
  Format botocore files for imports and string formatting fixes
  Update dependency versions and ruff config
* release-1.38.41:
  Bumping version to 1.38.41
  Update endpoints model
  Update to latest models
* release-1.38.42:
  Bumping version to 1.38.42
  Update to latest models
* release-1.38.43:
  Bumping version to 1.38.43
  Update to latest models
* release-1.38.44:
  Bumping version to 1.38.44
  Update to latest models
* release-1.38.45:
  Bumping version to 1.38.45
  Update endpoints model
  Update to latest models
  Merge customizations for KeyspacesStreams
* release-1.38.46:
  Bumping version to 1.38.46
  Update endpoints model
  Update to latest models
* release-1.39.0:
  Bumping version to 1.39.0
  Update endpoints model
  Update to latest models
  Merge customizations for Bedrock
  Add query compat protocol tests (boto#3500)
* release-1.39.1:
  Bumping version to 1.39.1
  Update endpoints model
  Update to latest models
  Remove unused isort configuration (boto#3505)
  Remove legacy reference to Requests (boto#3503)
  Remove outdated/unnecessary compat shims (boto#3502)
  Add CloudWatch forwards compatibility test (boto#3501)
* release-1.39.2:
  Bumping version to 1.39.2
  Update to latest models
* release-1.39.3:
  Bumping version to 1.39.3
  Update endpoints model
  Update to latest models
* release-1.39.4:
  Bumping version to 1.39.4
  Update endpoints model
  Update to latest models
* release-1.39.5:
  Bumping version to 1.39.5
  Update endpoints model
  Update to latest models
  Add changes to parse all output members from the response body even when the same shape is reused for input. (boto#3509)
  Disable elastic beanstalk smoketests (boto#3519)
  Add test to ensure request checksum is calculated once (boto#3516)
  Replace bespoke NullHandler with Python's impl (boto#3510)
* release-1.39.6:
  Bumping version to 1.39.6
  Update endpoints model
  Update to latest models
  Consolidate STS settings (boto#3521)
* release-1.39.7:
  Bumping version to 1.39.7
  Update to latest models
  Query compatibility bugfix (boto#3515)
* release-1.39.8:
  Bumping version to 1.39.8
  Update endpoints model
  Update to latest models
* release-1.39.9:
  Bumping version to 1.39.9
  Update endpoints model
  Update to latest models
  Merge customizations for Outposts
  Experimental plugin autoloading (boto#3514)
  Add official Python 3.14 support with first release candidate (boto#3522)
* release-1.39.10:
  Bumping version to 1.39.10
  Update endpoints model
  Update to latest models
  Merge customizations for CloudFront
* release-1.39.11:
  Bumping version to 1.39.11
  Update endpoints model
  Update to latest models
* release-1.39.12:
  Bumping version to 1.39.12
  Update endpoints model
  Update to latest models
  Restrict bedrock bearer auth by trait (boto#3526)
* release-1.39.13:
  Bumping version to 1.39.13
  Update endpoints model
  Update to latest models
  Merge customizations for DataZone
* release-1.39.14:
  Bumping version to 1.39.14
  Update to latest models
  Merge customizations for SocialMessaging
* release-1.39.15:
  Bumping version to 1.39.15
  Update endpoints model
  Update to latest models
  Add SSOTokenLoader expiration check (boto#3441)
* release-1.39.16:
  Bumping version to 1.39.16
  Update endpoints model
  Update to latest models
* release-1.39.17:
  Bumping version to 1.39.17
  Update endpoints model
  Update to latest models
* release-1.43.14:
  Bumping version to 1.43.14
  Update endpoints model
  Update to latest models
  Improving caching of S3 endpoints by omitting unused parameters (boto#3692)
@codecov-commenter

codecov-commenter commented May 26, 2026

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.57%. Comparing base (ee748e4) to head (aead776).
⚠️ Report is 286 commits behind head on develop.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #3715      +/-   ##
===========================================
- Coverage    92.59%   92.57%   -0.03%     
===========================================
  Files           68       68              
  Lines        15634    15801     +167     
===========================================
+ Hits         14477    14627     +150     
- Misses        1157     1174      +17     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

* release-1.43.15:
  Bumping version to 1.43.15
  Update to latest models
  Temporarily disable route53 smoke tests (boto#3714)
* release-1.43.16:
  Bumping version to 1.43.16
  Update to latest models
* release-1.43.17:
  Bumping version to 1.43.17
  Update endpoints model
  Update to latest models
  Merge customizations for resiliencehubv2
* release-1.43.18:
  Bumping version to 1.43.18
  Update endpoints model
  Update to latest models
  Merge customizations for QuickSight
  Add a changelog for retries update (boto#3717)
* release-1.43.19:
  Bumping version to 1.43.19
  Update to latest models
* release-1.43.20:
  Bumping version to 1.43.20
  Update to latest models
* release-1.43.21:
  Bumping version to 1.43.21
  Update endpoints model
  Update to latest models
* release-1.43.22:
  Bumping version to 1.43.22
  Update to latest models
  Merge customizations for SocialMessaging
* release-1.43.23:
  Bumping version to 1.43.23
  Update to latest models
* release-1.43.24:
  Bumping version to 1.43.24
  Update endpoints model
  Update to latest models
  Merge customizations for QuickSight
* release-1.43.25:
  Bumping version to 1.43.25
  Update to latest models
* release-1.43.26:
  Bumping version to 1.43.26
  Update endpoints model
  Update to latest models
  Cleanup dead urllib3 import specification in setup.cfg (boto#3723)
* release-1.43.27:
  Bumping version to 1.43.27
  Update to latest models
  Bump github/codeql-action in the github-actions group (boto#3720)
* release-1.43.28:
  Bumping version to 1.43.28
  Update to latest models
* release-1.43.29:
  Bumping version to 1.43.29
  Update endpoints model
  Update to latest models
  Bump https://github.com/astral-sh/ruff-pre-commit (boto#3726)
* release-1.43.30:
  Bumping version to 1.43.30
  Update to latest models
@miketheman miketheman force-pushed the remove-dateutil branch 2 times, most recently from 0c5b530 to d9fc203 Compare June 16, 2026 13:55
Use a helper from the standard library to set up local timezones,
reducing reliance on `dateutil` library

Signed-off-by: Mike Fiedler <miketheman@gmail.com>
Signed-off-by: Mike Fiedler <miketheman@gmail.com>
Mechanical change, removed test shim to improve clarity.

Signed-off-by: Mike Fiedler <miketheman@gmail.com>
Swap dateutil.tz.tzutc for datetime.timezone.utc in credentials, tokens,
and utils. One use in utils.py stays for now — it's inside a
dateutil.parser.parse call and will be handled with a parser refactor.

Refs: https://docs.python.org/3/library/datetime.html#datetime.timezone.utc

Signed-off-by: Mike Fiedler <miketheman@gmail.com>
Replace mock.patch('datetime.datetime', spec=True) with a small
_FrozenDatetime subclass that overrides only now(). Subclassing keeps
fromisoformat / strptime / fromtimestamp callable through inheritance.
Same pattern as tests/unit/test_utils.py's MockDatetime fixture.

Signed-off-by: Mike Fiedler <miketheman@gmail.com>
Replace usages of dateutil.parser in any tests with stdlib equivalent
behaviors. Leave a note about Python 3.11 support change.

Signed-off-by: Mike Fiedler <miketheman@gmail.com>
Replace dateutil.parser.parse with datetime.fromisoformat in
credentials, tokens, and utils. A shared
_normalize_iso8601_for_python_310 helper in botocore.compat handles the
ISO 8601 variants Python 3.10's strict fromisoformat doesn't accept:
Z/UTC/GMT zone suffixes, compact basic form, space date/time separator,
tz offsets without a colon, and fractional seconds with anything other
than 3 or 6 digits. 3.11+ accepts these natively; the helper and its
callers are marked for removal when 3.10 support ends.

_parse_timestamp_with_tzinfo's string fallback also goes through
email.utils.parsedate_to_datetime and a couple of strptime formats for
legacy HTTP Expires shapes. parse_timestamp now raises ValueError for
non-string input instead of leaking AttributeError.

test_timestamp_example's tm_isdst expectation moves from 0 to -1.
stdlib's datetime.timezone.utc returns None from dst() where dateutil's
tzutc returned timedelta(0). The rendered datetime() in shared-example
docs picks up the new value; the output is illustrative, not runnable.

Refs:
  - https://docs.python.org/3/library/datetime.html#datetime.datetime.fromisoformat
  - https://docs.python.org/3/library/email.utils.html#email.utils.parsedate_to_datetime

Signed-off-by: Mike Fiedler <miketheman@gmail.com>
No imports, external API preserved.

Signed-off-by: Mike Fiedler <miketheman@gmail.com>

@SamRemis SamRemis left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Note]

This did pass our all of our test suites when ran locally, including integ tests against live services

Comment thread botocore/utils.py
return email.utils.parsedate_to_datetime(value)
except (TypeError, ValueError):
pass
for fmt in ("%m/%d/%Y", "%d %b %Y"):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Question]

This catches a few non-Smithy shapes but rejects others that dateutil handled. For example, it accepts 06/25/2026 but not 2026/06/25, and 15 Jan 2026 but not January 15, 2026. Why these two formats specifically?

Comment thread botocore/compat.py
@@ -28,7 +28,6 @@

from botocore.vendored import six
from botocore.exceptions import MD5UnavailableError

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Discussion point]

Starting off with my biggest comment:

We have a specification for what accepted timestamp shapes are, so we should only ever need to accept those for parsing. This PR properly handles all officially supported datetime formats, so theoretically this is a fully safe change to make despite handling fewer formats than dateutil.

There are still a few places that I'm still concerned about breaking:

  1. AWS services that are incorrectly returning these values which botocore accepts
  2. Non-AWS third party services relying on botocore's overly permissive behavior

It's a bad thing that botocore is so permissive. We have seen AWS service teams test their service with other datetime formats against botocore and seen it work, then be surprised when the same datetime format broke the Java SDK. If we ever want to tighten up this behavior, we have to rip the bandaid off at some point or fully commit to supporting this forever.

One way we could make this a little more cautiously is to start by using this new code path first and emitting a warning when we have to fall back to the old one, then ripping out the legacy path in three months with a minor version bump. What do you think?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use built-in fromisoformat instead of python-dateutil

4 participants