Re-evaluate NO_PROXY per-request so it is honored across redirects

[fl0Lec]

Problem

URLLib3Session's proxy/no-proxy decision is currently frozen at session-construction time. EndpointCreator._get_proxies calls get_environ_proxies(endpoint_url), which in turn filters getproxies() through should_bypass_proxies against the initial endpoint URL. If NO_PROXY matches that URL, the session is given an empty proxy dict and never consults a proxy again for any subsequent request through that Endpoint — even if the request URL changes.

The symptom shows up most often on S3 cross-region redirects:

1. A pod in ap-northeast-1 constructs boto3.client('s3') without an explicit region.
2. The first request goes to s3.ap-northeast-1.amazonaws.com. With a typical NO_PROXY=*.s3.ap-northeast-1.amazonaws.com, that request correctly goes direct.
3. S3 returns 301 PermanentRedirect to <bucket>.s3.us-east-1.amazonaws.com.
4. S3RegionRedirectorv2.redirect_from_error (botocore/utils.py:1757) rewrites request_dict['url'] and re-issues through the same Endpoint / URLLib3Session. The us-east-1 URL does not match NO_PROXY and should go through the corporate proxy — but the session's proxy dict was frozen empty at step 2, so the redirect also bypasses the proxy and gets blocked by corporate egress policy.

The same pattern affects any 301/302/307/308 redirect that crosses host boundaries, and the inverse case (initial URL routes through the proxy, redirect target matches NO_PROXY and should go direct) suffers from a related staleness — the cached proxy decision is wrong for the new URL.

Related issues

Cross-cutting NO_PROXY / proxy bugs that point at the same root cause or the same surface:

• Is the current NO_PROXY situation permanent? boto3#4380 — "Is the current NO_PROXY situation permanent?" (acknowledged NO_PROXY weakness)
• s3 client global endpoint_url results in PermanentRedirect when region provided boto3#4360 — s3 client global endpoint_url results in PermanentRedirect
• proxy option in botocore.config.Config does not mirror environment variable behavior. boto3#2420 — Config-proxy vs env-var mismatch (closed stale)
• Default Proxy Config is not respected #2707 — Default Proxy Config is not respected (confirmed bug, open)
• Proxy configuration is not respected when fetching credentials from IMDS endpoint #2644 — Proxy not respected for IMDS endpoint
• When using a proxy with SSL connections redirects do not work boto#926 — Legacy boto: when using a proxy with SSL connections redirects do not work
• S3 buckets are global and shall not bother clients with region_name after creation boto3#145 — S3 buckets are global and shall not bother clients with region_name (open since 2014; this PR fixes the symptom even when the underlying request hits the wrong region first)
• NO_PROXY not considered in boto lambda local invocation (sam local-start) boto3#3179 — NO_PROXY not considered in lambda local invocation (closed stale)

Fix

Defer the NO_PROXY decision from session construction to send time:

• botocore.utils.get_environ_proxies grows a no_proxy_filter parameter (default True, preserving the previous signature). When False, it returns the raw environment proxies without applying should_bypass_proxies upfront.
• botocore.endpoint.EndpointCreator._get_proxies now calls get_environ_proxies(url, no_proxy_filter=False) so the URLLib3Session always sees the configured proxies, regardless of whether the initial URL would have matched NO_PROXY.
• botocore.httpsession.ProxyConfiguration.proxy_url_for calls should_bypass_proxies on the URL it is given, returning None when NO_PROXY matches. This re-evaluates per request, so the decision follows the actual URL being sent — including redirect targets.

The bypass check now applies uniformly to env-sourced and Config(proxies=...) proxies. This is consistent with how requests already behaves and also addresses the long-standing complaint in #2707 that NO_PROXY is not honored for explicitly configured proxies.

Diff summary

File	Change
botocore/endpoint.py	_get_proxies passes no_proxy_filter=False
botocore/httpsession.py	ProxyConfiguration.proxy_url_for calls should_bypass_proxies per request (lazy import to avoid a circular import with botocore.utils)
botocore/utils.py	get_environ_proxies grows no_proxy_filter=True kwarg
tests/unit/test_http_session.py	3 new unit regression tests
tests/functional/test_proxy_redirect.py	New end-to-end integration test with real local HTTP + proxy servers
.changes/next-release/	Changelog entry

Tests

Existing suite

All existing unit tests pass (4588 passed, 95 skipped, 0 regressions). The proxy/endpoint-touching subset of functional tests (-k "proxy or redirect or endpoint") passes cleanly: 18,147 passed, 20 skipped, 0 failed.

New regression tests

Three new tests in tests/unit/test_http_session.py:

1. TestProxyConfiguration.test_proxy_url_for_honors_no_proxy_per_call — ProxyConfiguration.proxy_url_for returns None for a URL matching NO_PROXY and the proxy URL for one that does not, even though both share the same ProxyConfiguration instance.
2. TestURLLib3Session.test_no_proxy_env_var_bypasses_proxy_per_request — first request to a URL that matches NO_PROXY goes through the non-proxy pool manager; a subsequent request through the same session to a URL that does not match NO_PROXY goes through the proxy manager (this is the cross-region redirect case).
3. TestURLLib3Session.test_no_proxy_env_var_matches_redirect_target — inverse direction: initial request goes through the proxy, a subsequent request to a URL matching NO_PROXY goes direct.

One new end-to-end test in tests/functional/test_proxy_redirect.py:

4. TestNoProxyAcrossRedirect.test_no_proxy_re_evaluated_across_sequential_requests — spins up two real local HTTP servers (one acting as the backend, one acting as the proxy) on dynamically-allocated ports, then makes two sequential URLLib3Session.send() calls through the same session to URLs whose hosts differ only in name (127.0.0.1 vs localhost). With NO_PROXY=127.0.0.1 set, the first request must hit the backend directly and the second must hit the proxy. The test asserts the correct server received each request, demonstrating the fix end-to-end through real socket I/O. Hermetic (loopback only), deterministic, and completes in ≈0.15s.

Each new test was verified to fail without the fix and pass with it by reverting the production diff and re-running.

Backward compatibility

• No public API change — the only signature touched (get_environ_proxies) gains a kwarg with a default that preserves the prior behavior for any external caller.
• For users who rely on the default proxy resolution (env-only): NO_PROXY now reliably follows the actual request URL. The only observable behavior change is that the redirect target is now correctly evaluated against NO_PROXY, which is the fix.
• For users who pass Config(proxies=...) explicitly: NO_PROXY env now also gates that proxy, matching requests semantics and resolving Default Proxy Config is not respected #2707. Users who set Config(proxies=...) but want to ignore NO_PROXY would need to unset NO_PROXY — but no observable historical caller relies on NO_PROXY being silently ignored.
• IMDS path (botocore.utils.IMDSFetcher, line 425) was intentionally left untouched — it still uses no_proxy_filter=True (the default). Fixing Proxy configuration is not respected when fetching credentials from IMDS endpoint #2644 would be a follow-up.

Acknowledgment of prior unmerged proxy PRs

I understand the maintainers have historically declined to broaden proxy support — most notably #2541 (SOCKS) was closed as "not under consideration." This PR is intentionally different:

• Scoped: fixes a confirmed bug, does not add a new proxy protocol or capability.
• Minimal: production-code diff is 16 added lines across 3 files (excluding docstrings and comments). No new dependencies, no new public API.
• Backward compatible: get_environ_proxies kwarg default preserves prior behavior for external callers.
• Well covered: three unit regression tests + one end-to-end integration test, each verified to fail without the fix.
• Addresses multiple open issues at once: the same minimal change unblocks #4360, #4380, Default Proxy Config is not respected #2707, and partially Provide caller with information on raised exception after retry #926.

Happy to iterate on the approach — alternatives considered include applying should_bypass_proxies only in URLLib3Session.send() (rejected: leaves _get_proxies returning {} for env-bypass cases, so explicit-proxy semantics diverge), or adding a new opt-in flag (rejected: defeats the purpose of fixing the bug for the common case).