Support authenticated forward HTTP/HTTPS proxies across the client stack by pditommaso · Pull Request #7331 · nextflow-io/nextflow

pditommaso · 2026-07-13T20:47:46Z

Summary

Behind an authenticating corporate proxy, Nextflow's HTTP paths (plugin downloads, registry calls, SCM, Tower, Wave, data-links) failed with an immediate 407: proxy routing and credentials from HTTP_PROXY / HTTPS_PROXY / ALL_PROXY / NO_PROXY were never propagated to the java.net.http.HttpClient clients Nextflow builds.

This PR threads the parsed proxy configuration into every HTTP client construction site, adds the missing environment-variable handling, and fixes the JDK's HTTPS-tunnelling credential quirk.

Changes

Proxy plumbing (HxClient stack)

Move ProxyConfig from nextflow to nf-commons so plugins can reach it.
Add a static registry + HxProxyConfig view (ProxyConfig.proxyConfig()) resolved from the launch environment.
Launcher.setProxy registers the parsed proxy plus NO_PROXY hosts, so HttpPluginRepository, RegistryClientFactory, RepositoryProvider, SraExplorer, TowerClient, AuthCommandImpl and WaveClient (both the main and token-refresh clients) honour the same routing and credentials.
Clients are configured on HxClient.Builder directly (not via .httpClient(...)), so withProxyConfig(...) isn't a silent no-op and the internal token-refresh clients inherit the proxy too. The now-unused ProxyConfig.configure(HttpClient.Builder) helper is removed.
Bump lib-httpx to 2.4.0 (carries withProxyConfig / HxProxyConfig).

Remaining raw-HttpClient sites in nf-tower

DataLinksResourceHandler: converted from a plain JDK HttpClient to HxClient (no bearer token, so the pre-signed-URL fetch still does not carry the Seqera Authorization header) with withProxyConfig.
TowerXAuth: kept on a plain JDK HttpClient — its token-refresh flow owns a CookieManager and reads the JWT / JWT_REFRESH_TOKEN cookies back out of it, and HxClient.Builder has no cookieHandler() hook. The proxy selector + proxy authenticator are wired explicitly instead (rationale documented inline).

Environment-variable handling (Launcher / ProxyConfig)

ALL_PROXY / all_proxy is used as a fallback when no scheme-specific proxy variable is set (scheme-specific still takes precedence).
ProxyConfig.parse percent-decodes the userinfo user:pass so credentials with special characters (@, :, …) work; a literal + is preserved (userinfo is not form-encoded).

HTTPS CONNECT tunnelling (the core #7305 fix)

The JDK default jdk.http.auth.tunneling.disabledSchemes=Basic strips proxy credentials from the HTTPS CONNECT request, so HTTPS targets behind an authenticating proxy fail with 407 even when the authenticator is wired. Launcher now clears the property when the proxy carries credentials, without ever overriding a value the user set (e.g. via NXF_OPTS).

Testing

Each change lands as a separate commit with unit tests (RED→GREEN): proxy resolution + withProxyConfig wiring, a WireMock integration test proving a request routes through an authenticating proxy, ALL_PROXY fallback, credential URL-decoding, disabledSchemes clearing (including the no-credentials and user-set guardrails), and per-site tests for DataLinksResourceHandler and TowerXAuth.

Not included (follow-ups)

Docs for ALL_PROXY and the NXF_OPTS='-Djdk.http.auth.tunneling.disabledSchemes=' fallback in env-vars.mdx.
An end-to-end HTTPS CONNECT-tunnelling integration test — the current WireMock test deliberately covers only plain-HTTP, so the disabledSchemes fix is unit-covered but not exercised end-to-end.

Behind an authenticating corporate proxy, HxClient-based paths (plugin downloads, registry calls, Tower, Wave) failed with 407 because proxy routing and credentials were never propagated to them. Move ProxyConfig to nf-commons and add static register/configure helpers plus an HxProxyConfig view (proxyConfig()). Launcher.setProxy now registers the parsed proxy and NO_PROXY hosts so every HxClient construction site and WaveClient's token-refresh/main clients honour the same proxy and credentials. Bump lib-httpx to 2.4.0 for the proxy passthroughs. Fixes #7305 Replaces #7306 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

netlify · 2026-07-13T20:47:52Z

✅ Deploy Preview for nextflow-docs ready!

Name	Link
🔨 Latest commit	`0d5c006`
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs/deploys/6a5f7af11eab440008e82ae1
😎 Deploy Preview	https://deploy-preview-7331--nextflow-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

…nfigure Supplying a client verbatim via HxClient.Builder.httpClient(...) makes withProxyConfig a no-op and skips proxy propagation to the internal token-refresh clients. Configure version/redirects/timeout/executor directly on HxClient.Builder instead and apply the proxy with withProxyConfig, so the main and token-refresh clients both route through the proxy. Removes the now-unused ProxyConfig.configure(HttpClient.Builder) helper and stops the local libseqera includeBuild in favour of the published lib-httpx 2.4.0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> # Conflicts: # plugins/nf-seqera/build.gradle

Convert its plain JDK HttpClient to HxClient (no bearer token) and apply the resolved ProxyConfig, so signed-URL fetches honour HTTP_PROXY/HTTPS_PROXY. Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

TowerXAuth keeps its plain JDK HttpClient (cookie-based refresh flow); apply the resolved proxy selector and, when present, the proxy authenticator so the token-refresh call honours HTTP_PROXY/HTTPS_PROXY. Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

- setProxy() falls back to ALL_PROXY/all_proxy when no scheme-specific proxy variable is set (scheme-specific still takes precedence). - ProxyConfig.parse() percent-decodes the userinfo username/password so credentials with special characters work; a literal '+' is preserved. Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

The JDK default (Basic) strips proxy credentials from the HTTPS CONNECT request, so HTTPS targets behind an authenticating proxy fail with 407 even when the authenticator is wired. Clear the property when the proxy carries credentials, without overriding a value the user set via NXF_OPTS. Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

…ing caveat Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

LauncherTest exercises Launcher.setProxy, which populates the static ProxyConfig registry. @RestoreSystemProperties restores system properties but not that registry, so a leaked proxy caused later tests (AssetManagerTest cloning via RepositoryProvider, now proxy-aware) to route through a dead proxy and hang until the CI job timeout. Clear the registry in cleanup(). Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

bentsherman · 2026-07-14T13:58:51Z

Does this PR also resolve #5634 ?

ewels

I'm away from a keyboard so can't test right now, but skimming the code it seems sensible from what I can understand. My buddy Claudio is happy too.

bentsherman

Compared with #5634 and it looks like this PR fixes part of it. So I will just merge this one and submit a follow-up PR with any remaining fixes needed for the older issue

…5634) For Basic proxy authentication the JDK hard-codes "http" as the requesting protocol regardless of the destination scheme (see the Basic branch of sun.net.www.protocol.http.HttpURLConnection#getHttpProxyAuthentication, which passes the string literal "http"; Digest/NTLM instead pass url.getProtocol()). ProxyConfig.authenticator()'s guard therefore never matched for an https_proxy-derived config authenticating an HTTPS CONNECT tunnel (protocol="https" != requesting "http"), silently discarding valid proxy credentials and causing a 407. Relax the guard to also accept an http requesting-protocol when the configured protocol is https (the CONNECT-tunnel case), keeping the host/port/requestor-type checks intact. Adds a regression test driving the real authenticator with the exact field values the JDK supplies for an HTTPS tunnel. This complements #7331 (which cleared jdk.http.auth.tunneling.disabledSchemes so the authenticator is consulted at all); without this fix the consulted authenticator still rejected the credentials. Signed-off-by: Ben Sherman <bentshermann@gmail.com>

pditommaso requested a review from bentsherman July 13, 2026 20:48

pditommaso mentioned this pull request Jul 14, 2026

Add support for authenticated forward HTTP proxies #7306

Closed

pditommaso force-pushed the update-lib-httpx-2.4.0 branch from 766eab4 to 821cb35 Compare July 14, 2026 07:34

pditommaso and others added 6 commits July 14, 2026 09:54

Merge remote-tracking branch 'origin/master' into update-lib-httpx-2.4.0

07e4713

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> # Conflicts: # plugins/nf-seqera/build.gradle

Document why TowerXAuth stays on plain HttpClient

65bb379

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

pditommaso changed the title ~~Support authenticated forward HTTP proxies across the HxClient stack~~ Support authenticated forward HTTP/HTTPS proxies across the client stack Jul 14, 2026

pditommaso requested a review from ewels July 14, 2026 08:20

docs: document ALL_PROXY, proxy credential encoding and HTTPS tunnell…

d8a281d

…ing caveat Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

pditommaso requested a review from a team as a code owner July 14, 2026 08:22

pditommaso mentioned this pull request Jul 14, 2026

Support authenticated forward HTTP proxies across Nextflow's HTTP stack #7305

Closed

3 tasks

ewels approved these changes Jul 20, 2026

View reviewed changes

Merge branch 'master' into update-lib-httpx-2.4.0

dba0f6f

bentsherman approved these changes Jul 21, 2026

View reviewed changes

Merge branch 'master' into update-lib-httpx-2.4.0

0d5c006

bentsherman merged commit 064a038 into master Jul 21, 2026
23 of 24 checks passed

bentsherman deleted the update-lib-httpx-2.4.0 branch July 21, 2026 14:14

bentsherman mentioned this pull request Jul 21, 2026

Accept http requesting-protocol for https proxy CONNECT tunnel auth #7369

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Support authenticated forward HTTP/HTTPS proxies across the client stack#7331

Support authenticated forward HTTP/HTTPS proxies across the client stack#7331
bentsherman merged 12 commits into
masterfrom
update-lib-httpx-2.4.0

pditommaso commented Jul 13, 2026 •

edited

Loading

Uh oh!

netlify Bot commented Jul 13, 2026 •

edited

Loading

Uh oh!

bentsherman commented Jul 14, 2026

Uh oh!

ewels left a comment

Uh oh!

bentsherman left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

pditommaso commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Testing

Not included (follow-ups)

Related

Uh oh!

netlify Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for nextflow-docs ready!

Uh oh!

bentsherman commented Jul 14, 2026

Uh oh!

ewels left a comment

Choose a reason for hiding this comment

Uh oh!

bentsherman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pditommaso commented Jul 13, 2026 •

edited

Loading

netlify Bot commented Jul 13, 2026 •

edited

Loading