Skip to content

fix(sdl): fetch oci modules from public registries through the anonymous token flow - #1480

Merged
Brad-Edwards merged 14 commits into
devfrom
1479-oci-anonymous-token
Nov 11, 2026
Merged

Brad-Edwards merged 14 commits into
devfrom
1479-oci-anonymous-token

Conversation

@doublewhy

@doublewhy doublewhy commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Public registries answer anonymous /v2/ requests with 401 and a Bearer challenge, so oci: imports failed. Now: one token request to the HTTPS realm (no redirect), one retry with the token as an unredirected header, so blob 307s reach storage without it.

Requirement UIDs

  • None. Transport fix with no requirement scope.

Related Issues

Closes #1479

#1484 depends on this PR. Tracked, not delivered here: token refresh (#1542), image-index tags (#1540), live RAES-module and proxy qualification (#1544).

ADR Impact

  • None: under ADR-053 only how an allowed registry is fetched changes.

Changes

Done-when (verbatim) Change Test Lane
Review: "Merge current dev into this branch with real conflict resolution; preserve one bounded challenge/retry, HTTPS-only no-redirect token acquisition, digest/size limits and add_unredirected_header for Authorization." f02583f3 test_rejected_token_gets_exactly_one_retry, test_token_realm_must_be_an_absolute_https_url, test_token_endpoint_redirect_is_not_followed, test_oci_import_rejects_tampered_config_blob, test_oci_bytes_request_rejects_oversized_response, test_bearer_token_never_follows_a_blob_redirect, test_locked_import_fetches_the_pinned_manifest_through_the_token_flow test-shard
Review: "Refresh combined-source captures if synchronization changes the implementation digest; rerun targeted token-flow/module-registry/redirect tests and required CI." ebe04761 - contracts
Review: "Keep #1484 dependent on this resolved branch. Retain explicit tracking of token refresh (#1542), index support (#1540), and live/proxy qualification (#1544) without claiming those are delivered." Related Issues - -
A new test module, implementations/python/tests/test_issue_<this issue>_oci_anonymous_token.py, runs hermetic loopback servers: new module - test-shard
An end-to-end parse_sdl_file import must succeed with one token request, one anonymous attempt and one retry per registry URL, and no Authorization at the storage origin. _registry_get test_import_uses_one_anonymous_token_and_retries_each_request_once test-shard
Further cases: a registry that rejects the token (exactly one retry), an http realm, a redirecting token endpoint, a 401 without a Bearer challenge, two Bearer challenges, an access_token-only response, malformed and oversized token responses, a 401 from a redirect target, and the challenge grammar. The module fails on dev. _anonymous_token, _parse_bearer_challenge 56 of 59 fail without the fix test-shard
Show that the storage-origin test fails when the token is added as an ordinary header. add_unredirected_header redirect, end-to-end, locked test-shard
Run test_sdl_module_registry.py with default markers and with -m integration. - - test-shard, integration
Rerun the reproduction above against ghcr.io, Docker Hub and ECR Public through the production path. - - manual
Run nox -s verify-fast-feedback -- --base-rev origin/dev, nox -s lint, make policy, both research evidence checkers and the three evidence test modules. - - fast-feedback, contracts

Test Plan

  • Unit tests pass
  • Integration tests pass if applicable
  • Full completion suite required in CI before merge

Merged head, pytest on the 7 modules that import raes.module_registry or use oci:: 368 passed; -m integration: 5 passed. The locked test fails with dev's resolution.py (401) and with add_header (storage 403).

Live (2026-10-25): the issue's script stops at module validation on Docker Hub and ECR Public; ghcr.io's index tag answers 404 (#1540). One config blob per registry: 401, one 200 retry, Authorization unredirected, digests matched.

Evidence: collect_research_evidence.py on the merged head (38 of 42 stages passed, 4 preregistered; 20 of 20 formal observations as expected), then research_capture record: capture-917dcfe8 (spec), capture-75bb6056 (formal). Both checkers, the three evidence modules, nox -s contracts, verify-fast-feedback, lint: passed.

Ground Control Checks

  • Repository policy command passes

make policy passed on this head.

Traceability

  • IMPLEMENTS: implementations/python/packages/raes/module_registry/{_registry_auth,__init__,resolution}.py
  • TESTS: implementations/python/tests/test_issue_1479_oci_anonymous_token.py

Checklist

  • Coding standards followed; FM: not applicable (HTTP transport only); no executable or runtime adoption is claimed.
  • No schema or architectural doc change; fix cuts a patch release.

@doublewhy
doublewhy force-pushed the 1479-oci-anonymous-token branch 2 times, most recently from d013e1b to 7dd887f Compare October 10, 2026 17:55
@doublewhy
doublewhy marked this pull request as ready for review October 10, 2026 18:20
doublewhy added a commit that referenced this pull request Oct 12, 2026
…ter entry

Each open entry in the divergence register now names its issue: #1525
to #1557 for the production changes, and #1521 for the smaller
corrections. The entries that open pull requests now address move to
"Addressed by pull requests" with their issues: GH-7 (#1491, #1490),
NS-1 (#1476, #1475), NS-2 (#1480, #1479), MF-2 (#1483, #1482), CL-2
(#1474, #1473), ST-2 (#1472, #1559) and NS-7 and NS-8 (#1488, #1560
and #1561). ST-1 names #1558, GH-3 records that #1415 merged, and
"What remains" names #1522 for the unverified and unexamined
boundaries.
@doublewhy
doublewhy force-pushed the 1479-oci-anonymous-token branch from 7dd887f to 7f1db16 Compare October 13, 2026 10:08
@doublewhy
doublewhy marked this pull request as draft October 13, 2026 19:39
…ous token flow

ghcr.io, Docker Hub and ECR Public answer an anonymous registry request
with 401 and a WWW-Authenticate Bearer challenge, and the module
registry reported every such import as a fetch failure.

A 401 from the requested URL that carries exactly one Bearer challenge
is now answered with one anonymous token request to the challenge's
HTTPS realm (no redirects followed) and one retry. The challenge is
parsed strictly, tokens are cached per realm, service and scope for one
import resolution, and the token is attached as an unredirected header,
so urllib never sends it on the 307 to a registry's blob storage.
Timeouts, size limits, digests and the urlopen seam are unchanged.
…quested url

A realm whose host is percent-encoded or not IDNA-encodable passed the
realm check and then escaped parse_sdl_file as http.client.InvalidURL
or UnicodeEncodeError. The realm check now refuses a "%" in the
authority and any host the IDNA codec rejects, so such a challenge
fails with an SDLParseError before any token request.

The token response's oversize message named the token URL built from
the challenge's realm, service and scope. It now names the requested
URL, like every other message in the flow.

A second parse_sdl_file import against the same registry now pins the
one-cache-per-import token lifetime.
…nder the line cap

Dev's #1426 grew resolution.py to 492 lines, so the token threading put it at
501, over the ADR-015 500-line cap. Formatting only.
Collected with blitz-tools/collect_research_evidence.py on this branch's source (6 pinned artifacts through the production pipeline, 10 retained cases via replay_case, 4 production processor cases, 14 participant functions), then recorded with python -m tools.research_capture record.
@doublewhy
doublewhy force-pushed the 1479-oci-anonymous-token branch from 7f1db16 to deaed7d Compare October 13, 2026 21:47
@doublewhy
doublewhy marked this pull request as ready for review October 14, 2026 03:38
@Brad-Edwards

Copy link
Copy Markdown
Collaborator

Recommendation: do not merge as is.

Reviewed head deaed7d2b9fc3146f2593c02784f0150a51b4eac, targeting dev.

  1. Merge conflicts against dev (DIRTY). The security-sensitive module-registry facade/resolver must be reviewed and tested after actual conflict resolution.

Required action:

Coverage and outstanding work: #1479's anonymous flow, access_token compatibility, challenge rejection, token bounds and redirect non-disclosure cases are represented. The use of urllib's unredirected header is appropriate. No additional confirmed source defect identified in the reviewed patch.

Verification limits: Green head checks and complete snapshot do not validate the source resulting from current-dev conflict resolution. Live full RAES-module/public-registry and proxy coverage is explicitly limited.

Conflict in module_registry/resolution.py (imports only): dev moved
_select_tag into _digests and dropped _normalize_exact_or_range from
resolution; this branch adds _RegistryTokens. Kept dev's _digests import
and the _registry_auth import. Dev's editor path mode refuses oci: before
any registry request, and its iterative locked: chain ends in the same
_resolve_oci_import, which creates one _RegistryTokens per import.
Dev's locked: chain resolver (#1339) reaches _resolve_oci_import with the
manifest pinned by digest, so no tags/list request precedes the challenge.
Cover one and two locked hops: one token request, one retry per registry
URL, and no Authorization at the storage origin.
Collected on the merge of current dev with blitz-tools'
collect_research_evidence.py and recorded with research_capture record:
specification capture-917dcfe8, formal capture-75bb6056.
@doublewhy

Copy link
Copy Markdown
Contributor Author

Review of 2026-10-25 addressed on head ebe04761:

  1. Merge commit f02583f3 (no rebase). One conflict: the _digests import in module_registry/resolution.py (feat!: make reference eligibility explicit per purpose and align the editor with validation #1446 moved _select_tag); kept dev's import plus _RegistryTokens. _registry_auth.py unchanged. Single retry: test_rejected_token_gets_exactly_one_retry. HTTPS-only, no-redirect token: test_token_realm_must_be_an_absolute_https_url, test_token_endpoint_redirect_is_not_followed. Digest/size: test_oci_import_rejects_tampered_config_blob, test_oci_bytes_request_rejects_oversized_response. Unredirected header: test_bearer_token_never_follows_a_blob_redirect. Dev's locked: chain into the token flow: new test_locked_import_fetches_the_pinned_manifest_through_the_token_flow (eb32f9f1).
  2. Digest changed; combined-source captures capture-917dcfe8 (spec) and capture-75bb6056 (formal), collected on the merged head. Targeted tests: 368 passed, 5 integration. CI run 40178997258 succeeded; 28 checks pass (deploy skipped).
  3. fix(sdl): report invalid import declarations, provenance and directory imports as sdl diagnostics #1484 stays dependent on this branch; OCI imports do not refresh expired registry tokens #1542, OCI imports fail when the tag is an image index #1540 and Verify OCI imports against a live public registry #1544 stay open and are listed as not delivered.

@doublewhy

Copy link
Copy Markdown
Contributor Author

This branch conflicts with dev again after the Quality refactor merges, in module_registry/. The fix is not on dev: the registry client still has no token-challenge handling, so #1479 still reproduces. #1718 (refactor 08) keeps #1479's feature out of that chunk. Should this land before #1718 restructures module_registry/, or after it on the new layout? I'll merge dev again only for the option you pick.

@Brad-Edwards
Brad-Edwards merged commit 3fa7a6d into dev Nov 11, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants