Skip to content

fix(api): refuse an upstream host that no sync run resolves as given - #903

Merged
ThomasK33 merged 8 commits into
mainfrom
upstream-host-shape
Nov 17, 2026
Merged

ThomasK33 merged 8 commits into
mainfrom
upstream-host-shape

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

POST /repos stored upstream hosts that the registration check could not vet and that no sync run could use as given. The create answered 201, and the trouble came later, run by run:

Host (upstream.host or base_repo.upstream_host) Before Now
A name that Go's resolver refuses before it asks DNS: <<<.com, a..b.example, a label of 64 bytes, a name of 254 bytes 201: the registration check failed open, and every run failed 400 is not a valid host
An IPv4 literal with a zone: https://1.2.3.4%25eth0 201, as above 400
A number form that curl reads as an IPv4 address: 2130706433, 1.2.3, 01.2.3.4, 0x7f000001, 0x7f.1, 1.2.3.0x4 201: the registration check read it as a name and failed open; curl dials 127.0.0.1, 1.2.0.3, 1.2.3.4, 127.0.0.1, 127.0.0.1 and 1.2.3.4, an address that the check never saw (the dial-time guard still vets it) 400
A port outside 1-65535: forge.example:0, forge.example:65536 201 400
A base URL with a query or fragment, even an empty one: https://forge.example?, https://forge.example/git#x 201: the fetch URL https://forge.example?/o/n.git asks the wrong path 400
A base URL with a space (https://forge.example/a b) or a scheme that is not lower case (HTTPS://forge.example) 201: curl refuses the space; GIT_ALLOW_PROTOCOL refuses HTTPS, and the worker's https:// prefix check would not route it through the egress proxy 400

validUpstreamHost (internal/api/handlers_sync.go) now also requires a dialable authority (dialableAuthority, dialableHostname): an IP address without a zone, or a name that passes the rule of Go's resolver, net.isDomainName (resolverName: at most 253 bytes, or 254 with a final dot; labels of 1 to 63 letters, digits, _ and - that do not start or end with -; not only digits and dots) and that curl does not read as an IPv4 address (curlIPv4Name: one to four labels, a final dot aside, each a decimal, octal or hexadecimal number of at most 32 bits, all but the last at most 255 and the last within the bits the others leave, as curl's ipv4_normalize reads them; 0x100.1 and 0x100000000 stay names); a port from 1 to 65535, or an empty one (the default); and, in a base URL, a lower-case http:// or https:// and no ?, # or space. A sync run resolves through the egress proxy, whose Go resolver refuses any other name. The . root name, which Go accepts, is refused: it names no host.

A non-ASCII name such as bücher.example → 400; give its ASCII form, xn--bcher-kva.example. Go's resolver refuses the Unicode form, so the registration check cannot look it up, and whether git and curl convert it to the ASCII form depends on the locale that the server's git runs in. This is a change for a deployment where that conversion worked: such a create answered 201 before.

Hosts that keep working include forge.example, forge.example., a_b.example, 1-1, xn--bcher-kva.example, localhost, 0x1.0x2.0x3.0x4.0x5 and 0x100.1 (names for curl), 10.0.0.1:1, [::1]:65535, forge.example:, https://forge.example/sub/path/, https://[2001:db8::1]:443/git and https://forge.example/a%3Fb.

The docs (docs/reference/http-api.md, the host bullet) and internal/api/README.md state the rule.

Evidence

  • Tests: TestValidUpstreamHost lists 40 accepted and 54 refused hosts, including the boundaries (a 63-byte label first and last, a 253-byte name, ports 1 and 65535, five numeric labels). TestSyncCreateRefusesUndialableHosts sends seven of them in both host fields: 400 is not a valid host naming the field, nothing stored.
  • Mutants: 18 overlay mutants all fail the two tests: no port check, no ?# check, no space check, an upper-case scheme admitted, numeric names admitted, a last label of 64 bytes admitted, one of 63 refused, - or _ not counted as a letter, no label limit, no length limit, a zone admitted, no curlIPv4Name, three labels instead of four, no hex, only lower-case 0x, any number of labels, a label that starts with -.
  • curl 8.22.0: curl -v http://2130706433:9/ tries 127.0.0.1:9; 1.2.3 tries 1.2.0.3; 01.2.3.4 tries 1.2.3.4; <<<.com is rejected as a bad hostname. The review found that 0x7f000001 and 0x7f.1 go to 127.0.0.1, and that bücher.example is converted to xn--bcher-kva.example under a UTF-8 locale but refused in the C locale.
  • Lanes: go test -race -tags=openstorage_viewcheck ./internal/api/... without pg passes at 2b1a6a1 (388 s), at bfffd09 (349 s) and at the merge 2e98dad (359 s). The sync worker is unchanged. PostgreSQL and S3-WAL rely on CI.
  • lint, vet (linux and darwin), gofmt, the docs gate and both fingerprint gates pass. handlers_sync.go is covered by the lane-schedule model, so its fingerprints are re-recorded (c5b0d36, aeba00c, b8dfa9d).

Reviews

  • Round 1 (2b1a6a1): the Codex security review found nothing. The Codex review raised two P2s, and a fresh read-only review raised three, four P3s and a differential test of resolverName against Go's isDomainName (6.7M exhaustive names, 2M random, 60 s of fuzzing: no difference but .):
    • Hex IPv4 forms (0x7f000001) passed (Codex and the review): bfffd09 adds curlIPv4Name.
    • "curl refuses bücher.example" held only in the C locale (review), and Codex asked to keep IDN hosts working. bfffd09 keeps the refusal with the reason that holds everywhere (above), and the docs say to give the ASCII form; keeping Unicode names would need IDNA conversion for the registration check (a direct golang.org/x/net/idna dependency) and would still depend on the server's locale.
    • Four mutants survived (review): bfffd09 adds the boundary cases.
    • P3s fixed in bfffd09: an upper-case scheme and a space in a base URL are refused, the docs say that an empty port is the default and that a name may have 254 bytes with a final dot, and internal/api/README.md states the rule.
  • Round 2 (aeba00c): the Codex security review found nothing. The Codex review (P2) found that names which only look numeric (0x100.1, 0x100000000) were refused although curl keeps them names, and a fresh review found two test gaps (upper-case hex digits, 64-byte labels of digits or dashes) and three P3s. 78f3a1a parses each label as curl does, with IPv4's ranges, and agrees with curl 8.22 on 33 forms (0x7f.1., 0377.0377.0377.0377, 0xff.0xffffff read as addresses; 0400.1.1.1, 0xff.0x1000000, 4294967296 read as names); the tests add those boundaries, and the docs name the number forms and Go's all-digits rule. The first table row no longer lists bücher.example as failing every run. 8 more overlay mutants: 7 killed, and the 64-bit parse is equivalent (the range checks cut the same values).
  • Round 3: 2e98dad merges main (fix(githttp): refuse a ref at an object that no registered pack holds #896, fix(events): deny every cloud metadata endpoint under every egress policy #901, fix(api): cut the webhook URL and the ref policy op that a refusal quotes #902). fix(api): cut the webhook URL and the ref policy op that a refusal quotes #902 dropped the strconv import of handlers_sync.go with quoteValue, and this branch needs it, so the merge keeps it; b8dfa9d re-records the lane-schedule fingerprints on the merged tree.
  • Round 3 review (b8dfa9d): the Codex security review found nothing. A fresh read-only review: PASS. It compared curlIPv4Name with libcurl 8.22 (the one git links) on about 4.3 million names, exhaustive over the listed labels in one to five labels with and without a final dot, plus random and boundary names, and through git and a capture proxy: no difference. curl 8.18 keeps a final-dot or 0X form a name where the PR refuses it, which is over-refusal of all-numeric names only. Its 66 mutants: 59 killed, 6 equivalent. The Codex review (P2) said [forge.example] passes; Go 1.27.4's url.Parse (the version go.mod and the release image pin) refuses a bracketed host that is not IPv6 (invalid host, invalid IP-literal), so validUpstreamHost never reaches the name. All six Codex requests of this PR are used.
  • Left open (P3): a name that only /etc/hosts holds and that Go's DNS rule refuses (such as one with ~) is refused, although the run's resolver reads /etc/hosts first. Older links stored with any of these hosts keep failing on each run; nothing rewrites them.

Found by the reviews of #900 (desk follow-up).


Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: xhigh

POST /repos stored upstream hosts that the registration check could not
vet and that the run could not use as given:

- a hostname that Go's resolver refuses before it asks DNS, such as
  `<<<.com` or an IPv4 literal with a zone (`https://1.2.3.4%25eth0`):
  the courtesy check failed open, the create answered 201, and every
  run failed;
- a name of only digits and dots that is no IP address, such as
  `2130706433`, `1.2.3` or `01.2.3.4`: curl reads them as 127.0.0.1,
  1.2.0.3 and 1.2.3.4, so the run dialed an address that the
  registration check never saw (the dial-time guard still vets it);
- a port outside 1-65535;
- a base URL with a query or a fragment, even an empty one
  (`https://h?`), which ends the path of every fetch URL.

validUpstreamHost now also requires a dialable authority: an IP address
without a zone, or a name that passes the rule of Go's resolver
(net.isDomainName: at most 253 bytes, labels of 1 to 63 letters, digits,
'_' and '-', not only digits and dots); a port from 1 to 65535; and no
'?' or '#' in a base URL. Such a host answers 400 `is not a valid host`
in both host fields, as any other invalid host does.

Signed-off-by: Thomas Kosiewski <tk@coder.com>

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_

Change-Id: I962873fa0be93de3eb41a79ac0805a745e5200a2
Signed-off-by: Thomas Kosiewski <tk@coder.com>
handlers_sync.go is covered code: validUpstreamHost and the new
dialableAuthority and dialableHostname.

Signed-off-by: Thomas Kosiewski <tk@coder.com>

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_

Change-Id: I5138f4324d2d709f49fab59b06c14a9308a4d86c
Signed-off-by: Thomas Kosiewski <tk@coder.com>
curl refuses bücher.example as a bad URL, so the host rule says so, and
the dialableHostname comment says what bounds a zone the egress answer
would repeat.

Signed-off-by: Thomas Kosiewski <tk@coder.com>

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_

Change-Id: I410340caebd88cf2c0996dbc8b4c69cfbbfb51c3
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-17T00:07:19.470324Z b8dfa9d Manual request
🔒 Security Review ✅ Completed 2026-11-17T00:03:33.354944Z b8dfa9d Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 2b1a6a1f4a

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2b1a6a1f4a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/api/handlers_sync.go Resolved
Comment thread internal/api/handlers_sync.go Resolved
curl reads a host such as 0x7f000001, 0x7f.1 or 1.2.3.0x4 as an IPv4
address (127.0.0.1, 1.2.3.4), as it does 2130706433, but the letter x
let it pass the resolver rule. curlIPv4Name refuses a name of one to four
labels that are each a decimal, octal or hexadecimal number.

A base URL must start with `http://` or `https://` in lower case
(GIT_ALLOW_PROTOCOL refuses `HTTPS`, and the worker would not route it
through the egress proxy) and hold no space, which curl refuses.

The non-ASCII rule now gives the reason that holds everywhere: the
registration check cannot look such a name up, and whether git and curl
convert it depends on the locale of the server's git. The docs list the
rules, say that an empty port is the default, and that a name may have
254 bytes with a final dot. Tests add the boundary cases that four
mutants survived.

Signed-off-by: Thomas Kosiewski <tk@coder.com>

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_

Change-Id: I07e32b3cc42a2039da7d52b8eb3b7948205bc89e
Signed-off-by: Thomas Kosiewski <tk@coder.com>
handlers_sync.go is covered code: validUpstreamHost, dialableHostname,
and the new resolverName and curlIPv4Name.

Signed-off-by: Thomas Kosiewski <tk@coder.com>

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_

Change-Id: If37f3da4f5d67fdab58a35b91559b468a662832d
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: aeba00c736

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: aeba00c736

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/api/handlers_sync.go Resolved
curlIPv4Name refused any one to four labels made of digits or 0x hex,
so names that only look numeric (0x100.1, 0x100000000) answered 400
although curl keeps them names. It now parses each label as curl's URL
parser does: decimal, octal (leading 0) or hexadecimal (0x, 0X), at most
32 bits, all but the last at most 255 and the last within the bits the
others leave. Against curl 8.22 it agrees on 33 forms, from 0x7f.1. and
0377.0377.0377.0377 to 0400.1.1.1 and 4294967296.

Tests add upper-case hex digits, the range boundaries, names that only
look numeric, and 64-byte labels of digits and of dashes; the docs say
which number forms are refused.

Signed-off-by: Thomas Kosiewski <tk@coder.com>

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_

Change-Id: Idf4ed0893728a8e3c43baba6eda5a5647f7eecad
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Main gained #896, #901 and #902. #902 dropped the strconv import of
handlers_sync.go with quoteValue; this branch's port and IPv4 checks use
it, so the merge keeps it. The other conflicts are the lane-schedule
fingerprints and sync commit; the next commit re-records them on the
merged tree.

Signed-off-by: Thomas Kosiewski <tk@coder.com>

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_

Change-Id: Ib2c61d6a44b3a74698bce02f2eef91f7fe0568a3
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Signed-off-by: Thomas Kosiewski <tk@coder.com>

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_

Change-Id: Ib69e3836b0cd3da64045e0fdff26182c2c9bec88
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: b8dfa9daff

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b8dfa9daff

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/api/handlers_sync.go Resolved
@ThomasK33
ThomasK33 added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit 324dd32 Nov 17, 2026
16 of 17 checks passed
@ThomasK33
ThomasK33 deleted the upstream-host-shape branch November 17, 2026 00:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant