Skip to content

test(api): guard the request types of unmarshalBody, stop its loop at no progress - #899

Open
ThomasK33 wants to merge 2 commits into
mainfrom
decode-request-type-guard
Open

ThomasK33 wants to merge 2 commits into
mainfrom
decode-request-type-guard

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Why

#883 taught the walk of unmarshalBody to name more members (map keys, embedded fields, struct tags). Its rounds of review drew 5, 3 and 7 findings and did not converge, so this PR replaces it. unmarshalBody finds the member that failed by the dotted path that encoding/json reports. That path is right for every request type that main decodes with it: they hold scalars, structs, pointers, slices, json.RawMessage and (for branches/create) one self-decoding type, and nothing else. This PR keeps it that way instead of widening the walk.

Two gaps remain around that fact:

  1. Nothing fails when a request type gains a field that the path is wrong for. Measured on main (dc6ef0cf, Go 1.27.4): a map key that holds a dot, a map key that is a path escape (Go 1.27 reports a/b as a~1b), a map[int]int key that does not decode, and a type that refuses null each leave the loop with a member that nulling does not clear, or with none.
  2. In those cases the loop does not stop. It nulls a member that is not the failing one (or the failing one, which stays an error), decodes again, gets the same error and repeats until it holds 100 problems or the 256 KiB work budget is spent. The answer is the one error (the 100 copies are one name), but the cost is 100 decodes: 0.64 ms for a body of 50 bytes and 2.9 ms for one of 2 KiB, against 13 µs and 26 µs here.

No request type of main reaches either gap today.

What changed

  • unmarshalBody (internal/api/decode_body.go) stops when a pass leaves the body unchanged (the member is null already, or nulling it is a no-op) and when a decode names a field that an earlier pass named. Neither changes an answer: the 100 copies were one problem. A body with one wrong-type field still returns the error of encoding/json itself. The doc comment says what the path is right for and points at the guard test.
  • TestRequestTypesStayInsideTheDecodeWalk (new, internal/api/decode_walk_guard_test.go) walks every request type that the server decodes with unmarshalBody (16 types) and fails for: a map, an interface, an embedded field, a member name outside [A-Za-z0-9_], two names of one struct that differ only in case, a tag option other than omitempty, omitzero and string, a type with its own UnmarshalJSON or UnmarshalText that is not on a two-entry list (json.RawMessage, wrefCreateBranchRequest, each with the test that shows the walk right for it), and any kind that is no JSON type.
  • The same test reads the syntax of the package (go/parser) and fails when a function that is not in decodeWalkSites names unmarshalBody, or names it a different number of times. So a new endpoint, or a new decode in an old one, must be listed with its request types before the test passes. A function that takes its destination as a parameter of type any, interface{} or a type parameter is a helper: it is listed with no types, and its name then joins the scan, so that its callers must be listed with theirs (the three helpers of main, wrefDecodeBody, wrefDecodeOptionalBody and gcredDecodeBody, are listed so). A helper that is listed with types, or a typed function listed without, fails. A Codex review of the first head found that a new helper hid its callers; this closes it.
  • TestScanWalkSites checks the scan on a synthetic source (a typed handler, a decode through a value, a helper with each kind of destination parameter, a method helper, a caller of a helper). TestDecodeWalkProblems checks the checker: 20 synthetic types, each kind that it must flag (at any depth through pointers, slices and arrays) and each shape that it must pass. TestDecodeWalkHandlesWhatTheGuardAllows decodes a type with every allowed shape and a wrong value in each, and requires that unmarshalBody names every field, one decode each.
  • TestUnmarshalBodyStopsWithoutProgress covers the loop: the four cases above plus a type that refuses a null it was sent.

Scope

Test code and the loop of one function. No answer changes for any body that a request type of main can send; the status codes, the messages and invalid_params are as on main. AGENTS.md and the docs are unchanged: the message of a failing guard says what to do, and the comment of unmarshalBody says what the walk is right for.

Design calls

  • The guard is a test, not a hook in unmarshalBody. A runtime hook would check only the types that some test sends. The syntax scan sees every function. Its limits: it does not see the type of the destination, so a handler that swaps its request type and keeps the same number of decodes is not seen; and a helper that takes a destination in another form (a func(any) error value, a reflect.Value) is not found as a helper. golang.org/x/tools (go/packages, which would see the types) is not vendored.
  • The walk is not extended. That was fix(api): find map keys with dots, slashes or tildes in unmarshalBody #883. A type outside the shapes above gets its own decode (as strictJSON does for a body with unknown members), after a person has read what unmarshalBody would do with it.
  • Embedded fields fail the guard although the walk names them today. I checked on Go 1.27.4 that a member of an embedded struct is reported as a, with no embedded name in the path. The guard still fails them, because Go's rules for which promoted member wins a name are not modelled by the name checks, and the walk nulls by name. An allowed embedded field would need those checks first. Say so in review if you prefer to allow them now.
  • string is allowed (json:"quoted,string"): checked on Go 1.27.4, the error keeps the field path, and the allowed-shapes test decodes one.
  • Stop rule, not a better search. The walk could find the failing member through the type, but that is fix(api): find map keys with dots, slashes or tildes in unmarshalBody #883. The stop keeps the cost of a wrong path at 2 decodes and the answer at what it was.

Verification

  • Against main: go test -overlay with main's decode_body.go fails the five cases of TestUnmarshalBodyStopsWithoutProgress with 100 decodes each; with this head they take 2 (1 for a member that is null already). Mutants: no bytes.Equal check fails the null case; no named check fails the other four (the answer is not the lone encoding/json error).
  • Guard mutants on the real types: a map field in wmMergeRequest, an embedded struct in createWebhookRequest, a map under usyncBaseRepo.Auth, a new function that calls unmarshalBody, and a second decode in handleMerge each fail TestRequestTypesStayInsideTheDecodeWalk with a message that names the type or function. A new helper with dst any fails until it is listed; once listed with no types, its new caller fails until it is listed; once listed with a map type, the map fails. Eight mutants of the scan (the helper rule both ways, the interface, any and type-parameter checks, the receiver strip with and without a receiver, the place count) each fail TestScanWalkSites or the inventory test. 14 mutants of walkProblems (each rule, the pointer step, the slice step, the struct step, the unexported and - skips) each fail TestDecodeWalkProblems; the one that removes the cycle check runs out of memory instead of passing.
  • The SQLite lane go test ./internal/api/... -tags=openstorage_viewcheck -count=1 -race -skip '/^pg$|//^pg$' passes on this head (342.7 s, Go 1.27.4). golangci-lint: 0 issues. Docs gate: 0 broken links, 80 script tests pass. Both fingerprint gates pass. The PostgreSQL lane runs in CI.

Generated with xum • Model: anthropic:claude-sonnet-5-5 • Thinking: xhigh

… no progress

Two gaps remain around that fact:

1. Nothing fails when a request type gains a field that the path is wrong for. Measured on main (`dc6ef0cf`, Go 1.27.4): a map key that holds a dot, a map key that is a path escape (Go 1.27 reports `a/b` as `a~1b`), a `map[int]int` key that does not decode, and a type that refuses null each leave the loop with a member that nulling does not clear, or with none.
2. In those cases the loop does not stop. It nulls a member that is not the failing one (or the failing one, which stays an error), decodes again, gets the same error and repeats until it holds 100 problems or the 256 KiB work budget is spent. The answer is the one error (the 100 copies are one name), but the cost is 100 decodes: 0.64 ms for a body of 50 bytes and 2.9 ms for one of 2 KiB, against 13 µs and 26 µs here.

No request type of main reaches either gap today.

- `unmarshalBody` (`internal/api/decode_body.go`) stops when a pass leaves the body unchanged (the member is null already, or nulling it is a no-op) and when a decode names a field that an earlier pass named. Neither changes an answer: the 100 copies were one problem. A body with one wrong-type field still returns the error of `encoding/json` itself. The doc comment says what the path is right for and points at the guard test.
- `TestRequestTypesStayInsideTheDecodeWalk` (new, `internal/api/decode_walk_guard_test.go`) walks every request type that the server decodes with `unmarshalBody` (16 types) and fails for: a map, an interface, an embedded field, a member name outside `[A-Za-z0-9_]`, two names of one struct that differ only in case, a tag option other than `omitempty`, `omitzero` and `string`, a type with its own `UnmarshalJSON` or `UnmarshalText` that is not on a two-entry list (`json.RawMessage`, `wrefCreateBranchRequest`, each with the test that shows the walk right for it), and any kind that is no JSON type.
- The same test reads the syntax of the package (`go/parser`) and fails when a function that is not in `decodeWalkSites` names `unmarshalBody` or one of its three helpers, or names it a different number of times. So a new endpoint, or a new decode in an old one, must be listed with its request types before the test passes.
- `TestDecodeWalkProblems` checks the checker: 20 synthetic types, each kind that it must flag (at any depth through pointers, slices and arrays) and each shape that it must pass. `TestDecodeWalkHandlesWhatTheGuardAllows` decodes a type with every allowed shape and a wrong value in each, and requires that `unmarshalBody` names every field, one decode each.
- `TestUnmarshalBodyStopsWithoutProgress` covers the loop: the four cases above plus a type that refuses a null it was sent.

Test code and the loop of one function. No answer changes for any body that a request type of main can send; the status codes, the messages and `invalid_params` are as on main. `AGENTS.md` and the docs are unchanged: the message of a failing guard says what to do, and the comment of `unmarshalBody` says what the walk is right for.

- **The guard is a test, not a hook in `unmarshalBody`.** A runtime hook would check only the types that some test sends. The syntax scan sees every function. Its limit: it does not see the type of the destination, so a handler that swaps its request type and keeps the same number of decodes is not seen. `golang.org/x/tools` (`go/packages`, which would see it) is not vendored.
- **The walk is not extended.** That was #883. A type outside the shapes above gets its own decode (as `strictJSON` does for a body with unknown members), after a person has read what `unmarshalBody` would do with it.
- **Embedded fields fail the guard although the walk names them today.** I checked on Go 1.27.4 that a member of an embedded struct is reported as `a`, with no embedded name in the path. The guard still fails them, because Go's rules for which promoted member wins a name are not modelled by the name checks, and the walk nulls by name. An allowed embedded field would need those checks first. Say so in review if you prefer to allow them now.
- **`string` is allowed** (`json:"quoted,string"`): checked on Go 1.27.4, the error keeps the field path, and the allowed-shapes test decodes one.
- **Stop rule, not a better search.** The walk could find the failing member through the type, but that is #883. The stop keeps the cost of a wrong path at 2 decodes and the answer at what it was.

- Against main: `go test -overlay` with main's `decode_body.go` fails the five cases of `TestUnmarshalBodyStopsWithoutProgress` with 100 decodes each; with this head they take 2 (1 for a member that is null already). Mutants: no `bytes.Equal` check fails the null case; no `named` check fails the other four (the answer is not the lone `encoding/json` error).
- Guard mutants on the real types: a map field in `wmMergeRequest`, an embedded struct in `createWebhookRequest`, a map under `usyncBaseRepo.Auth`, a new function that calls `unmarshalBody`, and a second decode in `handleMerge` each fail `TestRequestTypesStayInsideTheDecodeWalk` with a message that names the type or function. 14 mutants of `walkProblems` (each rule, the pointer step, the slice step, the struct step, the unexported and `-` skips) each fail `TestDecodeWalkProblems`; the one that removes the cycle check runs out of memory instead of passing.
- The SQLite lane `go test ./internal/api/... -tags=openstorage_viewcheck -count=1 -race -skip '/^pg$|//^pg$'` passes on this head (342.7 s, Go 1.27.4). golangci-lint: 0 issues. Docs gate: 0 broken links, 80 script tests pass. Both fingerprint gates pass. The PostgreSQL lane runs in CI.

Signed-off-by: Thomas Kosiewski <tk@coder.com>

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_

Change-Id: I27b4a6128df3fc230faf6586b85e01fea831c8f3
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T19:44:23.707358Z 73d1237 Manual request
🔒 Security Review ✅ Completed 2026-11-16T19:40:23.126599Z 73d1237 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 805436e848

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 805436e848

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/api/decode_walk_guard_test.go Resolved
A helper that takes its destination as a parameter hid its callers from the scan of decodeWalkSites. A function with such a parameter must now be listed as a helper, and a listed helper joins the scan, so its callers are listed with their request types.

Change-Id: I395f3387c6c6696a0a7f5ef3f9888fa8eb37ec13
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 73d123783a

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 73d123783a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +121 to +122
case reflect.Slice, reflect.Array:
walk(t.Elem(), path+"[]")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject byte slices from the allowed walk shapes

When a request type gains a []byte (or another byte-slice type), this branch recursively accepts it as an ordinary slice of uint8. However, encoding/json treats JSON strings for byte slices as base64; malformed base64 returns a base64.CorruptInputError, so unmarshalBody does not receive a field-bearing UnmarshalTypeError and returns a generic invalid-body response, potentially hiding other bad fields. Special-case byte slices as unsupported, or explicitly handle and test their decode errors.

Useful? React with 👍 / 👎.

Comment on lines +150 to +152
for _, opt := range strings.Split(opts, ",") {
if opt != "" && opt != "omitempty" && opt != "omitzero" && opt != "string" {
add(fpath, "the json tag %q has the option %q, which the walk was not checked for", tag, opt)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject ,string fields that bypass field attribution

When any guarded request field uses the allowed ,string option and receives an unquoted value (for example, 3 for an int tagged json:",string"), encoding/json returns an invalid use of ,string struct tag error rather than a field-bearing UnmarshalTypeError. unmarshalBody therefore returns a generic invalid-body error and can hide subsequent wrong fields, even though this guard approves the request type. Either reject ,string here or add handling that preserves the field path for all of its failure modes.

Useful? React with 👍 / 👎.

Comment on lines +231 to +233
case *ast.Ident:
takes = takes || n.Name == "any" || typeParams[n.Name]
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Recognize named interface destination parameters

When a decode helper accepts its destination through a named interface or alias rather than literal any/interface{}, this check leaves takesDst false. The helper is then registered with concrete types instead of as a helper, so its name never joins walkEntries; a later caller can pass an unsafe request type without changing the helper's scanned place count, and the guard still passes. Resolve named parameter types or otherwise require such destination-forwarding helpers to be registered transitively.

Useful? React with 👍 / 👎.

Comment on lines +117 to +120
case reflect.Bool, reflect.String,
reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64,
reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64,
reflect.Float32, reflect.Float64:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Exclude json.Number from ordinary string handling

When a guarded request contains a json.Number field, this switch accepts it solely because its reflected kind is string. For input such as {"number":"not-a-number"}, however, encoding/json returns a plain invalid number literal error without the field path; unmarshalBody consequently emits a generic invalid-body response and can hide later wrong fields. Reject json.Number explicitly or add a tested exception that preserves attribution for its invalid-string case.

Useful? React with 👍 / 👎.

Comment on lines +127 to +131
case reflect.Struct:
if seen[t] {
return
}
seen[t] = true

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Track cycles through slices and arrays

When a request type is recursively defined through a slice or an array behind a pointer (for example, type Nodes []*Nodes), walkProblems revisits the same reflected type indefinitely because cycle detection is applied only after reaching the Struct case. Adding such an otherwise valid JSON request field makes this guard recurse until stack or memory exhaustion instead of producing a test result. Mark composite types as seen before descending through slices and arrays as well.

Useful? React with 👍 / 👎.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant