Repository navigation
Conversation
… no progress Two gaps remain around that fact: 1. Nothing fails when a request type gains a field that the path is wrong for. Measured on main (`dc6ef0cf`, Go 1.27.4): a map key that holds a dot, a map key that is a path escape (Go 1.27 reports `a/b` as `a~1b`), a `map[int]int` key that does not decode, and a type that refuses null each leave the loop with a member that nulling does not clear, or with none. 2. In those cases the loop does not stop. It nulls a member that is not the failing one (or the failing one, which stays an error), decodes again, gets the same error and repeats until it holds 100 problems or the 256 KiB work budget is spent. The answer is the one error (the 100 copies are one name), but the cost is 100 decodes: 0.64 ms for a body of 50 bytes and 2.9 ms for one of 2 KiB, against 13 µs and 26 µs here. No request type of main reaches either gap today. - `unmarshalBody` (`internal/api/decode_body.go`) stops when a pass leaves the body unchanged (the member is null already, or nulling it is a no-op) and when a decode names a field that an earlier pass named. Neither changes an answer: the 100 copies were one problem. A body with one wrong-type field still returns the error of `encoding/json` itself. The doc comment says what the path is right for and points at the guard test. - `TestRequestTypesStayInsideTheDecodeWalk` (new, `internal/api/decode_walk_guard_test.go`) walks every request type that the server decodes with `unmarshalBody` (16 types) and fails for: a map, an interface, an embedded field, a member name outside `[A-Za-z0-9_]`, two names of one struct that differ only in case, a tag option other than `omitempty`, `omitzero` and `string`, a type with its own `UnmarshalJSON` or `UnmarshalText` that is not on a two-entry list (`json.RawMessage`, `wrefCreateBranchRequest`, each with the test that shows the walk right for it), and any kind that is no JSON type. - The same test reads the syntax of the package (`go/parser`) and fails when a function that is not in `decodeWalkSites` names `unmarshalBody` or one of its three helpers, or names it a different number of times. So a new endpoint, or a new decode in an old one, must be listed with its request types before the test passes. - `TestDecodeWalkProblems` checks the checker: 20 synthetic types, each kind that it must flag (at any depth through pointers, slices and arrays) and each shape that it must pass. `TestDecodeWalkHandlesWhatTheGuardAllows` decodes a type with every allowed shape and a wrong value in each, and requires that `unmarshalBody` names every field, one decode each. - `TestUnmarshalBodyStopsWithoutProgress` covers the loop: the four cases above plus a type that refuses a null it was sent. Test code and the loop of one function. No answer changes for any body that a request type of main can send; the status codes, the messages and `invalid_params` are as on main. `AGENTS.md` and the docs are unchanged: the message of a failing guard says what to do, and the comment of `unmarshalBody` says what the walk is right for. - **The guard is a test, not a hook in `unmarshalBody`.** A runtime hook would check only the types that some test sends. The syntax scan sees every function. Its limit: it does not see the type of the destination, so a handler that swaps its request type and keeps the same number of decodes is not seen. `golang.org/x/tools` (`go/packages`, which would see it) is not vendored. - **The walk is not extended.** That was #883. A type outside the shapes above gets its own decode (as `strictJSON` does for a body with unknown members), after a person has read what `unmarshalBody` would do with it. - **Embedded fields fail the guard although the walk names them today.** I checked on Go 1.27.4 that a member of an embedded struct is reported as `a`, with no embedded name in the path. The guard still fails them, because Go's rules for which promoted member wins a name are not modelled by the name checks, and the walk nulls by name. An allowed embedded field would need those checks first. Say so in review if you prefer to allow them now. - **`string` is allowed** (`json:"quoted,string"`): checked on Go 1.27.4, the error keeps the field path, and the allowed-shapes test decodes one. - **Stop rule, not a better search.** The walk could find the failing member through the type, but that is #883. The stop keeps the cost of a wrong path at 2 decodes and the answer at what it was. - Against main: `go test -overlay` with main's `decode_body.go` fails the five cases of `TestUnmarshalBodyStopsWithoutProgress` with 100 decodes each; with this head they take 2 (1 for a member that is null already). Mutants: no `bytes.Equal` check fails the null case; no `named` check fails the other four (the answer is not the lone `encoding/json` error). - Guard mutants on the real types: a map field in `wmMergeRequest`, an embedded struct in `createWebhookRequest`, a map under `usyncBaseRepo.Auth`, a new function that calls `unmarshalBody`, and a second decode in `handleMerge` each fail `TestRequestTypesStayInsideTheDecodeWalk` with a message that names the type or function. 14 mutants of `walkProblems` (each rule, the pointer step, the slice step, the struct step, the unexported and `-` skips) each fail `TestDecodeWalkProblems`; the one that removes the cycle check runs out of memory instead of passing. - The SQLite lane `go test ./internal/api/... -tags=openstorage_viewcheck -count=1 -race -skip '/^pg$|//^pg$'` passes on this head (342.7 s, Go 1.27.4). golangci-lint: 0 issues. Docs gate: 0 broken links, 80 script tests pass. Both fingerprint gates pass. The PostgreSQL lane runs in CI. Signed-off-by: Thomas Kosiewski <tk@coder.com> --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_ Change-Id: I27b4a6128df3fc230faf6586b85e01fea831c8f3
|
@codex review |
|
@codex security review |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 805436e848
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A helper that takes its destination as a parameter hid its callers from the scan of decodeWalkSites. A function with such a parameter must now be listed as a helper, and a listed helper joins the scan, so its callers are listed with their request types. Change-Id: I395f3387c6c6696a0a7f5ef3f9888fa8eb37ec13 Signed-off-by: Thomas Kosiewski <tk@coder.com>
|
@codex review |
|
@codex security review |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 73d123783a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| case reflect.Slice, reflect.Array: | ||
| walk(t.Elem(), path+"[]") |
There was a problem hiding this comment.
Reject byte slices from the allowed walk shapes
When a request type gains a []byte (or another byte-slice type), this branch recursively accepts it as an ordinary slice of uint8. However, encoding/json treats JSON strings for byte slices as base64; malformed base64 returns a base64.CorruptInputError, so unmarshalBody does not receive a field-bearing UnmarshalTypeError and returns a generic invalid-body response, potentially hiding other bad fields. Special-case byte slices as unsupported, or explicitly handle and test their decode errors.
Useful? React with 👍 / 👎.
| for _, opt := range strings.Split(opts, ",") { | ||
| if opt != "" && opt != "omitempty" && opt != "omitzero" && opt != "string" { | ||
| add(fpath, "the json tag %q has the option %q, which the walk was not checked for", tag, opt) |
There was a problem hiding this comment.
Reject
,string fields that bypass field attribution
When any guarded request field uses the allowed ,string option and receives an unquoted value (for example, 3 for an int tagged json:",string"), encoding/json returns an invalid use of ,string struct tag error rather than a field-bearing UnmarshalTypeError. unmarshalBody therefore returns a generic invalid-body error and can hide subsequent wrong fields, even though this guard approves the request type. Either reject ,string here or add handling that preserves the field path for all of its failure modes.
Useful? React with 👍 / 👎.
| case *ast.Ident: | ||
| takes = takes || n.Name == "any" || typeParams[n.Name] | ||
| } |
There was a problem hiding this comment.
Recognize named interface destination parameters
When a decode helper accepts its destination through a named interface or alias rather than literal any/interface{}, this check leaves takesDst false. The helper is then registered with concrete types instead of as a helper, so its name never joins walkEntries; a later caller can pass an unsafe request type without changing the helper's scanned place count, and the guard still passes. Resolve named parameter types or otherwise require such destination-forwarding helpers to be registered transitively.
Useful? React with 👍 / 👎.
| case reflect.Bool, reflect.String, | ||
| reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, | ||
| reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, | ||
| reflect.Float32, reflect.Float64: |
There was a problem hiding this comment.
Exclude
json.Number from ordinary string handling
When a guarded request contains a json.Number field, this switch accepts it solely because its reflected kind is string. For input such as {"number":"not-a-number"}, however, encoding/json returns a plain invalid number literal error without the field path; unmarshalBody consequently emits a generic invalid-body response and can hide later wrong fields. Reject json.Number explicitly or add a tested exception that preserves attribution for its invalid-string case.
Useful? React with 👍 / 👎.
| case reflect.Struct: | ||
| if seen[t] { | ||
| return | ||
| } | ||
| seen[t] = true |
There was a problem hiding this comment.
Track cycles through slices and arrays
When a request type is recursively defined through a slice or an array behind a pointer (for example, type Nodes []*Nodes), walkProblems revisits the same reflected type indefinitely because cycle detection is applied only after reaching the Struct case. Adding such an otherwise valid JSON request field makes this guard recurse until stack or memory exhaustion instead of producing a test result. Mark composite types as seen before descending through slices and arrays as well.
Useful? React with 👍 / 👎.
Why
#883 taught the walk of
unmarshalBodyto name more members (map keys, embedded fields, struct tags). Its rounds of review drew 5, 3 and 7 findings and did not converge, so this PR replaces it.unmarshalBodyfinds the member that failed by the dotted path thatencoding/jsonreports. That path is right for every request type that main decodes with it: they hold scalars, structs, pointers, slices,json.RawMessageand (forbranches/create) one self-decoding type, and nothing else. This PR keeps it that way instead of widening the walk.Two gaps remain around that fact:
dc6ef0cf, Go 1.27.4): a map key that holds a dot, a map key that is a path escape (Go 1.27 reportsa/basa~1b), amap[int]intkey that does not decode, and a type that refuses null each leave the loop with a member that nulling does not clear, or with none.No request type of main reaches either gap today.
What changed
unmarshalBody(internal/api/decode_body.go) stops when a pass leaves the body unchanged (the member is null already, or nulling it is a no-op) and when a decode names a field that an earlier pass named. Neither changes an answer: the 100 copies were one problem. A body with one wrong-type field still returns the error ofencoding/jsonitself. The doc comment says what the path is right for and points at the guard test.TestRequestTypesStayInsideTheDecodeWalk(new,internal/api/decode_walk_guard_test.go) walks every request type that the server decodes withunmarshalBody(16 types) and fails for: a map, an interface, an embedded field, a member name outside[A-Za-z0-9_], two names of one struct that differ only in case, a tag option other thanomitempty,omitzeroandstring, a type with its ownUnmarshalJSONorUnmarshalTextthat is not on a two-entry list (json.RawMessage,wrefCreateBranchRequest, each with the test that shows the walk right for it), and any kind that is no JSON type.go/parser) and fails when a function that is not indecodeWalkSitesnamesunmarshalBody, or names it a different number of times. So a new endpoint, or a new decode in an old one, must be listed with its request types before the test passes. A function that takes its destination as a parameter of typeany,interface{}or a type parameter is a helper: it is listed with no types, and its name then joins the scan, so that its callers must be listed with theirs (the three helpers of main,wrefDecodeBody,wrefDecodeOptionalBodyandgcredDecodeBody, are listed so). A helper that is listed with types, or a typed function listed without, fails. A Codex review of the first head found that a new helper hid its callers; this closes it.TestScanWalkSiteschecks the scan on a synthetic source (a typed handler, a decode through a value, a helper with each kind of destination parameter, a method helper, a caller of a helper).TestDecodeWalkProblemschecks the checker: 20 synthetic types, each kind that it must flag (at any depth through pointers, slices and arrays) and each shape that it must pass.TestDecodeWalkHandlesWhatTheGuardAllowsdecodes a type with every allowed shape and a wrong value in each, and requires thatunmarshalBodynames every field, one decode each.TestUnmarshalBodyStopsWithoutProgresscovers the loop: the four cases above plus a type that refuses a null it was sent.Scope
Test code and the loop of one function. No answer changes for any body that a request type of main can send; the status codes, the messages and
invalid_paramsare as on main.AGENTS.mdand the docs are unchanged: the message of a failing guard says what to do, and the comment ofunmarshalBodysays what the walk is right for.Design calls
unmarshalBody. A runtime hook would check only the types that some test sends. The syntax scan sees every function. Its limits: it does not see the type of the destination, so a handler that swaps its request type and keeps the same number of decodes is not seen; and a helper that takes a destination in another form (afunc(any) errorvalue, areflect.Value) is not found as a helper.golang.org/x/tools(go/packages, which would see the types) is not vendored.strictJSONdoes for a body with unknown members), after a person has read whatunmarshalBodywould do with it.a, with no embedded name in the path. The guard still fails them, because Go's rules for which promoted member wins a name are not modelled by the name checks, and the walk nulls by name. An allowed embedded field would need those checks first. Say so in review if you prefer to allow them now.stringis allowed (json:"quoted,string"): checked on Go 1.27.4, the error keeps the field path, and the allowed-shapes test decodes one.Verification
go test -overlaywith main'sdecode_body.gofails the five cases ofTestUnmarshalBodyStopsWithoutProgresswith 100 decodes each; with this head they take 2 (1 for a member that is null already). Mutants: nobytes.Equalcheck fails the null case; nonamedcheck fails the other four (the answer is not the loneencoding/jsonerror).wmMergeRequest, an embedded struct increateWebhookRequest, a map underusyncBaseRepo.Auth, a new function that callsunmarshalBody, and a second decode inhandleMergeeach failTestRequestTypesStayInsideTheDecodeWalkwith a message that names the type or function. A new helper withdst anyfails until it is listed; once listed with no types, its new caller fails until it is listed; once listed with a map type, the map fails. Eight mutants of the scan (the helper rule both ways, the interface,anyand type-parameter checks, the receiver strip with and without a receiver, the place count) each failTestScanWalkSitesor the inventory test. 14 mutants ofwalkProblems(each rule, the pointer step, the slice step, the struct step, the unexported and-skips) each failTestDecodeWalkProblems; the one that removes the cycle check runs out of memory instead of passing.go test ./internal/api/... -tags=openstorage_viewcheck -count=1 -race -skip '/^pg$|//^pg$'passes on this head (342.7 s, Go 1.27.4). golangci-lint: 0 issues. Docs gate: 0 broken links, 80 script tests pass. Both fingerprint gates pass. The PostgreSQL lane runs in CI.Generated with
xum• Model:anthropic:claude-sonnet-5-5• Thinking:xhigh