Repository navigation
Conversation
unmarshalBody names every wrong-type field of a request body: it replaces the member that encoding/json reports with null and decodes again. It found that member by splitting UnmarshalTypeError.Field at each ".", but Go 1.27 spells the field as a JSON Pointer with each "/" written as ".": a map key keeps its ".", and writes "/" as ~1 and "~" as ~0. A key with any of these was not found, so the loop stopped early and left later wrong-type fields unnamed, or it nulled the wrong member (a key "a~1b" for the path of key "a/b") and decoded the same body up to 100 times. A map key that does not decode, or an UnmarshalJSON that refuses null, also spent up to 100 decodes with no progress. memberSpans now escapes each member name as a JSON Pointer token and lets a name with "." take several segments of the path, so it finds every value with that path. unmarshalBody stops when a pass leaves the body as it was or the next decode names the same field again. Member names still match without regard to case, as before. No wired request type has a map or such an UnmarshalJSON, so no route answers differently: a differential of 18593 generated requests over the wired routes gives byte-identical answers. _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_ Change-Id: Icf88f413d932ccc51334619dcdc85b784f996fbd Signed-off-by: Thomas Kosiewski <tk@coder.com>
|
@codex review |
|
@codex security review |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
|
Codex Review: Didn't find any major issues. Another round soon, please! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
unmarshalBody names every wrong-type field by replacing the value that encoding/json reports with null and decoding again. It found that value by matching UnmarshalTypeError.Field against member names without the Go type. A name with "." could then stand for any value: an unknown member such as "paths.0" was nulled too, which changed how many fields a grep body names (100 instead of 4 for one body), a key "a.b" in another case hid a wrong field, and a name longer than the path made the walk index past its end. The search now walks the destination's reflect.Type along the path in one pass over the body. A struct member is the field that encoding/json decodes it into (embedded and pointer-embedded fields, exact name first, then without regard to case), so an unknown member is never on the path. A map member is its exact key, which may span several segments. Of two values with one path, those of the type that the error names are nulled. The loop stops when a decode names the same field with the same type again. One pass also makes a body nested 9,000 deep take 39 ms instead of 4.4 s. A differential of 4,483 requests over the 16 wired route families, with unknown dotted members and the bodies of the review, is byte-identical to main. _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_ Change-Id: Idc7e34b1ec97df80c950b346cce572ad7db88db2 Signed-off-by: Thomas Kosiewski <tk@coder.com>
Resolves the import block of internal/api/decode_body.go, where #882 added unicode/utf8. _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_ Change-Id: I8272fd4c73e61867fd64a5139ccf93008d0e4e92 Signed-off-by: Thomas Kosiewski <tk@coder.com>
|
@codex review |
|
@codex security review |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b7de6f2008
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
unmarshalBody names every wrong-type field of a body by nulling the member that encoding/json reports and decoding again. Its walk to that member cut a struct tag at the first comma. On Go 1.27, whose encoding/json runs on json/v2, three tag forms read differently, so the walk lost the member and the 400 named only the first wrong field: - A name with a backslash, quote or backtick: encoding/json takes the letters, digits and "_" that start the tag (`bad\name` is "bad"), or the Go name when there are none. Invalid UTF-8 reads as U+FFFD. - case:strict: the name matches in its own case only, unless the tag also has case:ignore. - ,embed without a name: a struct field adds its fields as an embedded struct does. A map with string keys whose key type has no marshal methods, or a jsontext.Value, takes the members that no field takes: the shallowest one, unless two share its depth. jsonTag reads the name and the options as encoding/json does, bodyFields records case:strict and the field that takes the other members, and the walk follows such a member as a map key. No wired request type has these tags, so no answer of a wired route changes. TestUnmarshalBodyPaths gets rows for these forms and for the field rules that had none: the shallower or only tagged field of a name, the exact name before another case and the first of those, options after the name, the "-" tag, a named or unexported embedded field, a type embedded twice and recursively, and the order of ~1 and ~0. _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_ Change-Id: Iee9fc313a358fe1427181843705ff6218c5483cd Signed-off-by: Thomas Kosiewski <tk@coder.com>
|
@codex review |
|
@codex security review |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ac741937bb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if slices.ContainsFunc(w.spans, func(s valueSpan) bool { return s.typ == te.Type }) { | ||
| w.spans = slices.DeleteFunc(w.spans, func(s valueSpan) bool { return s.typ != te.Type }) |
There was a problem hiding this comment.
Preserve same-typed ambiguous path candidates
When ambiguous paths have the same recursive type, filtering solely by te.Type keeps and nulls multiple candidates, including values that have not failed yet. For example, with type R map[string]R, decoding {"r":{"a":{"b":1},"a.b":{"c":2}},"count":"x"} first reports r.a.b; both the nested value and the whole "a.b" map have type R, so this code nulls both and permanently hides the later r.a.b.c error. The resulting response therefore omits an invalid field despite unmarshalBody promising to name every wrong-type field.
Useful? React with 👍 / 👎.
| return t.Kind() == reflect.Map && t.Key().Kind() == reflect.String && !slices.ContainsFunc(marshalMethods, func(m string) bool { | ||
| _, ok := reflect.PointerTo(t.Key()).MethodByName(m) | ||
| return ok |
There was a problem hiding this comment.
Check method signatures before rejecting embedded maps
When an embedded map's string-like key type merely declares one of these method names with an unrelated signature, MethodByName classifies it as a custom-marshaled key even though encoding/json only checks whether the type implements the corresponding interface. For example, a key type with func (K) MarshalJSON() does not implement json.Marshaler, so encoding/json still routes unmatched members into map[K]int, while takesRest rejects that map; a body whose first unmatched member has the wrong value type then cannot be found by spanWalk, and later invalid fields are omitted. Check interface-compatible signatures rather than method names alone.
Useful? React with 👍 / 👎.
|
Status: this PR is held at its review cap (6 Codex requests used) and is not merged.
Generated with |
|
Closing this PR in favour of #899. Why: the walk of this branch does not converge. Its rounds of review drew 5, 3 and 7 findings, and each one was another type or another path (map keys, embedded fields, struct tags, types that decode themselves). Main is correct for every request type that the server decodes with What replaces it: #899 makes The branch Generated with |
Why
A 400 must name each invalid field of a request body (upstream
invalid_params"names each invalid field",docs/upstream/openapi.yaml:2937-2943) and must never repeat a large piece of request text, whatever the destination type.unmarshalBodynames every wrong-type field (#836): it replaces the value that encoding/json reports withnulland decodes again. It found that value by splittingUnmarshalTypeError.Fieldat each.and matching member names with no Go type. Go 1.27 builds encoding/json on json/v2 and spells the field as a JSON Pointer with each/written as.: a map key keeps its., writes/as~1and~as~0. Without the Go type, a name with.can stand for any value. A map key with.,/or~stopped the loop early or nulled the wrong member. Matching names that hold.across segments without the type (this PR's first round) nulled unknown members too: a grep body with 150 wrongpathsentries and an unknown 60 KiB member"paths.0"named 100 fields instead of main's 4. It also missed fields when a key with.and a nested key share one path. A map key that does not decode, or anUnmarshalJSONthat refusesnull, spent up to 100 decodes with no progress, and a body nested 9,000 deep took 4.4 s to search. The walk of the second round read a struct tag by cutting it at the first comma. Go 1.27.4 reads tags with json/v2's rules: a name with a backslash or a quote is cut to the letters and digits that start it (bad\nameisbad), or the Go name counts when none do,case:strictstops a name from matching in another case, and,embedinlines a struct or makes a map take the members that no field takes. For a type with such a tag the walk lost the member that the error named, and the 400 named only the first wrong field.What changed
nullMembers(internal/api/decode_body.go) walks the destination'sreflect.Typealong the path in one pass over the body (spanWalk). A struct member is the field that encoding/json decodes it into (bodyFields,fieldFor): fields of embedded structs and pointers to them, the shallowest or only tagged field of a name, exact name first, then the first field without regard to case unless its tag hascase:strict. A map member is its exact key, escaped as a JSON Pointer token, and a key with.takes several segments. Array elements match by index.jsonTagreads a tag as encoding/json does on Go 1.27.4 (parseFieldOptionsof json/v2): the name ends at the first,; when a backslash, a quote or a backtick comes first, the name is the run of letters, digits and_that starts the tag, and the Go name when that run is empty or starts with a digit; invalid UTF-8 in the name reads as U+FFFD. It reports the optionsembedandcase:strict(withoutcase:ignore).,embedwithout a name: a struct field adds its fields one level deeper, as an embedded struct does. A map with string keys whose key type has no marshal methods, or ajsontext.Value, takes the members that no field takes (the shallowest one, unless two share its depth). The walk follows such a member as a key of that map. Any other,embedfield is a plain field, as in encoding/json.a.band a keybinsidea), it nulls the ones of the type that the error names, or every one when none has that type (a map key, anUnmarshalJSONerror).unmarshalBodystops when a pass leaves the body as it was, or when the next decode names the same field with the same type again.TestUnmarshalBodyPathshas 26 rows: keys with.,/,~and~1, nested maps, map keys that differ in case, two values with one path, a map key that does not decode, anUnmarshalJSONthat refusesnull, fields of embedded and pointer-embedded structs, a field that two embedded structs hide, a field name with.and/, and 8 rows for the field rules: the shallower or only tagged field of a name and the-tag, the exact name before another case and the first of those, options after a name, a named embedded struct and a name with~1, an unexported embedded int and a type embedded twice and recursively, tag names with a backslash, a quote, a leading digit or invalid UTF-8,case:strictandcase:ignore,,embedon a struct and on maps, and,embedon ajsontext.Valueand on maps whose keys are no plain strings.TestUnmarshalBodyBoundschecks that unknown members whose names spell a path do not change how many fields a body names.Scope
No answer of a wired route changes: no wired request type has a map, an embedded struct, a field name with
.,/or~, or a json tag with a backslash, a quote,case:orembed(no Go file underinternal/orcmd/has such a tag outsidedecode_body_test.go), and the differential below shows identical answers. So the docs do not change. Answers change only for destination types that have these.Design calls
Offset. On Go 1.27.4Offsetis the end of the value for most errors. It is the start for an interface or an embedded pointer to an unexported struct, and it is relative to the inner value for an error from anUnmarshalJSON. Nulling one value would also name a repeated member twice and change today's answers.encoding/json/v2with a hook on every value. No production code imports json/v2 (Go 1.27.4 builds encoding/json on it by default; only tests import it).isValidTagwith the Go name for any name it rejects. On Go 1.27.4 encoding/json readsbad\nameasbad, so that rule would miss a memberbadwhose Go field has another name.,embedmap: a member that no field takes is a key of it. Option not taken: stop the walk at a struct with an,embedmap, which names only the first wrong field of such a body, even when the other wrong fields are plain fields.embed:xandcase:strict-x(staticcheck reports both) asembedandcase:strict.jsontext.Valueby its package path and name, and marshal methods of a map key type by their names. Option not taken: json/v2's interface checks, which needjsontexttypes and so an import of json/v2 in production code.UnmarshalJSONas its fields, which is right forwrefCreateBranchRequest(it decodes into an alias of itself). Option not taken: stop at such a type, which would name only the first wrong field of a branch create.a.band a nesteda,bare both wrong and leaves later fields unnamed.Verification
go test -overlaywith itsdecode_body.go), 4 of the 26 rows ofTestUnmarshalBodyPathsfail, the rows for the review's findings: the tag names row names onlybs.k(want 5 fields),case:strictonlyCS.k(want 4),,embedonlyin.k(want 4), and thejsontext.Valuerow onlyRaw.k(want 4). The 4 other new rows pass there and fail on mutants. On main bece294 (itsdecode_body.go), 15 of the 26 rows fail. All 26 pass on this head.TestUnmarshalBodyPaths. 34 fail it: dominance by depth or by tag (removed or reversed), no exact match first, the last match in another case, options kept in the name, the-tag kept, a named embedded struct inlined, an unexported embedded struct dropped, an unexported embedded int kept, a seen type followed (timeout) or not listed,followignored (timeout), the~1/~0order, no unescaping, unexported fields kept, pointers not unwrapped, member and path not compared, each rule ofjsonTag,case:strictignored or applied to exact names,case:ignoreignored,embedignored, no routing to the,embedmap, its depth rule reversed,jsontext.Value, key methods and string keys ignored. 4 survive and change no answer: the root type not marked seen, no tie rule between two,embedmaps of one depth, the map check on the field that takes other members, and an unexported,embedmap kept. On each of them the oracle below finds 0 differences over 80,000 bodies of 200 generated types.\,',", a leading digit or invalid UTF-8,-,-,,omitempty,omitzero,case:strict,case:ignore, both,,embedon structs, maps,jsontext.Valueand maps whose keys unmarshal) and random embedding (named, unexported, pointer, recursive, one type embedded twice). encoding/json decodes each candidate member alone, which tells where it lands (each field has a type of its own); a simulation ofunmarshalBodybuilt from those decodes gives the expected names, andunmarshalBodymust name the same list. On this head: 0 differences over 355,600 bodies of 900 types. On b7de6f2 the same bodies differ 136,960 times.TestDecodeNamesEveryWrongTypeField: generated bodies with unknown members that spell field paths (with.,/,~1,~0and other case), duplicate members, the review's grep and merge bodies, and for each field path a 60 KiB member and 500 members that spell it. Status,Content-Typeand body are byte-identical to main bece294 (itsdecode_body.gois the one of the merge base 535e967), with the request path ininstancereplaced in both; test binaries md5d5869e80(head) and08cec3be(main), answer files md546890eddboth. All answers are 400application/problem+json, 2,700 name two or more fields, 130 bodies are 60 KiB or more, and the largest answer has 12,668 bytes.type R map[string]Rtakes 8 ms (best of 3; main 3.92 s).go test ./internal/api/... -tags=openstorage_viewcheck -count=1 -race -timeout=40m -skip '/^pg$|//^pg$'passes (307 s). golangci-lint 2.15.0: 0 issues. Docs gate: 0 broken links, 80 script tests pass. Both fingerprint gates pass. The PostgreSQL lane runs in CI.',\or",case:strictand,embedon a map, and 14 mutants of the field rules that no row killed. On that head's merge with main, 960,000 generated bodies over the 16 wired types and a 15-minute fuzz gave 0 differences from main, and the per-type cache showed no race. This head fixes those tag forms and adds the rows above.Generated with
xum• Model:anthropic:claude-opus-5-5• Thinking:xhigh