Skip to content

fix(api): find map keys with dots, slashes or tildes in unmarshalBody - #883

Closed
ThomasK33 wants to merge 4 commits into
mainfrom
decode-member-path-keys
Closed

ThomasK33 wants to merge 4 commits into
mainfrom
decode-member-path-keys

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Why

A 400 must name each invalid field of a request body (upstream invalid_params "names each invalid field", docs/upstream/openapi.yaml:2937-2943) and must never repeat a large piece of request text, whatever the destination type. unmarshalBody names every wrong-type field (#836): it replaces the value that encoding/json reports with null and decodes again. It found that value by splitting UnmarshalTypeError.Field at each . and matching member names with no Go type. Go 1.27 builds encoding/json on json/v2 and spells the field as a JSON Pointer with each / written as .: a map key keeps its ., writes / as ~1 and ~ as ~0. Without the Go type, a name with . can stand for any value. A map key with ., / or ~ stopped the loop early or nulled the wrong member. Matching names that hold . across segments without the type (this PR's first round) nulled unknown members too: a grep body with 150 wrong paths entries and an unknown 60 KiB member "paths.0" named 100 fields instead of main's 4. It also missed fields when a key with . and a nested key share one path. A map key that does not decode, or an UnmarshalJSON that refuses null, spent up to 100 decodes with no progress, and a body nested 9,000 deep took 4.4 s to search. The walk of the second round read a struct tag by cutting it at the first comma. Go 1.27.4 reads tags with json/v2's rules: a name with a backslash or a quote is cut to the letters and digits that start it (bad\name is bad), or the Go name counts when none do, case:strict stops a name from matching in another case, and ,embed inlines a struct or makes a map take the members that no field takes. For a type with such a tag the walk lost the member that the error named, and the 400 named only the first wrong field.

What changed

  • nullMembers (internal/api/decode_body.go) walks the destination's reflect.Type along the path in one pass over the body (spanWalk). A struct member is the field that encoding/json decodes it into (bodyFields, fieldFor): fields of embedded structs and pointers to them, the shallowest or only tagged field of a name, exact name first, then the first field without regard to case unless its tag has case:strict. A map member is its exact key, escaped as a JSON Pointer token, and a key with . takes several segments. Array elements match by index.
  • jsonTag reads a tag as encoding/json does on Go 1.27.4 (parseFieldOptions of json/v2): the name ends at the first ,; when a backslash, a quote or a backtick comes first, the name is the run of letters, digits and _ that starts the tag, and the Go name when that run is empty or starts with a digit; invalid UTF-8 in the name reads as U+FFFD. It reports the options embed and case:strict (without case:ignore).
  • ,embed without a name: a struct field adds its fields one level deeper, as an embedded struct does. A map with string keys whose key type has no marshal methods, or a jsontext.Value, takes the members that no field takes (the shallowest one, unless two share its depth). The walk follows such a member as a key of that map. Any other ,embed field is a plain field, as in encoding/json.
  • When two values have one path (a key a.b and a key b inside a), it nulls the ones of the type that the error names, or every one when none has that type (a map key, an UnmarshalJSON error).
  • unmarshalBody stops when a pass leaves the body as it was, or when the next decode names the same field with the same type again.
  • TestUnmarshalBodyPaths has 26 rows: keys with ., /, ~ and ~1, nested maps, map keys that differ in case, two values with one path, a map key that does not decode, an UnmarshalJSON that refuses null, fields of embedded and pointer-embedded structs, a field that two embedded structs hide, a field name with . and /, and 8 rows for the field rules: the shallower or only tagged field of a name and the - tag, the exact name before another case and the first of those, options after a name, a named embedded struct and a name with ~1, an unexported embedded int and a type embedded twice and recursively, tag names with a backslash, a quote, a leading digit or invalid UTF-8, case:strict and case:ignore, ,embed on a struct and on maps, and ,embed on a jsontext.Value and on maps whose keys are no plain strings. TestUnmarshalBodyBounds checks that unknown members whose names spell a path do not change how many fields a body names.

Scope

No answer of a wired route changes: no wired request type has a map, an embedded struct, a field name with ., / or ~, or a json tag with a backslash, a quote, case: or embed (no Go file under internal/ or cmd/ has such a tag outside decode_body_test.go), and the differential below shows identical answers. So the docs do not change. Answers change only for destination types that have these.

Design calls

  • Walk the Go type with the path. Option not taken: null the value at the error's Offset. On Go 1.27.4 Offset is the end of the value for most errors. It is the start for an interface or an embedded pointer to an unexported struct, and it is relative to the inner value for an error from an UnmarshalJSON. Nulling one value would also name a repeated member twice and change today's answers.
  • Resolve struct members by encoding/json's rules in a list kept once for each type. Option not taken: decode through encoding/json/v2 with a hook on every value. No production code imports json/v2 (Go 1.27.4 builds encoding/json on it by default; only tests import it).
  • Read tags by json/v2's rules, which the default Go 1.27.4 build of encoding/json uses. Option not taken: v1's isValidTag with the Go name for any name it rejects. On Go 1.27.4 encoding/json reads bad\name as bad, so that rule would miss a member bad whose Go field has another name.
  • Follow an ,embed map: a member that no field takes is a key of it. Option not taken: stop the walk at a struct with an ,embed map, which names only the first wrong field of such a body, even when the other wrong fields are plain fields.
  • Read tag options as whole words between commas. Option not taken: json/v2's option scanner, which also reads the malformed options embed:x and case:strict-x (staticcheck reports both) as embed and case:strict.
  • Tell a jsontext.Value by its package path and name, and marshal methods of a map key type by their names. Option not taken: json/v2's interface checks, which need jsontext types and so an import of json/v2 in production code.
  • Pick between two values with one path by the type that the error names. Option not taken: decode each candidate alone to see which one is wrong, which costs decodes that the work bound does not count.
  • Walk a type with its own UnmarshalJSON as its fields, which is right for wrefCreateBranchRequest (it decodes into an alias of itself). Option not taken: stop at such a type, which would name only the first wrong field of a branch create.
  • Match map keys exactly, as encoding/json does, and struct members without regard to case. Option not taken: case-insensitive map keys, which name only the first of two wrong keys that differ in case.
  • The same-field stop rule compares the type too. Option not taken: the field alone, which stops when a key a.b and a nested a, b are both wrong and leaves later fields unnamed.
  • One pass with one decoder. Option not taken: a depth limit on the old walk, which still reads each level's subtree again.
  • Null a member rather than remove it, so array indices do not shift.

Verification

  • On the previous head b7de6f2 (go test -overlay with its decode_body.go), 4 of the 26 rows of TestUnmarshalBodyPaths fail, the rows for the review's findings: the tag names row names only bs.k (want 5 fields), case:strict only CS.k (want 4), ,embed only in.k (want 4), and the jsontext.Value row only Raw.k (want 4). The 4 other new rows pass there and fail on mutants. On main bece294 (its decode_body.go), 15 of the 26 rows fail. All 26 pass on this head.
  • Mutants: 38 mutants of the field rules, each run against TestUnmarshalBodyPaths. 34 fail it: dominance by depth or by tag (removed or reversed), no exact match first, the last match in another case, options kept in the name, the - tag kept, a named embedded struct inlined, an unexported embedded struct dropped, an unexported embedded int kept, a seen type followed (timeout) or not listed, follow ignored (timeout), the ~1/~0 order, no unescaping, unexported fields kept, pointers not unwrapped, member and path not compared, each rule of jsonTag, case:strict ignored or applied to exact names, case:ignore ignored, embed ignored, no routing to the ,embed map, its depth rule reversed, jsontext.Value, key methods and string keys ignored. 4 survive and change no answer: the root type not marked seen, no tie rule between two ,embed maps of one depth, the map check on the field that takes other members, and an unexported ,embed map kept. On each of them the oracle below finds 0 differences over 80,000 bodies of 200 generated types.
  • Oracle: a generator writes struct types with random tags (names with \, ', ", a leading digit or invalid UTF-8, -, -,, omitempty, omitzero, case:strict, case:ignore, both, ,embed on structs, maps, jsontext.Value and maps whose keys unmarshal) and random embedding (named, unexported, pointer, recursive, one type embedded twice). encoding/json decodes each candidate member alone, which tells where it lands (each field has a type of its own); a simulation of unmarshalBody built from those decodes gives the expected names, and unmarshalBody must name the same list. On this head: 0 differences over 355,600 bodies of 900 types. On b7de6f2 the same bodies differ 136,960 times.
  • Differential of 4,483 requests over the 16 route families of TestDecodeNamesEveryWrongTypeField: generated bodies with unknown members that spell field paths (with ., /, ~1, ~0 and other case), duplicate members, the review's grep and merge bodies, and for each field path a 60 KiB member and 500 members that spell it. Status, Content-Type and body are byte-identical to main bece294 (its decode_body.go is the one of the merge base 535e967), with the request path in instance replaced in both; test binaries md5 d5869e80 (head) and 08cec3be (main), answer files md5 46890edd both. All answers are 400 application/problem+json, 2,700 name two or more fields, 130 bodies are 60 KiB or more, and the largest answer has 12,668 bytes.
  • A body nested 9,000 deep into type R map[string]R takes 8 ms (best of 3; main 3.92 s).
  • SQLite lane go test ./internal/api/... -tags=openstorage_viewcheck -count=1 -race -timeout=40m -skip '/^pg$|//^pg$' passes (307 s). golangci-lint 2.15.0: 0 issues. Docs gate: 0 broken links, 80 script tests pass. Both fingerprint gates pass. The PostgreSQL lane runs in CI.
  • An independent review of b7de6f2 checked the walk against a json/v2 hook that records where each value lands: no mismatch in about 200,000 checks for plain tags, mismatches for tag names with ', \ or ", case:strict and ,embed on a map, and 14 mutants of the field rules that no row killed. On that head's merge with main, 960,000 generated bodies over the 16 wired types and a 15-minute fuzz gave 0 differences from main, and the per-type cache showed no race. This head fixes those tag forms and adds the rows above.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: xhigh

unmarshalBody names every wrong-type field of a request body: it
replaces the member that encoding/json reports with null and decodes
again. It found that member by splitting UnmarshalTypeError.Field at
each ".", but Go 1.27 spells the field as a JSON Pointer with each "/"
written as ".": a map key keeps its ".", and writes "/" as ~1 and "~"
as ~0. A key with any of these was not found, so the loop stopped
early and left later wrong-type fields unnamed, or it nulled the wrong
member (a key "a~1b" for the path of key "a/b") and decoded the same
body up to 100 times. A map key that does not decode, or an
UnmarshalJSON that refuses null, also spent up to 100 decodes with no
progress.

memberSpans now escapes each member name as a JSON Pointer token and
lets a name with "." take several segments of the path, so it finds
every value with that path. unmarshalBody stops when a pass leaves the
body as it was or the next decode names the same field again. Member
names still match without regard to case, as before.

No wired request type has a map or such an UnmarshalJSON, so no route
answers differently: a differential of 18593 generated requests over
the wired routes gives byte-identical answers.

_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_

Change-Id: Icf88f413d932ccc51334619dcdc85b784f996fbd
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T14:39:55.872094Z ac74193 Manual request
🔒 Security Review ✅ Completed 2026-11-16T14:36:40.194089Z ac74193 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: bb143a2cc0

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: bb143a2cc0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

unmarshalBody names every wrong-type field by replacing the value that
encoding/json reports with null and decoding again. It found that value
by matching UnmarshalTypeError.Field against member names without the
Go type. A name with "." could then stand for any value: an unknown
member such as "paths.0" was nulled too, which changed how many fields
a grep body names (100 instead of 4 for one body), a key "a.b" in
another case hid a wrong field, and a name longer than the path made
the walk index past its end.

The search now walks the destination's reflect.Type along the path in
one pass over the body. A struct member is the field that encoding/json
decodes it into (embedded and pointer-embedded fields, exact name first,
then without regard to case), so an unknown member is never on the path.
A map member is its exact key, which may span several segments. Of two
values with one path, those of the type that the error names are
nulled. The loop stops when a decode names the same field with the same
type again. One pass also makes a body nested 9,000 deep take 39 ms
instead of 4.4 s.

A differential of 4,483 requests over the 16 wired route families,
with unknown dotted members and the bodies of the review, is
byte-identical to main.

_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_

Change-Id: Idc7e34b1ec97df80c950b346cce572ad7db88db2
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Resolves the import block of internal/api/decode_body.go, where #882
added unicode/utf8.

_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_

Change-Id: I8272fd4c73e61867fd64a5139ccf93008d0e4e92
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: b7de6f2008

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b7de6f2008

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/api/decode_body.go Resolved
unmarshalBody names every wrong-type field of a body by nulling the
member that encoding/json reports and decoding again. Its walk to that
member cut a struct tag at the first comma. On Go 1.27, whose
encoding/json runs on json/v2, three tag forms read differently, so the
walk lost the member and the 400 named only the first wrong field:

- A name with a backslash, quote or backtick: encoding/json takes the
  letters, digits and "_" that start the tag (`bad\name` is "bad"), or
  the Go name when there are none. Invalid UTF-8 reads as U+FFFD.
- case:strict: the name matches in its own case only, unless the tag
  also has case:ignore.
- ,embed without a name: a struct field adds its fields as an embedded
  struct does. A map with string keys whose key type has no marshal
  methods, or a jsontext.Value, takes the members that no field takes:
  the shallowest one, unless two share its depth.

jsonTag reads the name and the options as encoding/json does,
bodyFields records case:strict and the field that takes the other
members, and the walk follows such a member as a map key. No wired
request type has these tags, so no answer of a wired route changes.

TestUnmarshalBodyPaths gets rows for these forms and for the field
rules that had none: the shallower or only tagged field of a name, the
exact name before another case and the first of those, options after
the name, the "-" tag, a named or unexported embedded field, a type
embedded twice and recursively, and the order of ~1 and ~0.

_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `xhigh`_

Change-Id: Iee9fc313a358fe1427181843705ff6218c5483cd
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: ac741937bb

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ac741937bb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +173 to +174
if slices.ContainsFunc(w.spans, func(s valueSpan) bool { return s.typ == te.Type }) {
w.spans = slices.DeleteFunc(w.spans, func(s valueSpan) bool { return s.typ != te.Type })

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve same-typed ambiguous path candidates

When ambiguous paths have the same recursive type, filtering solely by te.Type keeps and nulls multiple candidates, including values that have not failed yet. For example, with type R map[string]R, decoding {"r":{"a":{"b":1},"a.b":{"c":2}},"count":"x"} first reports r.a.b; both the nested value and the whole "a.b" map have type R, so this code nulls both and permanently hides the later r.a.b.c error. The resulting response therefore omits an invalid field despite unmarshalBody promising to name every wrong-type field.

Useful? React with 👍 / 👎.

Comment on lines +398 to +400
return t.Kind() == reflect.Map && t.Key().Kind() == reflect.String && !slices.ContainsFunc(marshalMethods, func(m string) bool {
_, ok := reflect.PointerTo(t.Key()).MethodByName(m)
return ok

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Check method signatures before rejecting embedded maps

When an embedded map's string-like key type merely declares one of these method names with an unrelated signature, MethodByName classifies it as a custom-marshaled key even though encoding/json only checks whether the type implements the corresponding interface. For example, a key type with func (K) MarshalJSON() does not implement json.Marshaler, so encoding/json still routes unmatched members into map[K]int, while takesRest rejects that map; a body whose first unmatched member has the wrong value type then cannot be found by spanWalk, and later invalid fields are omitted. Check interface-compatible signatures rather than method names alone.

Useful? React with 👍 / 👎.

@ThomasK33

Copy link
Copy Markdown
Member Author

Status: this PR is held at its review cap (6 Codex requests used) and is not merged.

  • Head ac741937 passes CI, and an independent review found no wired route whose answer differs from main (4,160 bodies over the 16 route families, 0 differences, 0 panics, nothing over the work budget).
  • Open findings on this head, all for destination types that no request type has: two Codex P2 threads (same-typed ambiguous path candidates under a recursive map type; key types whose method names match but whose signatures do not), and the independent review's ISSUES verdict (part of the round-3 tag rule has no test that catches its removal; an ,embed field keeps case:strict where encoding/json drops it; two more minor mismatches).
  • The findings of three rounds keep landing in one place: the walk reimplements encoding/json's field rules for types that the server does not decode. The maintainer decides whether to narrow the PR (support only the shapes that request types use and stop safely on anything else, with a test that pins every request type inside that set) or to close it.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: xhigh

@ThomasK33

Copy link
Copy Markdown
Member Author

Closing this PR in favour of #899.

Why: the walk of this branch does not converge. Its rounds of review drew 5, 3 and 7 findings, and each one was another type or another path (map keys, embedded fields, struct tags, types that decode themselves). Main is correct for every request type that the server decodes with unmarshalBody today: those types hold scalars, structs, pointers, slices, json.RawMessage and one self-decoding type, and nothing that a map key or an embedded field could trip.

What replaces it: #899 makes unmarshalBody stop when a pass makes no progress (main spends up to 100 decodes there), and adds a guard test that fails when a request type gains a map, an embedded field, an interface, an unusual tag or its own decoder, or when a new function decodes a body with unmarshalBody without being listed.

The branch decode-member-path-keys stays as it is (head ac741937bbbc5c4277f493322637d77f0953fb92). The two open review threads here are not carried over: #899 does not change what the walk names.


Generated with xum • Model: anthropic:claude-sonnet-5-5 • Thinking: xhigh

@ThomasK33 ThomasK33 closed this Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant