Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/explanation/wire-compatibility.md

Large diffs are not rendered by default.

7 changes: 6 additions & 1 deletion docs/reference/http-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,12 @@ are in [`../upstream/openapi.yaml`](../upstream/openapi.yaml).
or that has the dotted path of a member named or counted already, adds nothing. Past
the 100 names of an object the count compares 64-bit hashes of the names, so two of
them count as one in about one body of 1 MiB in a billion. An answer that names one
member keeps its name up to the cut of 16384 bytes above. The other checks run only
unknown member cuts a name of more than 256 bytes as the other bodies above do, to its
first 256 bytes or fewer at a character, and ends it with `…`: `error` and `detail`
add the length in bytes of the member name, without `author.`
(`unknown field "author.xx…" (1000000 bytes)`), and the `invalid_params` `name` ends
with `…` only, as at 32 bytes. Such an answer has at most 4,923 bytes and the request
path in `instance`. The other checks run only
when the members are fine, and then the decoder of a strictly decoded body names only
the first field of the wrong type (`{"note":1,"object_ref":2}` names `note`).
`POST /api/repos/{repo_name}/upstream/pull` takes `{ref?, after?}` and works the same
Expand Down
38 changes: 24 additions & 14 deletions internal/api/decode_body.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ import (
"net/http"
"strconv"
"strings"
"unicode/utf8"

"github.com/coder/openstorage/internal/errtext"
)

// maxDecodeWork bounds the bytes that unmarshalBody decodes for one body
Expand Down Expand Up @@ -117,19 +118,19 @@ func paramsText(params []invalidParam) string {
return strings.Join(items, "; ")
}

// maxUnknownNameBytes bounds the member name that unknownFieldText repeats.
// The longest field name of a request body has 25 bytes, so a misspelled
// name is repeated whole. A longer name is no field name, and the body
// limit lets it reach 1 MiB: `error` and `detail` repeat it with escapes
// of up to 6 bytes for each byte, which made answers of 12 MB.
// maxUnknownNameBytes bounds the member name that an answer naming one
// unknown member repeats (unknownNameText). The longest field name of a
// request body has 25 bytes, so a misspelled name is repeated whole. A
// longer name is no field name, and the body limit lets it reach 1 MiB:
// `error` and `detail` repeat it with escapes of up to 6 bytes for each
// byte, which made answers of 12 MB.
const maxUnknownNameBytes = 256

// unknownFieldText returns `unknown field "x"` for the error that
// encoding/json gives a member that DisallowUnknownFields refuses, and ""
// for any other error. That error has no type: its text is fixed (pinned by
// TestDecodeError), and it holds the member name only. A name of more than
// maxUnknownNameBytes is cut at a character to at most that many bytes, and
// ends with an ellipsis and its length: `unknown field "xx…" (1000000 bytes)`.
// TestDecodeError), and it holds the member name only, which
// unknownNameText cuts.
func unknownFieldText(err error) string {
rest, ok := strings.CutPrefix(err.Error(), "json: unknown field ")
if !ok || !strings.HasPrefix(rest, `"`) {
Expand All @@ -142,11 +143,20 @@ func unknownFieldText(err error) string {
if len(name) <= maxUnknownNameBytes {
return "unknown field " + rest
}
cut := maxUnknownNameBytes
for cut > 0 && !utf8.RuneStart(name[cut]) {
cut--
}
return "unknown field " + strconv.Quote(name[:cut]+"…") + " (" + strconv.Itoa(len(name)) + " bytes)"
return unknownNameText("", name)
}

// unknownNameText returns `unknown field "x"` for an unknown member name
// of the object at the dotted path path ("" for the body), x being
// path+name. A name of more than maxUnknownNameBytes is cut at a
// character to at most that many bytes, and the text ends with an
// ellipsis and the length of the name: `unknown field "xx…" (1000000 bytes)`.
func unknownNameText(path, name string) string {
text := "unknown field " + strconv.Quote(path+errtext.Cut(name, maxUnknownNameBytes))
if len(name) > maxUnknownNameBytes {
text += " (" + strconv.Itoa(len(name)) + " bytes)"
}
return text
}

// nullMembers returns body with every value at the dotted path field
Expand Down
17 changes: 11 additions & 6 deletions internal/api/handlers_write_refs.go
Original file line number Diff line number Diff line change
Expand Up @@ -1000,7 +1000,8 @@ func strictJSON(body []byte, dst any) error {
// that JSON escapes to at most 192: at most 105,203 bytes and the request
// path in `instance` (TestStrictBodiesBoundTheNamesTheyRepeat). No field
// name in upstream's spec has more than 25 bytes. An answer that names
// one member keeps its whole name, as before (errtext.MaxBytes cuts it).
// one unknown member cuts its name at maxUnknownNameBytes, as the other
// decoders do (unknownNameText).
const maxListedNameBytes = 32

// checkStrictObject verifies raw is a JSON object whose member names are
Expand Down Expand Up @@ -1050,9 +1051,11 @@ type strictProblem struct {
kind strictKind
}

// err returns the problem as an error. cut cuts an unknown member name at
// maxListedNameBytes: the other kinds name a field of the struct.
func (p strictProblem) err(cut bool) error {
// err returns the problem as an error. An unknown member name, which is
// request text, is cut: at maxListedNameBytes when listed (the answer
// names several problems), else as unknownNameText cuts it. The other
// kinds name a field of the struct.
func (p strictProblem) err(listed bool) error {
path := p.path + p.name
switch p.kind {
case strictRepeated:
Expand All @@ -1062,10 +1065,12 @@ func (p strictProblem) err(cut bool) error {
case strictNotObject:
return newFieldError(path, "must be an object", path+" must be a JSON object")
}
if cut {
path = p.path + errtext.Cut(p.name, maxUnknownNameBytes)
msg := unknownNameText(p.path, p.name)
if listed {
path = p.path + errtext.Cut(p.name, maxListedNameBytes)
msg = fmt.Sprintf("unknown field %q", path)
}
msg := fmt.Sprintf("unknown field %q", path)
if p.name == "" { // an empty member name has no dotted wire path
return errors.New(msg)
}
Expand Down
4 changes: 2 additions & 2 deletions internal/api/invalid_params_notes_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -388,8 +388,8 @@ func TestStrictBodiesNameEveryBadMember(t *testing.T) {
// bytes of each, so it has at most 105,203 bytes and the request path: 100
// items, each with two messages in `error` and in `detail` (a name in
// author and the same name at the top level as author.<name>). An answer
// that names one member keeps its name up to 16384 bytes
// (TestLongValuesAreCutInAnswers). A name that an earlier item shows after
// that names one unknown member cuts its name at 256 bytes
// (TestInvalidParamsOnBodyFailures). A name that an earlier item shows after
// the cut gives no item of its own and counts in `and N more`. Each object
// (all copies of author are one) keeps 100 unknown names, and `and N more`
// counts each member that the answer leaves out once: a repeat, or a member
Expand Down
16 changes: 13 additions & 3 deletions internal/api/problem_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1299,7 +1299,9 @@ func testInvalidParamsOnBodyFailures(t *testing.T, e *testEnv) {
type param struct{ name, reason string }
// The answer cuts an unknown member name of more than 256 bytes to at
// most 256, at a character, and states its length, so it stays small
// whatever escapes the name needs.
// whatever escapes the name needs: under 5 KiB when a strict route
// repeats a name of `<`, six bytes each in JSON, in `error`, `detail`
// and the `invalid_params` name.
long := func(n int, s string) string { return strings.Repeat(s, n) }
for _, tc := range []struct {
name, method, path, token string
Expand Down Expand Up @@ -1328,14 +1330,22 @@ func testInvalidParamsOnBodyFailures(t *testing.T, e *testEnv) {
`invalid JSON body: unknown field "` + long(256, "<") + `…" (1000000 bytes)`, nil},
{"upstream unknown field cut with a quote and a control character", http.MethodPost, "/api/v1/repos", freshTok, `{"upstream":{"\"\u0001` + long(300, "y") + `":1}}`,
`invalid upstream: unknown field "\"\x01` + long(254, "y") + `…" (302 bytes)`, nil},
{"strict unknown field of 256 bytes", http.MethodPost, "/api/repos/" + url + "/notes/append", gitTok, `{"` + long(256, "y") + `":1}`,
`invalid JSON body: unknown field "` + long(256, "y") + `"`, &param{long(256, "y"), "is not a known field"}},
{"strict unknown field of escaped bytes", http.MethodPost, "/api/repos/" + url + "/notes/append", gitTok, `{"` + long(1_000_000, "<") + `":1}`,
`invalid JSON body: unknown field "` + long(256, "<") + `…" (1000000 bytes)`, &param{long(256, "<") + "…", "is not a known field"}},
{"strict unknown field in author cut at a character", http.MethodPost, "/api/repos/" + url + "/notes/append", gitTok, `{"author":{"a` + long(400_000, "é") + `":1}}`,
`invalid JSON body: unknown field "author.a` + long(127, "é") + `…" (800001 bytes)`, &param{"author.a" + long(127, "é") + "…", "is not a known field"}},
{"strict pull unknown field cut with a quote and a control character", http.MethodPost, "/api/repos/" + url + "/upstream/pull", gitTok, `{"\"\u0001` + long(300, "y") + `":1}`,
`invalid JSON body: unknown field "\"\x01` + long(254, "y") + `…" (302 bytes)`, &param{"\"\x01" + long(254, "y") + "…", "is not a known field"}},
{"strict empty member name", http.MethodPost, "/api/repos/" + url + "/notes/append", gitTok, `{"":1}`, `invalid JSON body: unknown field ""`, nil},
{"syntax error", http.MethodPatch, "/api/v1/repo", repoTok, `{`, "invalid JSON body", nil},
{"wrong top-level type", http.MethodPatch, "/api/v1/repo", repoTok, `[]`, "invalid JSON body", nil},
} {
t.Run(tc.name, func(t *testing.T) {
resp, body := e.doReq(t, tc.method, tc.path, tc.token, []byte(tc.body), "")
if resp.StatusCode != http.StatusBadRequest || len(body) > 4<<10 {
t.Fatalf("status = %d, want 400 of at most 4 KiB (body of %d bytes %.300s)", resp.StatusCode, len(body), body)
if resp.StatusCode != http.StatusBadRequest || len(body) > 5<<10 {
t.Fatalf("status = %d, want 400 of at most 5 KiB (body of %d bytes %.300s)", resp.StatusCode, len(body), body)
}
requireEnvelope(t, resp, body)
var p problemBody
Expand Down
2 changes: 1 addition & 1 deletion internal/errtext/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Bounds the request text that an error answer repeats, so that an answer stays sm

## Callers
- `api` cuts `error`, `detail` and each `invalid_params` name of a problem body (`newProblem`), and the `result.message` of a commit write (`wcRefUpdate`) [../api/errors.go:newProblem] [../api/handlers_write_commit.go:wcRefUpdate]
- `api` cuts each unknown member name at 32 bytes in the answer of a strictly decoded body that names several members (`Cut`) [../api/handlers_write_refs.go:strictProblem.err]
- `api` cuts an unknown member name at 256 bytes in an answer that names one unknown member (`unknownNameText`), and each unknown member name at 32 bytes in the answer of a strictly decoded body that names several members (`Cut`) [../api/decode_body.go:unknownNameText] [../api/handlers_write_refs.go:strictProblem.err]
- `githttp` cuts the error text of the git and LFS routes [../githttp/server.go:httpError] [../githttp/lfs.go:lfsError]

## Verification
Expand Down
Loading