Skip to content

ingest: the comma guard over-rejects a quoted comma when another parameter is malformed #563

Description

@EricAndrechek

What

Ingest resolves Content-Type with mime.ParseMediaType, one media type per
header line (ingestFormatOne, internal/api/record_reader.go).

ErrInvalidMediaParameter is tolerated, because it means the media type parsed
and only a parameter is malformed — and parameters never decide the format.
But ParseMediaType returns that same error when a second declaration was
comma-joined on after a parameter:

application/json; charset=utf-8, application/x-ndjson
→ ErrInvalidMediaParameter, mediatype "application/json"

Honoring that would resolve a joined disagreement to its first member — an
NDJSON body read as one JSON object, ingesting record one and dropping the rest
behind a 200. That is the hazard RFC 9110 §8.3 explicitly warns about. The two
cases are indistinguishable from the error alone, so the guard refuses any
comma on a line that did not parse cleanly:

if err != nil && (!errors.Is(err, mime.ErrInvalidMediaParameter) || strings.ContainsRune(v, ',')) {

The cost

Failing closed is right, but the rule is line-wide, so it also refuses two
shapes where nothing was joined and the media type is unambiguous:

application/json; profile="a,b"; charset          → 415   (each half alone is 200)
application/json; p="x,y", application/json; x="  → 415   (repeated spelling is 200)

In both the comma sits inside a well-formed quoted value; only some other
parameter is malformed. Neither can silently truncate a batch — they are
over-rejections, never over-acceptances.

One caveat on that phrasing. It holds for the shapes tracked here. It is not
true of comma handling generally: a comma inside a quoted parameter value is
legal qdtext, so application/json; a=", application/x-ndjson; b=" parses
cleanly as one media type and is accepted, where the same pair sent as two
header lines is refused. That is RFC-correct — the value really is one media
type, and the server cannot know an intermediary built it by illegally joining
two singleton lines — but it is a divergence in the accepting direction, and it
is pinned alongside these in the same test.

Fix direction

Distinguish "a comma inside a quoted-string that parsed" from "a comma
introducing joined content" without re-hand-rolling the scanner that #552
deleted. Most promising: ask ParseMediaType about the prefix up to the last
successfully-parsed parameter, and test only the unparsed remainder for a comma.

Whatever the approach, it must keep application/json; charset=utf-8, application/x-ndjson refused — that is the whole reason the guard exists.

Why not now

The over-rejection is the safe direction, and narrowing it is exactly the class
of subtle parsing work that produced the original bug. Better done deliberately
than folded into #552.

Pinning

TestIngest_DuplicateContentTypeHeaders/joined and repeated still diverge pins
the second shape in both directions. Note it does not constrain a narrowed
guard on its own: that case's unparsed remainder also contains a comma, so a fix
scoped to "comma after the last parsed parameter" would keep it green — the
TestIngestFormat row for the non-joined shape is the one that would catch a
regression.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions