From 129c6e0b0318c3df87b7939010758caf474eddfd Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 10:15:59 +0200 Subject: [PATCH 01/36] docs: count only spaces toward a blank line --- SPEC.md | 6 +++--- tests/fixtures/decode/blank-lines.json | 8 ++++++++ 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/SPEC.md b/SPEC.md index 007377a..b1c4059 100644 --- a/SPEC.md +++ b/SPEC.md @@ -302,7 +302,7 @@ Quoting keeps "#"-leading data out of the comment rule: string values that equal Decoders classify each line of the comment-stripped sequence (§5.1) by its content after the leading indentation. The first matching class applies. Classification is lexical; whether a class is admissible at a given depth and position is determined by the enclosing construct (root form above, §8–§10). Within a tabular array's scope, lines at row depth are divided between the row and key-value classes by the disambiguation rules of §9.3, which are authoritative for that position and take precedence over the order below. Within a keyed tabular object's scope, every line at entry depth containing an unquoted colon is an entry row; §9.5 is likewise authoritative for that position. -1. Blank line – the content trims to empty. Handled per §12; blank lines never create or close structure. +1. Blank line – the line consists of spaces only (§12). Handled per §12; blank lines never create or close structure. 2. List-item line – the content is the bare marker "-" or begins with "- " (hyphen, space). The remainder after the marker is parsed per §9.2, §9.4, and §10. Outside the scope of an array in list form, a leading hyphen has no structural meaning and the line is classified by the remaining classes. 3. Array-header line – the content matches the header or keyed-header grammar of §6. A line whose first unquoted colon precedes its first unquoted "[" is never a header; it is a key-value line. Only unquoted occurrences count: a quoted key containing a colon can still open a header (e.g., `"a:b"[2]: 1,2` is a header), while `a:b[2]: x` is a key-value line with key `a`. 4. Key-value line – the content contains an unquoted colon and no earlier class applies. The key token precedes the first unquoted colon and is decoded per §7.4; the remainder after the colon is the value (§8). A line that contains an unquoted colon but fails the §6 header grammar falls through to this class (e.g., `foo [2]: bar`); the strict-mode header errors enumerated in §6 and §14.2 are unaffected by this fall-through. @@ -635,12 +635,12 @@ For an object appearing as a list item: - Tabs used as indentation MUST error (see §7.1 for tabs in quoted strings and as the HTAB delimiter). - Non-strict mode: - Depth MAY be computed as floor(indentSpaces / indentSize). - - Implementations MAY accept tab characters in indentation. When they do, leading tabs are indentation and MUST be removed from the line's content before classification (§5.2). Depth computation for tabs is implementation-defined and MUST be documented. + - Implementations MAY accept tab characters in indentation. When they do, leading tabs are indentation and MUST be removed from the line's content before classification (§5.2), and a line of only tabs and spaces counts as blank. Depth computation for tabs is implementation-defined and MUST be documented. - Trailing spaces: trailing spaces (U+0020) at the end of a line are not part of the line's content. Decoders MUST strip them after the CR exclusion above and before line classification (§5.2); a line whose content is `-` followed only by spaces is therefore the bare marker for an empty-object list item (§9.4, §10), not a list item carrying an empty token. - Token trimming: when a token is extracted – a key token before a key-value colon or an entry key's colon (§7.4, §9.5), a field entry in a field list (§6), or a value token after a key-value colon, after an array-header colon, or around each delimiter-separated token – decoders MUST trim surrounding spaces, exactly U+0020, no other characters. Any other whitespace (e.g., NBSP, or HTAB outside its delimiter role) is part of the token; internal semantics follow quoting rules. This trimming does not apply between a key and its bracket segment, where whitespace is a header syntax error (§6). - Comment lines are removed before any check in this section applies (§5.1). - Blank lines: - - A line whose content trims to empty is blank, regardless of leading-space count; the indentation checks above do not apply to blank lines. + - A line consisting of spaces only is blank, regardless of their count; the indentation checks above do not apply to blank lines. A line of tabs, or of spaces and tabs, is not blank: its tabs are indentation (above). - Header span: the lines from the first item, row, or entry line in a header's scope through the last line of that scope's content (which may be a deeper line inside its last item). A blank line inside any header span: in strict mode, MUST error; in non-strict mode, MAY be ignored and not counted as a row/item/entry. - All other blank lines – including between a header and the scope's first item, row, or entry line, and after a scope's content: in strict mode, decoders MUST ignore them (they do not create or close structures and are not counted); in non-strict mode, decoders SHOULD ignore them. - Trailing newline at end-of-file: decoders SHOULD accept; validators MAY warn. diff --git a/tests/fixtures/decode/blank-lines.json b/tests/fixtures/decode/blank-lines.json index 74e04cf..a476cee 100644 --- a/tests/fixtures/decode/blank-lines.json +++ b/tests/fixtures/decode/blank-lines.json @@ -227,6 +227,14 @@ "shouldError": true, "specSection": "14.2", "note": "The blank is after the nested object's content but inside the outer array's span (§12)" + }, + { + "name": "throws on a tab-only line, which is not blank", + "input": "a: 1\n\t\nb: 2", + "expected": null, + "shouldError": true, + "specSection": "14.2", + "minSpecVersion": "4.2" } ] } From 08a470ea828b7457c68c9891a512f6b19ac29acb Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 10:15:59 +0200 Subject: [PATCH 02/36] docs: accept raw control characters inside quoted strings --- SPEC.md | 4 ++-- tests/fixtures/decode/primitives.json | 9 +++++++++ 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/SPEC.md b/SPEC.md index b1c4059..ac52d6f 100644 --- a/SPEC.md +++ b/SPEC.md @@ -404,12 +404,12 @@ Normative escape grammar: ```abnf ; Core rules per RFC 5234 §B.1 (DIGIT, DQUOTE, HEXDIG); HEXDIG matches hex digits case-insensitively quoted-char = escaped-char / unescaped-char -unescaped-char = %x09 / %x20-21 / %x23-5B / %x5D-D7FF / %xE000-FFFF / %x10000-10FFFF +unescaped-char = %x00-09 / %x0B-21 / %x23-5B / %x5D-D7FF / %xE000-FFFF / %x10000-10FFFF escaped-char = %x5C ( %x5C / DQUOTE / %x6E / %x72 / %x74 / unicode-escape ) unicode-escape = %x75 4HEXDIG ``` -Tabs are allowed inside quoted strings and as a declared delimiter; they MUST NOT be used for indentation (§12). Within quoted strings, encoders MUST emit HTAB as `\t` per the escape table above; the literal HTAB in `unescaped-char` expresses decoder leniency only. +Tabs are allowed inside quoted strings and as a declared delimiter; they MUST NOT be used for indentation (§12). Within quoted strings, encoders MUST escape every U+0000–U+001F character per the escape table above; decoders accept the literal controls in `unescaped-char` in any mode. ### 7.2 Quoting Rules for String Values diff --git a/tests/fixtures/decode/primitives.json b/tests/fixtures/decode/primitives.json index 2347930..9a89d42 100644 --- a/tests/fixtures/decode/primitives.json +++ b/tests/fixtures/decode/primitives.json @@ -170,6 +170,15 @@ "input": "\"🚀 launch\"", "expected": "🚀 launch", "specSection": "7.1" + }, + { + "name": "decodes a raw control character inside a quoted string", + "input": "val: \"x\u0001y\"", + "expected": { + "val": "x\u0001y" + }, + "specSection": "7.1", + "minSpecVersion": "4.2" } ] } From b8ec40bc717c807e61334681a32411bf62e320f1 Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 10:15:59 +0200 Subject: [PATCH 03/36] docs: read a header-shaped line without a colon as a scalar line --- SPEC.md | 2 +- tests/fixtures/decode/arrays-nested.json | 11 +++++++++++ tests/fixtures/decode/root-form.json | 7 +++++++ tests/fixtures/decode/validation-errors.json | 4 ++-- 4 files changed, 21 insertions(+), 3 deletions(-) diff --git a/SPEC.md b/SPEC.md index ac52d6f..67b1cd2 100644 --- a/SPEC.md +++ b/SPEC.md @@ -373,7 +373,7 @@ Decoding requirements: - If a trailing tab or pipe appears inside the brackets, it selects the active delimiter; otherwise comma is active. - If a field list occurs between the bracket and the colon, parse field entries recursively using the active delimiter at every nesting level; quoted names MUST be unescaped per §7.1. Brace matching MUST ignore `{` and `}` inside quoted names. - A field list MUST contain at least one field entry at every nesting level: an empty field list (`{}`, including a nested `field{}`) is a header syntax error in strict mode; non-strict decoders MAY fall through to key-value parsing (§14.2). Unmatched braces in a field list are likewise header syntax errors. -- A colon MUST follow the bracket segment and optional field list; missing colon MUST error. +- A colon MUST follow the bracket segment and optional field list; missing colon MUST error. A line without any unquoted colon is neither a header nor a key context (§5.2): `items[2]` alone is a scalar line, and `- [2]` is a list item carrying the string `[2]`. - A non-keyed header without a field list: content after its colon is an inline primitive array (§9.1); nothing after the colon opens a block scope (§9.2, §9.4). A fields-bearing header – keyed or not – carries no inline content: in strict mode, non-whitespace content after its colon MUST error (§14.2); non-strict decoders MAY fall through to key-value parsing. - Keyless header positions: a keyless non-keyed header without a field list is valid only as the document's root header (§5) or as a list item after the `- ` marker (§9.2, §9.4); a keyless header with a field list – keyed or not – is valid only as the document's root header. In any other position, strict decoders MUST error (§14.2); non-strict decoders MAY parse the line as a key-value line, with the key treated as a literal token. diff --git a/tests/fixtures/decode/arrays-nested.json b/tests/fixtures/decode/arrays-nested.json index 7a549e9..cdf83ed 100644 --- a/tests/fixtures/decode/arrays-nested.json +++ b/tests/fixtures/decode/arrays-nested.json @@ -316,6 +316,17 @@ "specSection": "14.1", "note": "A legacy [0] is a declared length, so its content is decoded (§14.1)", "minSpecVersion": "4.1" + }, + { + "name": "parses a list item shaped like a header without a colon as a string", + "input": "a[1]:\n - [2]", + "expected": { + "a": [ + "[2]" + ] + }, + "specSection": "6", + "minSpecVersion": "4.2" } ] } diff --git a/tests/fixtures/decode/root-form.json b/tests/fixtures/decode/root-form.json index c5fc94c..2d7f029 100644 --- a/tests/fixtures/decode/root-form.json +++ b/tests/fixtures/decode/root-form.json @@ -99,6 +99,13 @@ "specSection": "5", "note": "The non-strict leniency for trailing content excludes scalar lines (§5.2)", "minSpecVersion": "4.1" + }, + { + "name": "parses a root line shaped like a header without a colon as a string", + "input": "items[2]", + "expected": "items[2]", + "specSection": "6", + "minSpecVersion": "4.2" } ] } diff --git a/tests/fixtures/decode/validation-errors.json b/tests/fixtures/decode/validation-errors.json index a55d2b8..b8a8be7 100644 --- a/tests/fixtures/decode/validation-errors.json +++ b/tests/fixtures/decode/validation-errors.json @@ -136,11 +136,11 @@ "specSection": "14.3" }, { - "name": "throws on array header missing colon", + "name": "throws on a header-shaped line without a colon above indented rows", "input": "items[2]{id,name}\n 1,Ada\n 2,Bob", "expected": null, "shouldError": true, - "specSection": "6" + "specSection": "5.2" }, { "name": "throws when inline values fall short of the declared length", From 92613366bbf170e0fd867db6c846a65fd61d4ca4 Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 10:15:59 +0200 Subject: [PATCH 04/36] docs: treat an indented first line as over-indented --- SPEC.md | 8 ++++---- tests/fixtures/decode/indentation-errors.json | 8 ++++++++ tests/fixtures/decode/root-form.json | 12 ++++++++++++ 3 files changed, 24 insertions(+), 4 deletions(-) diff --git a/SPEC.md b/SPEC.md index 67b1cd2..8da558f 100644 --- a/SPEC.md +++ b/SPEC.md @@ -276,8 +276,8 @@ TOON is a deterministic, line-oriented, indentation-based notation. - Root form discovery (applied to the comment-stripped line sequence, §5.1; line classes per §5.2): - If the first non-blank depth-0 line is a valid root array header per §6, decode a root array. - Else if the first non-blank depth-0 line is a valid keyless keyed header per §6 ([N:]{…}:), decode a root object in keyed tabular form (§9.5). - - Else if the first non-blank line is the literal token `[]`, decode an empty root array (§9.1). - - Else if the document has exactly one non-blank line and it is neither a valid array header nor a key-value line (quoted or unquoted key), decode a single primitive (examples: `hello`, `42`, `true`). + - Else if the first non-blank line is the literal token `[]` at depth 0, decode an empty root array (§9.1). + - Else if the document has exactly one non-blank line, at depth 0, and it is neither a valid array header nor a key-value line (quoted or unquoted key), decode a single primitive (examples: `hello`, `42`, `true`). - Otherwise, decode an object. - An empty document (no non-blank lines after comment removal, §5.1) decodes to an empty object `{}`. A document consisting only of comment and blank lines is therefore `{}`. - The root form spans the whole document: once a root array, an empty root array (`[]`), or a keyed tabular root object is complete, no further non-comment, non-blank line may follow. In strict mode, decoders MUST error on such trailing content (§14.2) – it MUST NOT be silently discarded. In non-strict mode, decoders MAY ignore it, except scalar lines, which are an error in any mode (§5.2). (A root object extends to the last line of the document, so this case does not arise for object roots.) @@ -458,7 +458,7 @@ Decoding of value tokens follows §4 (unquoted type inference, quoted strings, n - Lines in an object body are classified per §5.2; the rules below cover its key-value class. - A line "key:" with nothing after the colon at depth d opens an object; subsequent lines at depth > d belong to that object until the depth decreases to ≤ d. - In strict mode, the first line of a non-empty nested scope MUST be at exactly depth d+1; a depth increase of more than one level relative to the enclosing scope MUST error (§14.2). Conforming encoders never produce depth jumps; §10's depth model governs fields carried on a list-item hyphen line. - - A line deeper than the content depth of its enclosing scope whose preceding line did not open a scope belongs to no scope (e.g., a depth d+1 line directly under a depth-d primitive field). In strict mode, decoders MUST error (§14.2) – such lines MUST NOT be silently discarded. In non-strict mode, decoders MAY skip them, except scalar lines, which are an error in any mode (§5.2). + - A line deeper than the content depth of its enclosing scope whose preceding line did not open a scope belongs to no scope (e.g., a depth d+1 line directly under a depth-d primitive field, or the document's first line at depth 1 or more). In strict mode, decoders MUST error (§14.2) – such lines MUST NOT be silently discarded. In non-strict mode, decoders MAY skip them, except scalar lines, which are an error in any mode (§5.2). - A bare `key:` (no value after the colon) MUST decode as an empty or nested object, not an empty array. Empty arrays use the explicit `key: []` form (§9.1). - Lines "key: value" at the same depth are sibling fields. - Duplicate sibling keys at the same depth: see §14.3 for strict/non-strict behavior. @@ -734,7 +734,7 @@ When strict mode is enabled (default), decoders MUST error on the conditions lis - Any whitespace between a key and its bracket segment, and any content between a valid bracket segment and the colon (or field list), prevents array-header interpretation; decoders MUST NOT silently discard that content. In non-strict mode, decoders MAY fall through to key-value parsing. - Indentation and blank-line invariants per §12, evaluated after comment removal (§5.1). - Indentation depth jumps (§8): a line more than one level deeper than its enclosing scope (e.g., a depth d+2 line directly under a depth-d parent). -- Over-indented lines (§8): a line deeper than the content depth of its enclosing scope when the preceding line did not open a scope (e.g., a depth d+1 line directly under a depth-d primitive field). Decoders MUST NOT silently discard such lines. +- Over-indented lines (§8): a line deeper than the content depth of its enclosing scope when the preceding line did not open a scope (e.g., a depth d+1 line directly under a depth-d primitive field, or the document's first line at depth 1 or more). Decoders MUST NOT silently discard such lines. - Trailing content after a completed root form (§5): any non-comment, non-blank line following the inline values, rows, items, or entries of a root array or keyed tabular root object, or following a root `[]`. - Ill-formed UTF-8 in byte input (§4). - A scalar line (§5.2) anywhere other than root primitive position – e.g., a bare token line inside an array or object scope (any mode; §5.2). diff --git a/tests/fixtures/decode/indentation-errors.json b/tests/fixtures/decode/indentation-errors.json index 2c32119..bedf5ca 100644 --- a/tests/fixtures/decode/indentation-errors.json +++ b/tests/fixtures/decode/indentation-errors.json @@ -208,6 +208,14 @@ "specSection": "8", "note": "The non-strict skip for over-indented lines excludes scalar lines (§5.2)", "minSpecVersion": "4.1" + }, + { + "name": "throws on an indented first line", + "input": " a: 1", + "expected": null, + "shouldError": true, + "specSection": "8", + "minSpecVersion": "4.2" } ] } diff --git a/tests/fixtures/decode/root-form.json b/tests/fixtures/decode/root-form.json index 2d7f029..c1203c1 100644 --- a/tests/fixtures/decode/root-form.json +++ b/tests/fixtures/decode/root-form.json @@ -106,6 +106,18 @@ "expected": "items[2]", "specSection": "6", "minSpecVersion": "4.2" + }, + { + "name": "throws on an indented root primitive in non-strict mode", + "input": " hello", + "expected": null, + "shouldError": true, + "options": { + "strict": false + }, + "specSection": "5", + "note": "The non-strict skip for over-indented lines excludes scalar lines (§5.2)", + "minSpecVersion": "4.2" } ] } From 578a34d34387d8beba6ca1cf90371d76ce557362 Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 10:16:00 +0200 Subject: [PATCH 05/36] docs: limit non-strict decoders to the leniencies the spec names --- SPEC.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/SPEC.md b/SPEC.md index 8da558f..0d345a6 100644 --- a/SPEC.md +++ b/SPEC.md @@ -238,7 +238,7 @@ See Appendix E for non-normative language-specific examples. Decoders map text tokens to host values: -- Byte input: decoders that accept bytes MUST decode them as UTF-8. In strict mode, ill-formed UTF-8 (invalid or truncated sequences, or bytes encoding surrogate code points) MUST error; it MUST NOT be silently replaced with U+FFFD. Decoders that accept host strings (already decoded from bytes) are outside this rule. +- Byte input: decoders that accept bytes MUST decode them as UTF-8. In strict mode, ill-formed UTF-8 (invalid or truncated sequences, or bytes encoding surrogate code points) MUST error; it MUST NOT be silently replaced with U+FFFD. Non-strict decoders MAY instead replace each ill-formed sequence with U+FFFD. Decoders that accept host strings (already decoded from bytes) are outside this rule. - Quoted tokens (strings and keys): - MUST be unescaped per §7.1. Any other escape or an unterminated string MUST error. - Quoted primitives remain strings even if they look like numbers/booleans/null. @@ -709,7 +709,7 @@ Validators SHOULD verify: ## 14. Strict Mode Errors and Diagnostics (Authoritative Checklist) -When strict mode is enabled (default), decoders MUST error on the conditions listed below. Conditions marked "(any mode)" are errors in strict and non-strict mode alike. Error type, code, and message text are implementation-defined. +When strict mode is enabled (default), decoders MUST error on the conditions listed below. Conditions marked "(any mode)" are errors in strict and non-strict mode alike. For every other condition, a non-strict decoder applies the leniency this specification names for it, or errors where that leniency is optional; it MUST NOT recover in any other way. Error type, code, and message text are implementation-defined. ### 14.1 Array Count and Width Mismatches From e3efde3ec12a6202ebf47d7dd6bdd7aef2bbe2b8 Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 10:16:00 +0200 Subject: [PATCH 06/36] docs: let non-strict decoders adopt the depth of a jumped first line --- SPEC.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/SPEC.md b/SPEC.md index 0d345a6..31822f3 100644 --- a/SPEC.md +++ b/SPEC.md @@ -138,7 +138,7 @@ All normative text is in Sections 1–16; the appendices and all examples are in - Indentation level (depth): Leading indentation measured in fixed-size space units (indentSize). Depth 0 has no indentation. - Indentation unit (indentSize): A fixed number of spaces per level (default 2). Tabs MUST NOT be used for indentation. -- Content depth: The depth at which a scope's immediate content appears – 0 for the root scope, otherwise one level deeper than the depth at which the scope's opening line stands (see §10 for first fields carried on a list-item hyphen line). +- Content depth: The depth at which a scope's immediate content appears – 0 for the root scope, otherwise one level deeper than the depth at which the scope's opening line stands (see §10 for first fields carried on a list-item hyphen line, and §8 for the non-strict depth-jump leniency). - Row depth, entry depth: The content depth of a tabular array's or keyed tabular object's scope, at which its rows or entry rows appear (§9.3, §9.5). ### 1.4 Array and Tabular Terms @@ -457,7 +457,7 @@ Decoding of value tokens follows §4 (unquoted type inference, quoted strings, n - Decoding: - Lines in an object body are classified per §5.2; the rules below cover its key-value class. - A line "key:" with nothing after the colon at depth d opens an object; subsequent lines at depth > d belong to that object until the depth decreases to ≤ d. - - In strict mode, the first line of a non-empty nested scope MUST be at exactly depth d+1; a depth increase of more than one level relative to the enclosing scope MUST error (§14.2). Conforming encoders never produce depth jumps; §10's depth model governs fields carried on a list-item hyphen line. + - In strict mode, the first line of a non-empty nested scope MUST be at exactly depth d+1; a depth increase of more than one level relative to the enclosing scope MUST error (§14.2). In non-strict mode, decoders MAY instead take the first line's depth as the scope's content depth; a later line deeper than d but shallower than that depth is then treated as over-indented (below). Conforming encoders never produce depth jumps; §10's depth model governs fields carried on a list-item hyphen line. - A line deeper than the content depth of its enclosing scope whose preceding line did not open a scope belongs to no scope (e.g., a depth d+1 line directly under a depth-d primitive field, or the document's first line at depth 1 or more). In strict mode, decoders MUST error (§14.2) – such lines MUST NOT be silently discarded. In non-strict mode, decoders MAY skip them, except scalar lines, which are an error in any mode (§5.2). - A bare `key:` (no value after the colon) MUST decode as an empty or nested object, not an empty array. Empty arrays use the explicit `key: []` form (§9.1). - Lines "key: value" at the same depth are sibling fields. @@ -733,7 +733,7 @@ When strict mode is enabled (default), decoders MUST error on the conditions lis - Keyless headers outside their valid positions (§6): a keyless non-keyed header in object-field position (e.g., `[2]: x,y` under an object field, or as a non-first depth-0 line), or a keyless fields-bearing header as a list item (`- [2]{a}:`). - Any whitespace between a key and its bracket segment, and any content between a valid bracket segment and the colon (or field list), prevents array-header interpretation; decoders MUST NOT silently discard that content. In non-strict mode, decoders MAY fall through to key-value parsing. - Indentation and blank-line invariants per §12, evaluated after comment removal (§5.1). -- Indentation depth jumps (§8): a line more than one level deeper than its enclosing scope (e.g., a depth d+2 line directly under a depth-d parent). +- Indentation depth jumps (§8): the first line of a nested scope standing more than one level deeper than the line that opens it (e.g., a depth d+2 line directly under a depth-d `key:`). - Over-indented lines (§8): a line deeper than the content depth of its enclosing scope when the preceding line did not open a scope (e.g., a depth d+1 line directly under a depth-d primitive field, or the document's first line at depth 1 or more). Decoders MUST NOT silently discard such lines. - Trailing content after a completed root form (§5): any non-comment, non-blank line following the inline values, rows, items, or entries of a root array or keyed tabular root object, or following a root `[]`. - Ill-formed UTF-8 in byte input (§4). From 0d3edeec0dffb0765e7f9bb95ef433e9e33fcc70 Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 10:16:00 +0200 Subject: [PATCH 07/36] docs: keep a header whose length exceeds the integer range --- SPEC.md | 2 +- tests/fixtures/decode/arrays-primitive.json | 14 ++++++++++++++ tests/fixtures/decode/validation-errors.json | 9 +++++++++ 3 files changed, 24 insertions(+), 1 deletion(-) diff --git a/SPEC.md b/SPEC.md index 31822f3..7bb101e 100644 --- a/SPEC.md +++ b/SPEC.md @@ -366,7 +366,7 @@ Note: The grammar above specifies header syntax only. Tabular row disambiguation Whitespace MUST NOT appear between a key and its bracket segment, and content MUST NOT appear between `]` and `{`/`:` (e.g., `foo [2]:`, `[1][bar]:`, `[2]extra:`, `[2] :`). Such intervening whitespace or content, and the malformed bracket segments enumerated in the decoding requirements below, are strict-mode errors; non-strict decoders MAY parse the line as a key-value line, with the key treated as a literal token. Decoding requirements: -- The bracket segment MUST parse as a non-negative integer length N with no leading zeros (the single digit `0` is the only canonical form for length zero). Tokens like `[03]` or `[-1]` MUST NOT be interpreted as bracket segments. +- The bracket segment MUST parse as a non-negative integer length N with no leading zeros (the single digit `0` is the only canonical form for length zero). Tokens like `[03]` or `[-1]` MUST NOT be interpreted as bracket segments. N has no upper bound: a length the implementation cannot represent still forms a header, and its count is unmet (§14.1). - A bracket segment without a length token (`key[]:`) is not a header: strict mode MUST error; non-strict decoders MAY fall through to key-value parsing. This does not affect the empty-array value form `key: []` (§9.1), where `[]` follows the colon. - A colon immediately after the length and before the optional delimiter symbol marks a keyed header (§9.5): `[N:]` declares comma, `[N:]` tab, `[N:|]` pipe. The colon MUST occupy exactly that position – tokens such as `[2|:]`, `[2 :]`, or `[2:,]` are malformed bracket segments, and the length rules above apply unchanged (`[03:]` is malformed). - A keyed header MUST carry a field list: `key[2:]:` without braces is a header syntax error in strict mode; non-strict decoders MAY fall through to key-value parsing (§14.2). diff --git a/tests/fixtures/decode/arrays-primitive.json b/tests/fixtures/decode/arrays-primitive.json index f84fb8c..cffcd8e 100644 --- a/tests/fixtures/decode/arrays-primitive.json +++ b/tests/fixtures/decode/arrays-primitive.json @@ -208,6 +208,20 @@ ] }, "specSection": "9.3" + }, + { + "name": "keeps a header whose length exceeds integer range in non-strict mode", + "input": "a[99999999999999999999]: 1", + "expected": { + "a": [ + 1 + ] + }, + "options": { + "strict": false + }, + "specSection": "6", + "minSpecVersion": "4.2" } ] } diff --git a/tests/fixtures/decode/validation-errors.json b/tests/fixtures/decode/validation-errors.json index b8a8be7..2ce1e27 100644 --- a/tests/fixtures/decode/validation-errors.json +++ b/tests/fixtures/decode/validation-errors.json @@ -631,6 +631,15 @@ "shouldError": true, "specSection": "9.1", "note": "Nothing after the colon is list form with zero items, not one empty inline value" + }, + { + "name": "throws on a length beyond the integer range that wraps to the value count", + "input": "a[18446744073709551617]: 1", + "expected": null, + "shouldError": true, + "specSection": "6", + "note": "2^64 + 1 wraps to 1 in 64-bit integers; an unrepresentable length is unmet (§14.1)", + "minSpecVersion": "4.2" } ] } From b447f328efb0a64de38f162f1796688c470f765c Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 10:16:00 +0200 Subject: [PATCH 08/36] docs: decode an empty key token as the empty key --- SPEC.md | 2 +- tests/fixtures/decode/objects.json | 9 +++++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/SPEC.md b/SPEC.md index 7bb101e..d16e2db 100644 --- a/SPEC.md +++ b/SPEC.md @@ -442,7 +442,7 @@ Decoding of value tokens follows §4 (unquoted type inference, quoted strings, n - Quoted keys MUST be unescaped per §7.1; any other escape MUST error. - Keys (quoted or unquoted) MUST be followed by ":", optionally after spaces (§12); missing colon MUST error (see also §14.2). -- Unquoted key token (normative): an unquoted key token is the text before the first unquoted colon of a key-value line (§5.2) or entry row (§9.5), with surrounding spaces trimmed (§12); the text before a header's bracket segment; or a field name in a field list (§6). Decoders MUST accept any such token as a literal key, in strict and non-strict mode alike, even when it does not match §7.3's unquoted-key pattern: `foo-bar: 1`, `foo-bar[2]: 1,2`, and `items[1]{2key}:` are valid input. §7.3 constrains what encoders may emit unquoted, not what decoders accept. +- Unquoted key token (normative): an unquoted key token is the text before the first unquoted colon of a key-value line (§5.2) or entry row (§9.5), with surrounding spaces trimmed (§12); the text before a header's bracket segment; or a field name in a field list (§6). An empty token before a key-value or entry-row colon is the empty key: `: 1` decodes to `{"": 1}`. Decoders MUST accept any such token as a literal key, in strict and non-strict mode alike, even when it does not match §7.3's unquoted-key pattern: `foo-bar: 1`, `foo-bar[2]: 1,2`, and `items[1]{2key}:` are valid input. §7.3 constrains what encoders may emit unquoted, not what decoders accept. - Quoted-token boundary (normative): a token whose first character, after the trimming of §12, is `"` MUST be a complete quoted token – its closing `"` MUST be the token's last character. This applies wherever a token is extracted; any character after the closing quote MUST error. It overrides §4's "Otherwise → string" fallback. - Symmetrically for values: an unquoted value token that an encoder would have been required to quote (§7.2) is not an error. Decoders, strict mode included, MUST decode it per §4 – unless another rule of this specification assigns the token structural meaning (§5.2, §6, §9.1). Example: `key: -x` decodes to the string `-x`. §7.2 governs encoder output; it adds no decoder-side rejection. diff --git a/tests/fixtures/decode/objects.json b/tests/fixtures/decode/objects.json index a53710c..0c8d69b 100644 --- a/tests/fixtures/decode/objects.json +++ b/tests/fixtures/decode/objects.json @@ -585,6 +585,15 @@ }, "specSection": "16", "minSpecVersion": "4.1" + }, + { + "name": "parses an empty unquoted key as the empty key", + "input": ": 1", + "expected": { + "": 1 + }, + "specSection": "7.4", + "minSpecVersion": "4.2" } ] } From ba8ccd96010e16937f96a57d5ac8665f84f93cb5 Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 10:16:00 +0200 Subject: [PATCH 09/36] docs: reject an empty field entry --- SPEC.md | 4 ++-- tests/fixtures/decode/validation-errors.json | 8 ++++++++ 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/SPEC.md b/SPEC.md index d16e2db..4d0d5f7 100644 --- a/SPEC.md +++ b/SPEC.md @@ -372,7 +372,7 @@ Decoding requirements: - A keyed header MUST carry a field list: `key[2:]:` without braces is a header syntax error in strict mode; non-strict decoders MAY fall through to key-value parsing (§14.2). - If a trailing tab or pipe appears inside the brackets, it selects the active delimiter; otherwise comma is active. - If a field list occurs between the bracket and the colon, parse field entries recursively using the active delimiter at every nesting level; quoted names MUST be unescaped per §7.1. Brace matching MUST ignore `{` and `}` inside quoted names. -- A field list MUST contain at least one field entry at every nesting level: an empty field list (`{}`, including a nested `field{}`) is a header syntax error in strict mode; non-strict decoders MAY fall through to key-value parsing (§14.2). Unmatched braces in a field list are likewise header syntax errors. +- A field list MUST contain at least one field entry at every nesting level, and no field entry or field name may be empty: an empty field list (`{}`, including a nested `field{}`), an empty field entry (`{a,}`), or a nameless nested group (`{a,{b}}`) is a header syntax error in strict mode; non-strict decoders MAY fall through to key-value parsing (§14.2). Unmatched braces in a field list are likewise header syntax errors, with the same non-strict fall-through. - A colon MUST follow the bracket segment and optional field list; missing colon MUST error. A line without any unquoted colon is neither a header nor a key context (§5.2): `items[2]` alone is a scalar line, and `- [2]` is a list item carrying the string `[2]`. - A non-keyed header without a field list: content after its colon is an inline primitive array (§9.1); nothing after the colon opens a block scope (§9.2, §9.4). A fields-bearing header – keyed or not – carries no inline content: in strict mode, non-whitespace content after its colon MUST error (§14.2); non-strict decoders MAY fall through to key-value parsing. - Keyless header positions: a keyless non-keyed header without a field list is valid only as the document's root header (§5) or as a list item after the `- ` marker (§9.2, §9.4); a keyless header with a field list – keyed or not – is valid only as the document's root header. In any other position, strict decoders MUST error (§14.2); non-strict decoders MAY parse the line as a key-value line, with the key treated as a literal token. @@ -727,7 +727,7 @@ When strict mode is enabled (default), decoders MUST error on the conditions lis - Invalid escape sequences or unterminated strings in quoted tokens, and characters after a quoted token's closing quote (any mode; §4, §7.1, §7.4). - Header delimiter mismatch (§6): MUST error as a header syntax error, independent of row width/count checks. - Malformed bracket lengths and malformed keyed markers in headers (§6). -- Malformed field lists in headers: an empty field list (`{}`, including a nested `field{}`), unmatched braces, or a field name repeated within the same field list (`{a,a}`, including inside a nested group); see §6, §9.3, §9.5. These are diagnosed from the header line alone, independent of the declared count and of any following rows or entry rows. +- Malformed field lists in headers: an empty field list (`{}`, including a nested `field{}`), an empty field entry (`{a,}`), unmatched braces, or a field name repeated within the same field list (`{a,a}`, including inside a nested group); see §6, §9.3, §9.5. These are diagnosed from the header line alone, independent of the declared count and of any following rows or entry rows. - Keyed headers (§9.5): a missing field list (`key[2:]:`), a keyless keyed header anywhere other than as the document's root header, or a line at entry depth without an unquoted colon. - Non-whitespace content after a fields-bearing header's colon (§6), keyed or not (e.g., `items[2]{a,b}: 1,2`). - Keyless headers outside their valid positions (§6): a keyless non-keyed header in object-field position (e.g., `[2]: x,y` under an object field, or as a non-first depth-0 line), or a keyless fields-bearing header as a list item (`- [2]{a}:`). diff --git a/tests/fixtures/decode/validation-errors.json b/tests/fixtures/decode/validation-errors.json index 2ce1e27..570b2f4 100644 --- a/tests/fixtures/decode/validation-errors.json +++ b/tests/fixtures/decode/validation-errors.json @@ -640,6 +640,14 @@ "specSection": "6", "note": "2^64 + 1 wraps to 1 in 64-bit integers; an unrepresentable length is unmet (§14.1)", "minSpecVersion": "4.2" + }, + { + "name": "throws on an empty field entry", + "input": "a[0]{x,}:", + "expected": null, + "shouldError": true, + "specSection": "6", + "minSpecVersion": "4.2" } ] } From bde9666120c4632dbfd104aadef0faea3fb259dc Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 10:16:00 +0200 Subject: [PATCH 10/36] docs: give a leading hyphen meaning only at list item depth --- SPEC.md | 2 +- tests/fixtures/decode/objects.json | 14 ++++++++++++++ 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/SPEC.md b/SPEC.md index 4d0d5f7..e87b5d9 100644 --- a/SPEC.md +++ b/SPEC.md @@ -303,7 +303,7 @@ Quoting keeps "#"-leading data out of the comment rule: string values that equal Decoders classify each line of the comment-stripped sequence (§5.1) by its content after the leading indentation. The first matching class applies. Classification is lexical; whether a class is admissible at a given depth and position is determined by the enclosing construct (root form above, §8–§10). Within a tabular array's scope, lines at row depth are divided between the row and key-value classes by the disambiguation rules of §9.3, which are authoritative for that position and take precedence over the order below. Within a keyed tabular object's scope, every line at entry depth containing an unquoted colon is an entry row; §9.5 is likewise authoritative for that position. 1. Blank line – the line consists of spaces only (§12). Handled per §12; blank lines never create or close structure. -2. List-item line – the content is the bare marker "-" or begins with "- " (hyphen, space). The remainder after the marker is parsed per §9.2, §9.4, and §10. Outside the scope of an array in list form, a leading hyphen has no structural meaning and the line is classified by the remaining classes. +2. List-item line – the content is the bare marker "-" or begins with "- " (hyphen, space). The remainder after the marker is parsed per §9.2, §9.4, and §10. A leading hyphen marks a list item only at the item depth of an array in list form; elsewhere – including a list-item object's field depth (§10) – the line is classified by the remaining classes. 3. Array-header line – the content matches the header or keyed-header grammar of §6. A line whose first unquoted colon precedes its first unquoted "[" is never a header; it is a key-value line. Only unquoted occurrences count: a quoted key containing a colon can still open a header (e.g., `"a:b"[2]: 1,2` is a header), while `a:b[2]: x` is a key-value line with key `a`. 4. Key-value line – the content contains an unquoted colon and no earlier class applies. The key token precedes the first unquoted colon and is decoded per §7.4; the remainder after the colon is the value (§8). A line that contains an unquoted colon but fails the §6 header grammar falls through to this class (e.g., `foo [2]: bar`); the strict-mode header errors enumerated in §6 and §14.2 are unaffected by this fall-through. 5. Row line – within a tabular array's scope, a delimiter-separated value line at row depth (§9.3); within a keyed tabular object's scope, an entry row at entry depth (§9.5). diff --git a/tests/fixtures/decode/objects.json b/tests/fixtures/decode/objects.json index 0c8d69b..b9577b6 100644 --- a/tests/fixtures/decode/objects.json +++ b/tests/fixtures/decode/objects.json @@ -594,6 +594,20 @@ }, "specSection": "7.4", "minSpecVersion": "4.2" + }, + { + "name": "parses a hyphen-leading line at list-item field depth as a key-value line", + "input": "items[1]:\n - a: 1\n - y: 2", + "expected": { + "items": [ + { + "a": 1, + "- y": 2 + } + ] + }, + "specSection": "5.2", + "minSpecVersion": "4.2" } ] } From a37cbeb15e28c57296a23a6e5c7a008a3b0d6361 Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 10:16:00 +0200 Subject: [PATCH 11/36] docs: require the shortest round-trip digits for numbers --- SPEC.md | 2 +- tests/fixtures/encode/primitives.json | 8 ++++++++ 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/SPEC.md b/SPEC.md index e87b5d9..14e3a2a 100644 --- a/SPEC.md +++ b/SPEC.md @@ -206,7 +206,7 @@ Row, entry, and item terms: - If the fractional part is zero after normalization, emit as an integer (e.g., 1.0 → 1). - -0 MUST be normalized to 0. - For finite numbers outside the canonical range above (non-zero |n| < 1e-6, or |n| ≥ 1e21), encoders MAY emit exponent notation conforming to the JSON number grammar [RFC8259] §6 (e.g., 1e-7, 1e+21). Encoders SHOULD use lowercase `e` and an explicit exponent sign for byte-for-byte determinism. - - Encoders MUST emit sufficient precision so that, after any §3 host-type normalization, decode(encode(x)) equals x under the JSON-model equality defined below. + - Encoders MUST emit the fewest significant digits for which, after any §3 host-type normalization, decode(encode(x)) equals x under the JSON-model equality defined below; when several candidates qualify, they MUST emit the one closest to x, ties to even (as ECMAScript's Number::toString does for binary64). - If a source value is outside the implementation's documented numeric domain (e.g., arbitrary-precision decimals or integers exceeding that domain), the encoder MAY: - Emit a quoted string containing a lossless decimal representation (plain decimal or JSON exponent form); the chosen form MUST be documented. - Emit a number that round-trips to the host's numeric approximation (losing precision), provided it conforms to the rules above. diff --git a/tests/fixtures/encode/primitives.json b/tests/fixtures/encode/primitives.json index 826c95e..0f12f0c 100644 --- a/tests/fixtures/encode/primitives.json +++ b/tests/fixtures/encode/primitives.json @@ -281,6 +281,14 @@ "specSection": "7.2", "note": "Unquoted, a decoder would remove the U+FEFF as a byte-order mark (§12) and read the number 8", "minSpecVersion": "4.1" + }, + { + "name": "encodes a large integer-valued number with its shortest round-trip digits", + "input": 123456789012345680000, + "expected": "123456789012345680000", + "specSection": "2", + "note": "The binary64 value is exactly 123456789012345683968", + "minSpecVersion": "4.2" } ] } From 6342833bcaa1d7cc9fabc630768fb5c01f732a64 Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 10:16:01 +0200 Subject: [PATCH 12/36] docs: require unquoted output wherever quoting is not required --- SPEC.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/SPEC.md b/SPEC.md index 14e3a2a..be6d105 100644 --- a/SPEC.md +++ b/SPEC.md @@ -426,12 +426,12 @@ Encoders MUST quote a string value if any of the following is true: - It equals "#" or starts with "#" (any number sign at position 0). - It is a root primitive (§5) and starts with U+FEFF (§12). -Otherwise, the string MAY be emitted without quotes. Unicode, emoji, and strings with internal (non-leading/trailing) spaces are safe unquoted provided they do not violate the conditions. +Otherwise, encoders MUST emit the string without quotes. Unicode, emoji, and strings with internal (non-leading/trailing) spaces are safe unquoted provided they do not violate the conditions. ### 7.3 Key Encoding Object keys (including entry keys, §9.5) and the field names in a header's field list: -- MAY be unquoted only if they match: `^[A-Za-z_][A-Za-z0-9_.]*$`. +- MUST be unquoted if they match: `^[A-Za-z_][A-Za-z0-9_.]*$`. - Otherwise, they MUST be quoted and escaped per §7.1. Keys requiring quoting per the above rules MUST be quoted in all contexts, including array headers (e.g., "my-key"[N]:). @@ -442,7 +442,7 @@ Decoding of value tokens follows §4 (unquoted type inference, quoted strings, n - Quoted keys MUST be unescaped per §7.1; any other escape MUST error. - Keys (quoted or unquoted) MUST be followed by ":", optionally after spaces (§12); missing colon MUST error (see also §14.2). -- Unquoted key token (normative): an unquoted key token is the text before the first unquoted colon of a key-value line (§5.2) or entry row (§9.5), with surrounding spaces trimmed (§12); the text before a header's bracket segment; or a field name in a field list (§6). An empty token before a key-value or entry-row colon is the empty key: `: 1` decodes to `{"": 1}`. Decoders MUST accept any such token as a literal key, in strict and non-strict mode alike, even when it does not match §7.3's unquoted-key pattern: `foo-bar: 1`, `foo-bar[2]: 1,2`, and `items[1]{2key}:` are valid input. §7.3 constrains what encoders may emit unquoted, not what decoders accept. +- Unquoted key token (normative): an unquoted key token is the text before the first unquoted colon of a key-value line (§5.2) or entry row (§9.5), with surrounding spaces trimmed (§12); the text before a header's bracket segment; or a field name in a field list (§6). An empty token before a key-value or entry-row colon is the empty key: `: 1` decodes to `{"": 1}`. Decoders MUST accept any such token as a literal key, in strict and non-strict mode alike, even when it does not match §7.3's unquoted-key pattern: `foo-bar: 1`, `foo-bar[2]: 1,2`, and `items[1]{2key}:` are valid input. §7.3 governs how encoders emit keys, not what decoders accept. - Quoted-token boundary (normative): a token whose first character, after the trimming of §12, is `"` MUST be a complete quoted token – its closing `"` MUST be the token's last character. This applies wherever a token is extracted; any character after the closing quote MUST error. It overrides §4's "Otherwise → string" fallback. - Symmetrically for values: an unquoted value token that an encoder would have been required to quote (§7.2) is not an error. Decoders, strict mode included, MUST decode it per §4 – unless another rule of this specification assigns the token structural meaning (§5.2, §6, §9.1). Example: `key: -x` decodes to the string `-x`. §7.2 governs encoder output; it adds no decoder-side rejection. @@ -670,7 +670,7 @@ Conforming encoders MUST: - [ ] Use consistent indentation (default 2 spaces, no tabs) (§12) - [ ] Escape per §7.1 in quoted strings; never emit other escapes - [ ] Quote strings per §7.2 (the relevant delimiter is governed by §11.1: document delimiter for object-field values, active delimiter for inline array values, tabular row cells, and keyed entry-row cells) -- [ ] Quote and escape object keys, entry keys, and field names that do not match §7.3's unquoted-key pattern (§7.3) +- [ ] Quote and escape exactly those object keys, entry keys, and field names that do not match §7.3's unquoted-key pattern (§7.3) - [ ] Select the form from the value's shape and position, not by preference (§1.4, §9) - [ ] Emit declared lengths [N] matching the actual inline value, list item, tabular row, or entry row count (§6, §9) - [ ] Preserve object key order as encountered, except where tabular forms reorder to the header's field order (§2) From 0ae2d870f943a56ff2c0b600e04ff5853d56db69 Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 10:16:01 +0200 Subject: [PATCH 13/36] docs: quote primitive list items by the document delimiter --- SPEC.md | 8 ++++---- tests/fixtures/encode/delimiters.json | 18 ++++++++++++++++++ 2 files changed, 22 insertions(+), 4 deletions(-) diff --git a/SPEC.md b/SPEC.md index be6d105..2d23420 100644 --- a/SPEC.md +++ b/SPEC.md @@ -421,7 +421,7 @@ Encoders MUST quote a string value if any of the following is true: - It contains a colon (:), double quote ("), or backslash (\\). - It contains brackets or braces ([, ], {, }). - It contains control characters in U+0000 through U+001F. -- It contains the relevant delimiter – the active delimiter for inline array values, tabular row cells, and keyed entry-row cells; the document delimiter for object field values (§11.1, which is authoritative for delimiter-aware quoting). +- It contains the relevant delimiter – the active delimiter for inline array values, tabular row cells, and keyed entry-row cells; the document delimiter for object field values, primitive list items, and root primitives (§11.1, which is authoritative for delimiter-aware quoting). - It equals "-" or starts with "-" (any hyphen at position 0). - It equals "#" or starts with "#" (any number sign at position 0). - It is a root primitive (§5) and starts with U+FEFF (§12). @@ -603,11 +603,11 @@ For an object appearing as a list item: ### 11.1 Encoding Rules -- Document delimiter: Encoders select a document delimiter (option: comma, tab, pipe; default comma). Encoders MUST declare it as the active delimiter of every header they emit (§6); it also governs delimiter-aware quoting for object field values (key: value) and root primitives. +- Document delimiter: Encoders select a document delimiter (option: comma, tab, pipe; default comma). Encoders MUST declare it as the active delimiter of every header they emit (§6); it also governs delimiter-aware quoting for object field values (key: value), primitive list items (`- value`), and root primitives. - Active delimiter: Inside a header's scope, the active delimiter governs quoting only for inline array values, tabular row cells, and keyed entry-row cells (§9.5). - Delimiter-aware quoting: - Inline array values, tabular row cells, and keyed entry-row cells: strings containing the active delimiter MUST be quoted. - - Object field values (key: value): encoders use the document delimiter to decide delimiter-aware quoting, regardless of whether the object appears within an array's scope. Entry rows (§9.5) are not object-field lines: the content after the entry key's colon is a delimiter-joined cell sequence, quoted per the active delimiter. + - Object field values (key: value) and primitive list items (`- value`): encoders use the document delimiter to decide delimiter-aware quoting, regardless of the enclosing array's scope. Entry rows (§9.5) are not object-field lines: the content after the entry key's colon is a delimiter-joined cell sequence, quoted per the active delimiter. - Strings containing non-active delimiters do not require quoting unless another condition applies (§7.2). ### 11.2 Decoding Rules @@ -669,7 +669,7 @@ Conforming encoders MUST: - [ ] Produce UTF-8 output with LF (U+000A) line endings (§1.2) - [ ] Use consistent indentation (default 2 spaces, no tabs) (§12) - [ ] Escape per §7.1 in quoted strings; never emit other escapes -- [ ] Quote strings per §7.2 (the relevant delimiter is governed by §11.1: document delimiter for object-field values, active delimiter for inline array values, tabular row cells, and keyed entry-row cells) +- [ ] Quote strings per §7.2 (the relevant delimiter is governed by §11.1: document delimiter for object-field values and primitive list items, active delimiter for inline array values, tabular row cells, and keyed entry-row cells) - [ ] Quote and escape exactly those object keys, entry keys, and field names that do not match §7.3's unquoted-key pattern (§7.3) - [ ] Select the form from the value's shape and position, not by preference (§1.4, §9) - [ ] Emit declared lengths [N] matching the actual inline value, list item, tabular row, or entry row count (§6, §9) diff --git a/tests/fixtures/encode/delimiters.json b/tests/fixtures/encode/delimiters.json index 03c9271..9954553 100644 --- a/tests/fixtures/encode/delimiters.json +++ b/tests/fixtures/encode/delimiters.json @@ -248,6 +248,24 @@ "delimiter": "|" }, "specSection": "11" + }, + { + "name": "quotes a primitive list item containing the document delimiter", + "input": { + "a": [ + "x|y", + "p,q", + { + "b": 1 + } + ] + }, + "expected": "a[3|]:\n - \"x|y\"\n - p,q\n - b: 1", + "options": { + "delimiter": "|" + }, + "specSection": "11.1", + "minSpecVersion": "4.2" } ] } From 30b26b342383ee3e0ec572a67e440ac3ef5c3516 Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 10:16:01 +0200 Subject: [PATCH 14/36] docs: reject a space between a field name and its nested field group --- SPEC.md | 6 +++--- tests/fixtures/decode/validation-errors.json | 8 ++++++++ 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/SPEC.md b/SPEC.md index 2d23420..8543d00 100644 --- a/SPEC.md +++ b/SPEC.md @@ -363,7 +363,7 @@ The ABNF does not express delimiter equality between the `bracket-seg` and `fiel Note: The grammar above specifies header syntax only. Tabular row disambiguation is defined in §9.3. -Whitespace MUST NOT appear between a key and its bracket segment, and content MUST NOT appear between `]` and `{`/`:` (e.g., `foo [2]:`, `[1][bar]:`, `[2]extra:`, `[2] :`). Such intervening whitespace or content, and the malformed bracket segments enumerated in the decoding requirements below, are strict-mode errors; non-strict decoders MAY parse the line as a key-value line, with the key treated as a literal token. +Whitespace MUST NOT appear between a key and its bracket segment or between a field name and its nested field group, and content MUST NOT appear between `]` and `{`/`:` (e.g., `foo [2]:`, `{b {c}}`, `[1][bar]:`, `[2]extra:`, `[2] :`). Such intervening whitespace or content, and the malformed bracket segments enumerated in the decoding requirements below, are strict-mode errors; non-strict decoders MAY parse the line as a key-value line, with the key treated as a literal token. Decoding requirements: - The bracket segment MUST parse as a non-negative integer length N with no leading zeros (the single digit `0` is the only canonical form for length zero). Tokens like `[03]` or `[-1]` MUST NOT be interpreted as bracket segments. N has no upper bound: a length the implementation cannot represent still forms a header, and its count is unmet (§14.1). @@ -637,7 +637,7 @@ For an object appearing as a list item: - Depth MAY be computed as floor(indentSpaces / indentSize). - Implementations MAY accept tab characters in indentation. When they do, leading tabs are indentation and MUST be removed from the line's content before classification (§5.2), and a line of only tabs and spaces counts as blank. Depth computation for tabs is implementation-defined and MUST be documented. - Trailing spaces: trailing spaces (U+0020) at the end of a line are not part of the line's content. Decoders MUST strip them after the CR exclusion above and before line classification (§5.2); a line whose content is `-` followed only by spaces is therefore the bare marker for an empty-object list item (§9.4, §10), not a list item carrying an empty token. - - Token trimming: when a token is extracted – a key token before a key-value colon or an entry key's colon (§7.4, §9.5), a field entry in a field list (§6), or a value token after a key-value colon, after an array-header colon, or around each delimiter-separated token – decoders MUST trim surrounding spaces, exactly U+0020, no other characters. Any other whitespace (e.g., NBSP, or HTAB outside its delimiter role) is part of the token; internal semantics follow quoting rules. This trimming does not apply between a key and its bracket segment, where whitespace is a header syntax error (§6). + - Token trimming: when a token is extracted – a key token before a key-value colon or an entry key's colon (§7.4, §9.5), a field entry in a field list (§6), or a value token after a key-value colon, after an array-header colon, or around each delimiter-separated token – decoders MUST trim surrounding spaces, exactly U+0020, no other characters. Any other whitespace (e.g., NBSP, or HTAB outside its delimiter role) is part of the token; internal semantics follow quoting rules. This trimming does not apply between a key and its bracket segment or between a field name and its nested field group, where whitespace is a header syntax error (§6). - Comment lines are removed before any check in this section applies (§5.1). - Blank lines: - A line consisting of spaces only is blank, regardless of their count; the indentation checks above do not apply to blank lines. A line of tabs, or of spaces and tabs, is not blank: its tabs are indentation (above). @@ -731,7 +731,7 @@ When strict mode is enabled (default), decoders MUST error on the conditions lis - Keyed headers (§9.5): a missing field list (`key[2:]:`), a keyless keyed header anywhere other than as the document's root header, or a line at entry depth without an unquoted colon. - Non-whitespace content after a fields-bearing header's colon (§6), keyed or not (e.g., `items[2]{a,b}: 1,2`). - Keyless headers outside their valid positions (§6): a keyless non-keyed header in object-field position (e.g., `[2]: x,y` under an object field, or as a non-first depth-0 line), or a keyless fields-bearing header as a list item (`- [2]{a}:`). -- Any whitespace between a key and its bracket segment, and any content between a valid bracket segment and the colon (or field list), prevents array-header interpretation; decoders MUST NOT silently discard that content. In non-strict mode, decoders MAY fall through to key-value parsing. +- Any whitespace between a key and its bracket segment or between a field name and its nested field group, and any content between a valid bracket segment and the colon (or field list), prevents array-header interpretation; decoders MUST NOT silently discard that content. In non-strict mode, decoders MAY fall through to key-value parsing. - Indentation and blank-line invariants per §12, evaluated after comment removal (§5.1). - Indentation depth jumps (§8): the first line of a nested scope standing more than one level deeper than the line that opens it (e.g., a depth d+2 line directly under a depth-d `key:`). - Over-indented lines (§8): a line deeper than the content depth of its enclosing scope when the preceding line did not open a scope (e.g., a depth d+1 line directly under a depth-d primitive field, or the document's first line at depth 1 or more). Decoders MUST NOT silently discard such lines. diff --git a/tests/fixtures/decode/validation-errors.json b/tests/fixtures/decode/validation-errors.json index 570b2f4..f4e2163 100644 --- a/tests/fixtures/decode/validation-errors.json +++ b/tests/fixtures/decode/validation-errors.json @@ -648,6 +648,14 @@ "shouldError": true, "specSection": "6", "minSpecVersion": "4.2" + }, + { + "name": "throws on a space between a field name and its nested field group", + "input": "a[1]{b {c}}:\n 1", + "expected": null, + "shouldError": true, + "specSection": "6", + "minSpecVersion": "4.2" } ] } From bb827ce27eacce33261f0fa6a75ae769bc9bc45f Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 10:16:01 +0200 Subject: [PATCH 15/36] docs: define quote state for structural characters --- SPEC.md | 6 ++++-- tests/fixtures/decode/delimiters.json | 12 ++++++++++++ tests/fixtures/decode/objects.json | 9 +++++++++ tests/fixtures/decode/root-form.json | 7 +++++++ 4 files changed, 32 insertions(+), 2 deletions(-) diff --git a/SPEC.md b/SPEC.md index 8543d00..fb0b01c 100644 --- a/SPEC.md +++ b/SPEC.md @@ -302,6 +302,8 @@ Quoting keeps "#"-leading data out of the comment rule: string values that equal Decoders classify each line of the comment-stripped sequence (§5.1) by its content after the leading indentation. The first matching class applies. Classification is lexical; whether a class is admissible at a given depth and position is determined by the enclosing construct (root form above, §8–§10). Within a tabular array's scope, lines at row depth are divided between the row and key-value classes by the disambiguation rules of §9.3, which are authoritative for that position and take precedence over the order below. Within a keyed tabular object's scope, every line at entry depth containing an unquoted colon is an entry row; §9.5 is likewise authoritative for that position. +Throughout this specification, a `"` opens a quoted span wherever it occurs in a line, and the next unescaped `"` closes it; without one, the span runs to the end of the line. A colon, delimiter, bracket, or brace inside a quoted span is quoted; any other is unquoted. + 1. Blank line – the line consists of spaces only (§12). Handled per §12; blank lines never create or close structure. 2. List-item line – the content is the bare marker "-" or begins with "- " (hyphen, space). The remainder after the marker is parsed per §9.2, §9.4, and §10. A leading hyphen marks a list item only at the item depth of an array in list form; elsewhere – including a list-item object's field depth (§10) – the line is classified by the remaining classes. 3. Array-header line – the content matches the header or keyed-header grammar of §6. A line whose first unquoted colon precedes its first unquoted "[" is never a header; it is a key-value line. Only unquoted occurrences count: a quoted key containing a colon can still open a header (e.g., `"a:b"[2]: 1,2` is a header), while `a:b[2]: x` is a key-value line with key `a`. @@ -371,7 +373,7 @@ Decoding requirements: - A colon immediately after the length and before the optional delimiter symbol marks a keyed header (§9.5): `[N:]` declares comma, `[N:]` tab, `[N:|]` pipe. The colon MUST occupy exactly that position – tokens such as `[2|:]`, `[2 :]`, or `[2:,]` are malformed bracket segments, and the length rules above apply unchanged (`[03:]` is malformed). - A keyed header MUST carry a field list: `key[2:]:` without braces is a header syntax error in strict mode; non-strict decoders MAY fall through to key-value parsing (§14.2). - If a trailing tab or pipe appears inside the brackets, it selects the active delimiter; otherwise comma is active. -- If a field list occurs between the bracket and the colon, parse field entries recursively using the active delimiter at every nesting level; quoted names MUST be unescaped per §7.1. Brace matching MUST ignore `{` and `}` inside quoted names. +- If a field list occurs between the bracket and the colon, parse field entries recursively using the active delimiter at every nesting level; quoted names MUST be unescaped per §7.1. Brace matching MUST ignore `{` and `}` inside quoted spans (§5.2). - A field list MUST contain at least one field entry at every nesting level, and no field entry or field name may be empty: an empty field list (`{}`, including a nested `field{}`), an empty field entry (`{a,}`), or a nameless nested group (`{a,{b}}`) is a header syntax error in strict mode; non-strict decoders MAY fall through to key-value parsing (§14.2). Unmatched braces in a field list are likewise header syntax errors, with the same non-strict fall-through. - A colon MUST follow the bracket segment and optional field list; missing colon MUST error. A line without any unquoted colon is neither a header nor a key context (§5.2): `items[2]` alone is a scalar line, and `- [2]` is a list item carrying the string `[2]`. - A non-keyed header without a field list: content after its colon is an inline primitive array (§9.1); nothing after the colon opens a block scope (§9.2, §9.4). A fields-bearing header – keyed or not – carries no inline content: in strict mode, non-whitespace content after its colon MUST error (§14.2); non-strict decoders MAY fall through to key-value parsing. @@ -956,7 +958,7 @@ These sketches illustrate structure and common decoding helpers. They are inform - Length N as decimal integer. - A colon immediately after the length marks a keyed header (§9.5); it requires a field list. Entry rows split at their first unquoted colon into entry key and cell sequence; the cells then split on the active delimiter. - Optional delimiter symbol at the end: HTAB or pipe (comma otherwise). -- If a "{ … }" field list occurs between the "]" and the ":", parse field entries recursively using the active delimiter: track brace depth, ignoring braces inside quoted names; a fieldname followed by "{" opens a nested field group. Unescape quoted names. The leaf-field list is the depth-first, pre-order walk of the resulting tree; rows assign cells to leaf fields in that order (§9.3). +- If a "{ … }" field list occurs between the "]" and the ":", parse field entries recursively using the active delimiter: track brace depth, ignoring braces inside quoted spans; a fieldname followed by "{" opens a nested field group. Unescape quoted names. The leaf-field list is the depth-first, pre-order walk of the resulting tree; rows assign cells to leaf fields in that order (§9.3). - Require a colon ":" after the bracket/field list. - Return the header (key?, length, delimiter, fields?) and any inline values after the colon. - Absence of a delimiter symbol in the bracket segment always means comma for that header (no inheritance). diff --git a/tests/fixtures/decode/delimiters.json b/tests/fixtures/decode/delimiters.json index d12802a..0c1ab22 100644 --- a/tests/fixtures/decode/delimiters.json +++ b/tests/fixtures/decode/delimiters.json @@ -244,6 +244,18 @@ }, "specSection": "6", "minSpecVersion": "4.1" + }, + { + "name": "treats a delimiter after a mid-token quote as quoted", + "input": "k[2]: a\"b,c\",d", + "expected": { + "k": [ + "a\"b,c\"", + "d" + ] + }, + "specSection": "5.2", + "minSpecVersion": "4.2" } ] } diff --git a/tests/fixtures/decode/objects.json b/tests/fixtures/decode/objects.json index b9577b6..cb46ff1 100644 --- a/tests/fixtures/decode/objects.json +++ b/tests/fixtures/decode/objects.json @@ -608,6 +608,15 @@ }, "specSection": "5.2", "minSpecVersion": "4.2" + }, + { + "name": "parses a key whose closed mid-token quote span hides a colon", + "input": "a\"b:c\"d: 1", + "expected": { + "a\"b:c\"d": 1 + }, + "specSection": "5.2", + "minSpecVersion": "4.2" } ] } diff --git a/tests/fixtures/decode/root-form.json b/tests/fixtures/decode/root-form.json index c1203c1..d1b04f7 100644 --- a/tests/fixtures/decode/root-form.json +++ b/tests/fixtures/decode/root-form.json @@ -118,6 +118,13 @@ "specSection": "5", "note": "The non-strict skip for over-indented lines excludes scalar lines (§5.2)", "minSpecVersion": "4.2" + }, + { + "name": "parses a root primitive whose colon follows an unclosed quote", + "input": "a\"b: 1", + "expected": "a\"b: 1", + "specSection": "5.2", + "minSpecVersion": "4.2" } ] } From e3169ef307bd175a6215dbccd99d41a114af271b Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 10:16:40 +0200 Subject: [PATCH 16/36] docs: record the 4.2 changes in the changelog --- CHANGELOG.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index dc7e431..23cd9f7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,23 @@ All notable changes to the TOON specification will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). The project follows the MAJOR.MINOR versioning policy described in [VERSIONING.md](./VERSIONING.md). +## [4.2] - Unreleased + +### Added + +- §5.2: a `"` opens a quoted span wherever it occurs in a line, so a colon, delimiter, bracket, or brace between it and the next unescaped `"` (or the line end) is quoted. +- §14: a non-strict decoder either errors or applies the leniency the specification names; §8 names one for depth jumps – the first line's depth becomes the scope's content depth. +- Conformance fixtures for the new decoder rules, the shortest digits, and list-item quoting. + +### Changed + +- Encoder output is fully determined: numbers use the shortest round-trip digits (§2), strings and keys stay unquoted unless §7.2 or §7.3 requires quotes, and primitive list items are quoted by the document delimiter like object field values (§7.2, §11.1). +- Decoder edge cases are settled: a line without an unquoted colon is never a header (§6); an indented first line is over-indented (§5, §8); a hyphen marks a list item only at item depth (§5.2); an empty key token is the empty key and an empty field entry is a header syntax error (§6, §7.4); whitespace between a field name and its nested field group is a header syntax error (§6); a length beyond the integer range still forms a header (§6); raw control characters inside quoted strings decode as themselves (§7.1); and a line of tabs is not blank (§12). + +### Compatibility + +Every decoder change concerns input no conforming 4.1 encoder emits: §7.1–§7.3 escape every control character and quote any string or key that is empty, starts with `-`, or contains `"`, a bracket, or a brace, and encoders never indent the first line, indent with tabs, put a space before a nested field group, or trigger a §14 condition. Each is MINOR under the encoder-unreachability rule in [VERSIONING.md](./VERSIONING.md). The encoder changes are tightenings: decoders read the output of every 4.1 encoder unchanged. + ## [4.1] - 2026-07-26 ### Added From 18d6610fe1789198489ca71c0fb0bc87d30461c8 Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 10:16:40 +0200 Subject: [PATCH 17/36] docs: bump the spec version to 4.2 --- SPEC.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/SPEC.md b/SPEC.md index fb0b01c..4cc7451 100644 --- a/SPEC.md +++ b/SPEC.md @@ -2,7 +2,7 @@ ## Token-Oriented Object Notation -**Version:** 4.1 +**Version:** 4.2 **Date:** 2026-07-26 @@ -663,7 +663,7 @@ Options: Strict-mode errors are enumerated in §14; validators MAY add informative diagnostics for style and encoding invariants. -Implementations SHOULD declare the specification version they target (e.g., `toon-spec: 4.1`) in their documentation. +Implementations SHOULD declare the specification version they target (e.g., `toon-spec: 4.2`) in their documentation. ### 13.1 Encoder Conformance Checklist From d669bc5bef3a33604d7ce93dc44c561aacca5f70 Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 10:28:49 +0200 Subject: [PATCH 18/36] test: indent the nested list items in the blank-line case to item depth --- tests/fixtures/decode/blank-lines.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/fixtures/decode/blank-lines.json b/tests/fixtures/decode/blank-lines.json index a476cee..1765ad0 100644 --- a/tests/fixtures/decode/blank-lines.json +++ b/tests/fixtures/decode/blank-lines.json @@ -55,7 +55,7 @@ }, { "name": "throws on blank line in nested list array", - "input": "outer[2]:\n - inner[2]:\n - a\n\n - b\n - x", + "input": "outer[2]:\n - inner[2]:\n - a\n\n - b\n - x", "expected": null, "shouldError": true, "options": { From ebbc712ed1d7e7cf77c65485ab55b17ad71b2be8 Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 12:17:27 +0200 Subject: [PATCH 19/36] =?UTF-8?q?docs:=20allow=20a=20quoted=20empty=20fiel?= =?UTF-8?q?d=20name=20and=20list=20the=20nameless=20nested=20group=20in=20?= =?UTF-8?q?=C2=A714?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- SPEC.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/SPEC.md b/SPEC.md index 4cc7451..1296fb9 100644 --- a/SPEC.md +++ b/SPEC.md @@ -374,7 +374,7 @@ Decoding requirements: - A keyed header MUST carry a field list: `key[2:]:` without braces is a header syntax error in strict mode; non-strict decoders MAY fall through to key-value parsing (§14.2). - If a trailing tab or pipe appears inside the brackets, it selects the active delimiter; otherwise comma is active. - If a field list occurs between the bracket and the colon, parse field entries recursively using the active delimiter at every nesting level; quoted names MUST be unescaped per §7.1. Brace matching MUST ignore `{` and `}` inside quoted spans (§5.2). -- A field list MUST contain at least one field entry at every nesting level, and no field entry or field name may be empty: an empty field list (`{}`, including a nested `field{}`), an empty field entry (`{a,}`), or a nameless nested group (`{a,{b}}`) is a header syntax error in strict mode; non-strict decoders MAY fall through to key-value parsing (§14.2). Unmatched braces in a field list are likewise header syntax errors, with the same non-strict fall-through. +- A field list MUST contain at least one field entry at every nesting level: an empty field list (`{}`, including a nested `field{}`), an empty field entry (`{a,}`), or a nameless nested group (`{a,{b}}`) is a header syntax error in strict mode; non-strict decoders MAY fall through to key-value parsing (§14.2). Unmatched braces in a field list are likewise header syntax errors, with the same non-strict fall-through. - A colon MUST follow the bracket segment and optional field list; missing colon MUST error. A line without any unquoted colon is neither a header nor a key context (§5.2): `items[2]` alone is a scalar line, and `- [2]` is a list item carrying the string `[2]`. - A non-keyed header without a field list: content after its colon is an inline primitive array (§9.1); nothing after the colon opens a block scope (§9.2, §9.4). A fields-bearing header – keyed or not – carries no inline content: in strict mode, non-whitespace content after its colon MUST error (§14.2); non-strict decoders MAY fall through to key-value parsing. - Keyless header positions: a keyless non-keyed header without a field list is valid only as the document's root header (§5) or as a list item after the `- ` marker (§9.2, §9.4); a keyless header with a field list – keyed or not – is valid only as the document's root header. In any other position, strict decoders MUST error (§14.2); non-strict decoders MAY parse the line as a key-value line, with the key treated as a literal token. @@ -444,7 +444,7 @@ Decoding of value tokens follows §4 (unquoted type inference, quoted strings, n - Quoted keys MUST be unescaped per §7.1; any other escape MUST error. - Keys (quoted or unquoted) MUST be followed by ":", optionally after spaces (§12); missing colon MUST error (see also §14.2). -- Unquoted key token (normative): an unquoted key token is the text before the first unquoted colon of a key-value line (§5.2) or entry row (§9.5), with surrounding spaces trimmed (§12); the text before a header's bracket segment; or a field name in a field list (§6). An empty token before a key-value or entry-row colon is the empty key: `: 1` decodes to `{"": 1}`. Decoders MUST accept any such token as a literal key, in strict and non-strict mode alike, even when it does not match §7.3's unquoted-key pattern: `foo-bar: 1`, `foo-bar[2]: 1,2`, and `items[1]{2key}:` are valid input. §7.3 governs how encoders emit keys, not what decoders accept. +- Unquoted key token (normative): an unquoted key token is the text before the first unquoted colon of a key-value line (§5.2) or entry row (§9.5), with surrounding spaces trimmed (§12); the text before a header's bracket segment; or a field name in a field list (§6). An empty token before a key-value or entry-row colon is the empty key: `: 1` decodes to `{"": 1}`. Decoders MUST accept any non-empty such token as a literal key, in strict and non-strict mode alike, even when it does not match §7.3's unquoted-key pattern: `foo-bar: 1`, `foo-bar[2]: 1,2`, and `items[1]{2key}:` are valid input. §7.3 governs how encoders emit keys, not what decoders accept. - Quoted-token boundary (normative): a token whose first character, after the trimming of §12, is `"` MUST be a complete quoted token – its closing `"` MUST be the token's last character. This applies wherever a token is extracted; any character after the closing quote MUST error. It overrides §4's "Otherwise → string" fallback. - Symmetrically for values: an unquoted value token that an encoder would have been required to quote (§7.2) is not an error. Decoders, strict mode included, MUST decode it per §4 – unless another rule of this specification assigns the token structural meaning (§5.2, §6, §9.1). Example: `key: -x` decodes to the string `-x`. §7.2 governs encoder output; it adds no decoder-side rejection. @@ -729,7 +729,7 @@ When strict mode is enabled (default), decoders MUST error on the conditions lis - Invalid escape sequences or unterminated strings in quoted tokens, and characters after a quoted token's closing quote (any mode; §4, §7.1, §7.4). - Header delimiter mismatch (§6): MUST error as a header syntax error, independent of row width/count checks. - Malformed bracket lengths and malformed keyed markers in headers (§6). -- Malformed field lists in headers: an empty field list (`{}`, including a nested `field{}`), an empty field entry (`{a,}`), unmatched braces, or a field name repeated within the same field list (`{a,a}`, including inside a nested group); see §6, §9.3, §9.5. These are diagnosed from the header line alone, independent of the declared count and of any following rows or entry rows. +- Malformed field lists in headers: an empty field list (`{}`, including a nested `field{}`), an empty field entry (`{a,}`), a nameless nested group (`{a,{b}}`), unmatched braces, or a field name repeated within the same field list (`{a,a}`, including inside a nested group); see §6, §9.3, §9.5. These are diagnosed from the header line alone, independent of the declared count and of any following rows or entry rows. - Keyed headers (§9.5): a missing field list (`key[2:]:`), a keyless keyed header anywhere other than as the document's root header, or a line at entry depth without an unquoted colon. - Non-whitespace content after a fields-bearing header's colon (§6), keyed or not (e.g., `items[2]{a,b}: 1,2`). - Keyless headers outside their valid positions (§6): a keyless non-keyed header in object-field position (e.g., `[2]: x,y` under an object field, or as a non-first depth-0 line), or a keyless fields-bearing header as a list item (`- [2]{a}:`). From e8add475a3e26cb057901b7f9795b146272ab3b5 Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 12:17:27 +0200 Subject: [PATCH 20/36] docs: tie a blank tab line to the non-strict tab leniency --- SPEC.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/SPEC.md b/SPEC.md index 1296fb9..e5697f9 100644 --- a/SPEC.md +++ b/SPEC.md @@ -304,7 +304,7 @@ Decoders classify each line of the comment-stripped sequence (§5.1) by its cont Throughout this specification, a `"` opens a quoted span wherever it occurs in a line, and the next unescaped `"` closes it; without one, the span runs to the end of the line. A colon, delimiter, bracket, or brace inside a quoted span is quoted; any other is unquoted. -1. Blank line – the line consists of spaces only (§12). Handled per §12; blank lines never create or close structure. +1. Blank line – blank per §12, which governs its handling; blank lines never create or close structure. 2. List-item line – the content is the bare marker "-" or begins with "- " (hyphen, space). The remainder after the marker is parsed per §9.2, §9.4, and §10. A leading hyphen marks a list item only at the item depth of an array in list form; elsewhere – including a list-item object's field depth (§10) – the line is classified by the remaining classes. 3. Array-header line – the content matches the header or keyed-header grammar of §6. A line whose first unquoted colon precedes its first unquoted "[" is never a header; it is a key-value line. Only unquoted occurrences count: a quoted key containing a colon can still open a header (e.g., `"a:b"[2]: 1,2` is a header), while `a:b[2]: x` is a key-value line with key `a`. 4. Key-value line – the content contains an unquoted colon and no earlier class applies. The key token precedes the first unquoted colon and is decoded per §7.4; the remainder after the colon is the value (§8). A line that contains an unquoted colon but fails the §6 header grammar falls through to this class (e.g., `foo [2]: bar`); the strict-mode header errors enumerated in §6 and §14.2 are unaffected by this fall-through. @@ -642,7 +642,7 @@ For an object appearing as a list item: - Token trimming: when a token is extracted – a key token before a key-value colon or an entry key's colon (§7.4, §9.5), a field entry in a field list (§6), or a value token after a key-value colon, after an array-header colon, or around each delimiter-separated token – decoders MUST trim surrounding spaces, exactly U+0020, no other characters. Any other whitespace (e.g., NBSP, or HTAB outside its delimiter role) is part of the token; internal semantics follow quoting rules. This trimming does not apply between a key and its bracket segment or between a field name and its nested field group, where whitespace is a header syntax error (§6). - Comment lines are removed before any check in this section applies (§5.1). - Blank lines: - - A line consisting of spaces only is blank, regardless of their count; the indentation checks above do not apply to blank lines. A line of tabs, or of spaces and tabs, is not blank: its tabs are indentation (above). + - A line consisting of spaces only is blank, regardless of their count; the indentation checks above do not apply to blank lines. A line of tabs, or of spaces and tabs, is blank only under the non-strict tab leniency (above); otherwise its tabs are indentation. - Header span: the lines from the first item, row, or entry line in a header's scope through the last line of that scope's content (which may be a deeper line inside its last item). A blank line inside any header span: in strict mode, MUST error; in non-strict mode, MAY be ignored and not counted as a row/item/entry. - All other blank lines – including between a header and the scope's first item, row, or entry line, and after a scope's content: in strict mode, decoders MUST ignore them (they do not create or close structures and are not counted); in non-strict mode, decoders SHOULD ignore them. - Trailing newline at end-of-file: decoders SHOULD accept; validators MAY warn. From dcf8dd35192001594554ff3a67285672712eba6b Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 12:17:27 +0200 Subject: [PATCH 21/36] docs: drop the header colon MUST that contradicted the colon-less scalar line --- SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SPEC.md b/SPEC.md index e5697f9..6d5477d 100644 --- a/SPEC.md +++ b/SPEC.md @@ -375,7 +375,7 @@ Decoding requirements: - If a trailing tab or pipe appears inside the brackets, it selects the active delimiter; otherwise comma is active. - If a field list occurs between the bracket and the colon, parse field entries recursively using the active delimiter at every nesting level; quoted names MUST be unescaped per §7.1. Brace matching MUST ignore `{` and `}` inside quoted spans (§5.2). - A field list MUST contain at least one field entry at every nesting level: an empty field list (`{}`, including a nested `field{}`), an empty field entry (`{a,}`), or a nameless nested group (`{a,{b}}`) is a header syntax error in strict mode; non-strict decoders MAY fall through to key-value parsing (§14.2). Unmatched braces in a field list are likewise header syntax errors, with the same non-strict fall-through. -- A colon MUST follow the bracket segment and optional field list; missing colon MUST error. A line without any unquoted colon is neither a header nor a key context (§5.2): `items[2]` alone is a scalar line, and `- [2]` is a list item carrying the string `[2]`. +- A line without any unquoted colon is neither a header nor a key context (§5.2): `items[2]` alone is a scalar line, and `- [2]` is a list item carrying the string `[2]`. - A non-keyed header without a field list: content after its colon is an inline primitive array (§9.1); nothing after the colon opens a block scope (§9.2, §9.4). A fields-bearing header – keyed or not – carries no inline content: in strict mode, non-whitespace content after its colon MUST error (§14.2); non-strict decoders MAY fall through to key-value parsing. - Keyless header positions: a keyless non-keyed header without a field list is valid only as the document's root header (§5) or as a list item after the `- ` marker (§9.2, §9.4); a keyless header with a field list – keyed or not – is valid only as the document's root header. In any other position, strict decoders MUST error (§14.2); non-strict decoders MAY parse the line as a key-value line, with the key treated as a literal token. From e58f46335f9fc3ac835b33a93aebb0ec6195bbb6 Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 12:17:27 +0200 Subject: [PATCH 22/36] docs: measure a depth jump against the scope's content depth --- SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SPEC.md b/SPEC.md index 6d5477d..a4ba61d 100644 --- a/SPEC.md +++ b/SPEC.md @@ -735,7 +735,7 @@ When strict mode is enabled (default), decoders MUST error on the conditions lis - Keyless headers outside their valid positions (§6): a keyless non-keyed header in object-field position (e.g., `[2]: x,y` under an object field, or as a non-first depth-0 line), or a keyless fields-bearing header as a list item (`- [2]{a}:`). - Any whitespace between a key and its bracket segment or between a field name and its nested field group, and any content between a valid bracket segment and the colon (or field list), prevents array-header interpretation; decoders MUST NOT silently discard that content. In non-strict mode, decoders MAY fall through to key-value parsing. - Indentation and blank-line invariants per §12, evaluated after comment removal (§5.1). -- Indentation depth jumps (§8): the first line of a nested scope standing more than one level deeper than the line that opens it (e.g., a depth d+2 line directly under a depth-d `key:`). +- Indentation depth jumps (§8): the first line of a nested scope standing deeper than the scope's content depth (§1.3; e.g., a depth d+2 line directly under a depth-d `key:`). - Over-indented lines (§8): a line deeper than the content depth of its enclosing scope when the preceding line did not open a scope (e.g., a depth d+1 line directly under a depth-d primitive field, or the document's first line at depth 1 or more). Decoders MUST NOT silently discard such lines. - Trailing content after a completed root form (§5): any non-comment, non-blank line following the inline values, rows, items, or entries of a root array or keyed tabular root object, or following a root `[]`. - Ill-formed UTF-8 in byte input (§4). From 9186183db050e43d14989856de91d9e3beed2c28 Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 12:17:27 +0200 Subject: [PATCH 23/36] docs: read the `[]` root rule on the first depth-0 line --- SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SPEC.md b/SPEC.md index a4ba61d..0abb8fe 100644 --- a/SPEC.md +++ b/SPEC.md @@ -276,7 +276,7 @@ TOON is a deterministic, line-oriented, indentation-based notation. - Root form discovery (applied to the comment-stripped line sequence, §5.1; line classes per §5.2): - If the first non-blank depth-0 line is a valid root array header per §6, decode a root array. - Else if the first non-blank depth-0 line is a valid keyless keyed header per §6 ([N:]{…}:), decode a root object in keyed tabular form (§9.5). - - Else if the first non-blank line is the literal token `[]` at depth 0, decode an empty root array (§9.1). + - Else if the first non-blank depth-0 line is the literal token `[]`, decode an empty root array (§9.1). - Else if the document has exactly one non-blank line, at depth 0, and it is neither a valid array header nor a key-value line (quoted or unquoted key), decode a single primitive (examples: `hello`, `42`, `true`). - Otherwise, decode an object. - An empty document (no non-blank lines after comment removal, §5.1) decodes to an empty object `{}`. A document consisting only of comment and blank lines is therefore `{}`. From 0b4949d1a805e517b8604012c85ff047206705d8 Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 12:17:27 +0200 Subject: [PATCH 24/36] docs: define item depth as a content depth --- SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SPEC.md b/SPEC.md index 0abb8fe..06fe822 100644 --- a/SPEC.md +++ b/SPEC.md @@ -139,7 +139,7 @@ All normative text is in Sections 1–16; the appendices and all examples are in - Indentation level (depth): Leading indentation measured in fixed-size space units (indentSize). Depth 0 has no indentation. - Indentation unit (indentSize): A fixed number of spaces per level (default 2). Tabs MUST NOT be used for indentation. - Content depth: The depth at which a scope's immediate content appears – 0 for the root scope, otherwise one level deeper than the depth at which the scope's opening line stands (see §10 for first fields carried on a list-item hyphen line, and §8 for the non-strict depth-jump leniency). -- Row depth, entry depth: The content depth of a tabular array's or keyed tabular object's scope, at which its rows or entry rows appear (§9.3, §9.5). +- Row depth, entry depth, item depth: The content depth of a tabular array's, keyed tabular object's, or list-form array's scope, at which its rows, entry rows, or list items appear (§9.3, §9.4, §9.5). ### 1.4 Array and Tabular Terms From 5217d91789446faad4bb153588b8960e4645a01a Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 12:17:27 +0200 Subject: [PATCH 25/36] docs: correct the 4.2 changelog --- CHANGELOG.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 23cd9f7..6b2077b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,17 +9,17 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). ### Added - §5.2: a `"` opens a quoted span wherever it occurs in a line, so a colon, delimiter, bracket, or brace between it and the next unescaped `"` (or the line end) is quoted. -- §14: a non-strict decoder either errors or applies the leniency the specification names; §8 names one for depth jumps – the first line's depth becomes the scope's content depth. +- §14: a non-strict decoder applies the leniency the specification names, or errors where that leniency is optional; 4.2 adds three – a depth jump's first line sets the scope's content depth (§8), ill-formed UTF-8 MAY become U+FFFD (§4), and unmatched braces MAY fall through to key-value parsing (§6). - Conformance fixtures for the new decoder rules, the shortest digits, and list-item quoting. ### Changed -- Encoder output is fully determined: numbers use the shortest round-trip digits (§2), strings and keys stay unquoted unless §7.2 or §7.3 requires quotes, and primitive list items are quoted by the document delimiter like object field values (§7.2, §11.1). -- Decoder edge cases are settled: a line without an unquoted colon is never a header (§6); an indented first line is over-indented (§5, §8); a hyphen marks a list item only at item depth (§5.2); an empty key token is the empty key and an empty field entry is a header syntax error (§6, §7.4); whitespace between a field name and its nested field group is a header syntax error (§6); a length beyond the integer range still forms a header (§6); raw control characters inside quoted strings decode as themselves (§7.1); and a line of tabs is not blank (§12). +- Encoder output is narrowed: numbers use the shortest round-trip digits (§2), strings and keys stay unquoted unless §7.2 or §7.3 requires quotes, and primitive list items are quoted by the document delimiter like object field values (§7.2, §11.1). +- Decoder edge cases are settled: a line without an unquoted colon is never a header (§6); an indented first line is over-indented (§5, §8); a hyphen marks a list item only at item depth (§5.2); an empty key token is the empty key and an empty field entry or nameless nested group is a header syntax error (§6, §7.4); whitespace between a field name and its nested field group is a header syntax error (§6); a length beyond the integer range still forms a header (§6); raw control characters inside quoted strings decode as themselves (§7.1); and a line of tabs is blank only under the non-strict tab leniency (§12). ### Compatibility -Every decoder change concerns input no conforming 4.1 encoder emits: §7.1–§7.3 escape every control character and quote any string or key that is empty, starts with `-`, or contains `"`, a bracket, or a brace, and encoders never indent the first line, indent with tabs, put a space before a nested field group, or trigger a §14 condition. Each is MINOR under the encoder-unreachability rule in [VERSIONING.md](./VERSIONING.md). The encoder changes are tightenings: decoders read the output of every 4.1 encoder unchanged. +Every decoder change concerns input no conforming 4.0 or 4.1 encoder emits: §7.1–§7.3 escape every control character and quote any string or key that is empty, starts with `-`, or contains `"`, a bracket, or a brace, and encoders never indent the first line, indent with tabs, put a space before a nested field group, or trigger a §14 condition. Each is MINOR under the encoder-unreachability rule in [VERSIONING.md](./VERSIONING.md). The encoder changes are tightenings: decoders read the output of every 4.0 or 4.1 encoder unchanged. ## [4.1] - 2026-07-26 From 0ec4592fbc34f665135fc41f3761fc14e24cacfa Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 12:17:27 +0200 Subject: [PATCH 26/36] docs: tighten 4.2 wording --- SPEC.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/SPEC.md b/SPEC.md index 06fe822..b1e9f90 100644 --- a/SPEC.md +++ b/SPEC.md @@ -206,7 +206,7 @@ Row, entry, and item terms: - If the fractional part is zero after normalization, emit as an integer (e.g., 1.0 → 1). - -0 MUST be normalized to 0. - For finite numbers outside the canonical range above (non-zero |n| < 1e-6, or |n| ≥ 1e21), encoders MAY emit exponent notation conforming to the JSON number grammar [RFC8259] §6 (e.g., 1e-7, 1e+21). Encoders SHOULD use lowercase `e` and an explicit exponent sign for byte-for-byte determinism. - - Encoders MUST emit the fewest significant digits for which, after any §3 host-type normalization, decode(encode(x)) equals x under the JSON-model equality defined below; when several candidates qualify, they MUST emit the one closest to x, ties to even (as ECMAScript's Number::toString does for binary64). + - Encoders MUST emit the fewest significant digits for which, after any §3 host-type normalization, decode(encode(x)) equals x under the JSON-model equality defined below; when several candidates qualify, they MUST emit the one closest to x, ties to even. - If a source value is outside the implementation's documented numeric domain (e.g., arbitrary-precision decimals or integers exceeding that domain), the encoder MAY: - Emit a quoted string containing a lossless decimal representation (plain decimal or JSON exponent form); the chosen form MUST be documented. - Emit a number that round-trips to the host's numeric approximation (losing precision), provided it conforms to the rules above. @@ -300,7 +300,7 @@ Quoting keeps "#"-leading data out of the comment rule: string values that equal ### 5.2 Line Classification -Decoders classify each line of the comment-stripped sequence (§5.1) by its content after the leading indentation. The first matching class applies. Classification is lexical; whether a class is admissible at a given depth and position is determined by the enclosing construct (root form above, §8–§10). Within a tabular array's scope, lines at row depth are divided between the row and key-value classes by the disambiguation rules of §9.3, which are authoritative for that position and take precedence over the order below. Within a keyed tabular object's scope, every line at entry depth containing an unquoted colon is an entry row; §9.5 is likewise authoritative for that position. +Decoders classify each line of the comment-stripped sequence (§5.1) by its content after the leading indentation. The first matching class applies. Classification is lexical except for class 2's item-depth condition; whether a class is admissible at a given depth and position is determined by the enclosing construct (root form above, §8–§10). Within a tabular array's scope, lines at row depth are divided between the row and key-value classes by the disambiguation rules of §9.3, which are authoritative for that position and take precedence over the order below. Within a keyed tabular object's scope, every line at entry depth containing an unquoted colon is an entry row; §9.5 is likewise authoritative for that position. Throughout this specification, a `"` opens a quoted span wherever it occurs in a line, and the next unescaped `"` closes it; without one, the span runs to the end of the line. A colon, delimiter, bracket, or brace inside a quoted span is quoted; any other is unquoted. @@ -411,7 +411,7 @@ escaped-char = %x5C ( %x5C / DQUOTE / %x6E / %x72 / %x74 / unicode-escape ) unicode-escape = %x75 4HEXDIG ``` -Tabs are allowed inside quoted strings and as a declared delimiter; they MUST NOT be used for indentation (§12). Within quoted strings, encoders MUST escape every U+0000–U+001F character per the escape table above; decoders accept the literal controls in `unescaped-char` in any mode. +Tabs are allowed inside quoted strings and as a declared delimiter; they MUST NOT be used for indentation (§12). Within quoted strings, decoders accept the literal controls in `unescaped-char` in any mode. ### 7.2 Quoting Rules for String Values @@ -671,7 +671,7 @@ Conforming encoders MUST: - [ ] Produce UTF-8 output with LF (U+000A) line endings (§1.2) - [ ] Use consistent indentation (default 2 spaces, no tabs) (§12) - [ ] Escape per §7.1 in quoted strings; never emit other escapes -- [ ] Quote strings per §7.2 (the relevant delimiter is governed by §11.1: document delimiter for object-field values and primitive list items, active delimiter for inline array values, tabular row cells, and keyed entry-row cells) +- [ ] Quote strings per §7.2, with the relevant delimiter per §11.1 - [ ] Quote and escape exactly those object keys, entry keys, and field names that do not match §7.3's unquoted-key pattern (§7.3) - [ ] Select the form from the value's shape and position, not by preference (§1.4, §9) - [ ] Emit declared lengths [N] matching the actual inline value, list item, tabular row, or entry row count (§6, §9) @@ -954,7 +954,7 @@ These sketches illustrate structure and common decoding helpers. They are inform ### B.2 Array Header Parsing -- Identify the optional key prefix first (quoted: a `"…"` literal at line start; unquoted: the characters up to the first `[`, which contain no whitespace, §6). This sketch applies only to lines §5.2 has already classified as array-header lines. The bracket segment `[ … ]` begins at the first `[` after the key; parse: +- Identify the optional key prefix first (quoted: a `"…"` literal at line start; unquoted: the characters up to the first unquoted `[`, which contain no whitespace, §6). This sketch applies only to lines §5.2 has already classified as array-header lines. The bracket segment `[ … ]` begins at the first `[` after the key; parse: - Length N as decimal integer. - A colon immediately after the length marks a keyed header (§9.5); it requires a field list. Entry rows split at their first unquoted colon into entry key and cell sequence; the cells then split on the active delimiter. - Optional delimiter symbol at the end: HTAB or pipe (comma otherwise). From 2db4eaba353a994a1f06bbc86a4ad947139e7fb5 Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 12:18:04 +0200 Subject: [PATCH 27/36] test: name the quote span and trim the field-group space case to its header --- tests/fixtures/decode/delimiters.json | 2 +- tests/fixtures/decode/validation-errors.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/fixtures/decode/delimiters.json b/tests/fixtures/decode/delimiters.json index 0c1ab22..6d9c38b 100644 --- a/tests/fixtures/decode/delimiters.json +++ b/tests/fixtures/decode/delimiters.json @@ -246,7 +246,7 @@ "minSpecVersion": "4.1" }, { - "name": "treats a delimiter after a mid-token quote as quoted", + "name": "treats a delimiter inside a mid-token quote span as quoted", "input": "k[2]: a\"b,c\",d", "expected": { "k": [ diff --git a/tests/fixtures/decode/validation-errors.json b/tests/fixtures/decode/validation-errors.json index f4e2163..3bac24b 100644 --- a/tests/fixtures/decode/validation-errors.json +++ b/tests/fixtures/decode/validation-errors.json @@ -651,7 +651,7 @@ }, { "name": "throws on a space between a field name and its nested field group", - "input": "a[1]{b {c}}:\n 1", + "input": "a[0]{b {c}}:", "expected": null, "shouldError": true, "specSection": "6", From b5afe71b050c91dbebff9adce519d9f122d80f9d Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 12:18:08 +0200 Subject: [PATCH 28/36] test: cover the 4.2 cases the second review found --- tests/fixtures/decode/arrays-tabular.json | 27 +++++++++++++++++++ tests/fixtures/decode/indentation-errors.json | 26 ++++++++++++++++++ tests/fixtures/decode/objects.json | 9 +++++++ tests/fixtures/decode/root-form.json | 11 ++++++++ tests/fixtures/decode/validation-errors.json | 8 ++++++ tests/fixtures/encode/objects.json | 11 ++++++++ 6 files changed, 92 insertions(+) diff --git a/tests/fixtures/decode/arrays-tabular.json b/tests/fixtures/decode/arrays-tabular.json index 269bffa..175feb5 100644 --- a/tests/fixtures/decode/arrays-tabular.json +++ b/tests/fixtures/decode/arrays-tabular.json @@ -290,6 +290,33 @@ ] }, "specSection": "9.3" + }, + { + "name": "ignores a brace inside a mid-name quote span in a field list", + "input": "a[1]{x\"}\"y}:\n 1", + "expected": { + "a": [ + { + "x\"}\"y": 1 + } + ] + }, + "specSection": "6", + "minSpecVersion": "4.2" + }, + { + "name": "parses a quoted empty field name", + "input": "[2]{\"\"}:\n 1\n 2", + "expected": [ + { + "": 1 + }, + { + "": 2 + } + ], + "specSection": "6", + "minSpecVersion": "4.2" } ] } diff --git a/tests/fixtures/decode/indentation-errors.json b/tests/fixtures/decode/indentation-errors.json index bedf5ca..399144d 100644 --- a/tests/fixtures/decode/indentation-errors.json +++ b/tests/fixtures/decode/indentation-errors.json @@ -216,6 +216,32 @@ "shouldError": true, "specSection": "8", "minSpecVersion": "4.2" + }, + { + "name": "throws on a hyphen-leading bare token under a primitive field in non-strict mode", + "input": "a: 1\n - x", + "expected": null, + "shouldError": true, + "options": { + "strict": false + }, + "specSection": "5.2", + "minSpecVersion": "4.2" + }, + { + "name": "adopts the depth of a jumped first line in non-strict mode", + "input": "a:\n b: 1\n c: 2\n d: 3", + "expected": { + "a": { + "b": 1, + "d": 3 + } + }, + "options": { + "strict": false + }, + "specSection": "8", + "minSpecVersion": "4.2" } ] } diff --git a/tests/fixtures/decode/objects.json b/tests/fixtures/decode/objects.json index cb46ff1..dd8d36e 100644 --- a/tests/fixtures/decode/objects.json +++ b/tests/fixtures/decode/objects.json @@ -617,6 +617,15 @@ }, "specSection": "5.2", "minSpecVersion": "4.2" + }, + { + "name": "reads a line whose only unquoted colon sits in the bracket segment as a key-value line", + "input": "a[2:]{x}", + "expected": { + "a[2": "]{x}" + }, + "specSection": "6", + "minSpecVersion": "4.2" } ] } diff --git a/tests/fixtures/decode/root-form.json b/tests/fixtures/decode/root-form.json index d1b04f7..36ee20c 100644 --- a/tests/fixtures/decode/root-form.json +++ b/tests/fixtures/decode/root-form.json @@ -125,6 +125,17 @@ "expected": "a\"b: 1", "specSection": "5.2", "minSpecVersion": "4.2" + }, + { + "name": "throws on a scalar line after a skipped over-indented first line in non-strict mode", + "input": " a: 1\nhello", + "expected": null, + "shouldError": true, + "options": { + "strict": false + }, + "specSection": "5", + "minSpecVersion": "4.2" } ] } diff --git a/tests/fixtures/decode/validation-errors.json b/tests/fixtures/decode/validation-errors.json index 3bac24b..c085dbb 100644 --- a/tests/fixtures/decode/validation-errors.json +++ b/tests/fixtures/decode/validation-errors.json @@ -656,6 +656,14 @@ "shouldError": true, "specSection": "6", "minSpecVersion": "4.2" + }, + { + "name": "throws on a nested field group without a name", + "input": "a[0]{x,{y}}:", + "expected": null, + "shouldError": true, + "specSection": "6", + "minSpecVersion": "4.2" } ] } diff --git a/tests/fixtures/encode/objects.json b/tests/fixtures/encode/objects.json index 7441aa4..4d54d97 100644 --- a/tests/fixtures/encode/objects.json +++ b/tests/fixtures/encode/objects.json @@ -302,6 +302,17 @@ }, "expected": "\"\":\n a: 1", "specSection": "8" + }, + { + "name": "keeps strings that only resemble numbers or literals unquoted", + "input": { + "a": ".5", + "b": "NaN", + "c": "True" + }, + "expected": "a: .5\nb: NaN\nc: True", + "specSection": "7.2", + "minSpecVersion": "4.2" } ] } From 5206305869b606d5167c333d54fd84affa5a9a85 Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 12:23:01 +0200 Subject: [PATCH 29/36] test: cover the non-strict over-indentation skip in keyed, tabular, and list scopes --- tests/fixtures/decode/indentation-errors.json | 67 +++++++++++++++++++ 1 file changed, 67 insertions(+) diff --git a/tests/fixtures/decode/indentation-errors.json b/tests/fixtures/decode/indentation-errors.json index 399144d..405e343 100644 --- a/tests/fixtures/decode/indentation-errors.json +++ b/tests/fixtures/decode/indentation-errors.json @@ -242,6 +242,73 @@ }, "specSection": "8", "minSpecVersion": "4.2" + }, + { + "name": "skips a line above the adopted entry depth in non-strict mode", + "input": "m[1:]{a}:\n x: 1\n y: 2", + "expected": { + "m": { + "x": { + "a": 1 + } + } + }, + "options": { + "strict": false + }, + "specSection": "8", + "minSpecVersion": "4.2" + }, + { + "name": "skips a line above the adopted row depth in non-strict mode", + "input": "rows[2]{a}:\n 1\n x: 9\n 2", + "expected": { + "rows": [ + { + "a": 1 + }, + { + "a": 2 + } + ] + }, + "options": { + "strict": false + }, + "specSection": "8", + "minSpecVersion": "4.2" + }, + { + "name": "skips an over-indented line under a primitive list item in non-strict mode", + "input": "items[2]:\n - a\n b: 1\n - c", + "expected": { + "items": [ + "a", + "c" + ] + }, + "options": { + "strict": false + }, + "specSection": "8", + "minSpecVersion": "4.2" + }, + { + "name": "skips an over-indented line under a list-item field in non-strict mode", + "input": "items[1]:\n - a: 1\n deep: 1\n b: 2", + "expected": { + "items": [ + { + "a": 1, + "b": 2 + } + ] + }, + "options": { + "strict": false + }, + "specSection": "8", + "minSpecVersion": "4.2" } ] } From 6e5a685a1cbe1801e0c76fd9cc9180edf347b940 Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 14:03:59 +0200 Subject: [PATCH 30/36] test: remove fixture cases a broader case already covers --- tests/fixtures/decode/arrays-nested.json | 35 ---------- tests/fixtures/decode/arrays-primitive.json | 56 --------------- tests/fixtures/decode/arrays-tabular.json | 11 --- tests/fixtures/decode/blank-lines.json | 7 -- tests/fixtures/decode/comments.json | 18 ----- tests/fixtures/decode/delimiters.json | 48 ------------- tests/fixtures/decode/indentation-errors.json | 8 --- tests/fixtures/decode/numbers.json | 65 ----------------- tests/fixtures/decode/objects-keyed.json | 10 --- tests/fixtures/decode/objects.json | 40 ----------- tests/fixtures/decode/primitives.json | 42 ----------- tests/fixtures/decode/root-form.json | 27 -------- tests/fixtures/encode/arrays-nested.json | 30 -------- tests/fixtures/encode/arrays-objects.json | 46 ------------- tests/fixtures/encode/arrays-primitive.json | 16 ----- tests/fixtures/encode/delimiters.json | 69 ------------------- tests/fixtures/encode/objects.json | 9 --- tests/fixtures/encode/primitives.json | 12 ---- tests/fixtures/encode/whitespace.json | 8 --- 19 files changed, 557 deletions(-) diff --git a/tests/fixtures/decode/arrays-nested.json b/tests/fixtures/decode/arrays-nested.json index cdf83ed..685deba 100644 --- a/tests/fixtures/decode/arrays-nested.json +++ b/tests/fixtures/decode/arrays-nested.json @@ -103,14 +103,6 @@ }, "specSection": "9.2" }, - { - "name": "parses nested arrays of primitives", - "input": "pairs[2]:\n - [2]: a,b\n - [2]: c,d", - "expected": { - "pairs": [["a", "b"], ["c", "d"]] - }, - "specSection": "9.2" - }, { "name": "parses quoted strings and mixed lengths in nested arrays", "input": "pairs[2]:\n - [2]: a,b\n - [3]: \"c,d\",\"e:f\",\"true\"", @@ -127,14 +119,6 @@ }, "specSection": "9.2" }, - { - "name": "parses mixed-length inner arrays", - "input": "pairs[2]:\n - [1]: 1\n - [2]: 2,3", - "expected": { - "pairs": [[1], [2, 3]] - }, - "specSection": "9.2" - }, { "name": "parses root-level primitive array inline", "input": "[5]: x,y,\"true\",true,10", @@ -193,25 +177,6 @@ }, "specSection": "9.4" }, - { - "name": "parses arrays mixing objects and arrays", - "input": "items[2]:\n - a: 1\n - [2]: 1,2", - "expected": { - "items": [{ "a": 1 }, [1, 2]] - }, - "specSection": "9.4" - }, - { - "name": "parses quoted key with list-form array", - "input": "\"x-items\"[2]:\n - id: 1\n - id: 2", - "expected": { - "x-items": [ - { "id": 1 }, - { "id": 2 } - ] - }, - "specSection": "9.4" - }, { "name": "accepts bare bracket pair as empty inner array list item", "input": "items[2]:\n - []\n - [1]: x", diff --git a/tests/fixtures/decode/arrays-primitive.json b/tests/fixtures/decode/arrays-primitive.json index cffcd8e..96978fe 100644 --- a/tests/fixtures/decode/arrays-primitive.json +++ b/tests/fixtures/decode/arrays-primitive.json @@ -3,22 +3,6 @@ "category": "decode", "description": "Primitive array decoding – inline arrays of strings, numbers, booleans, quoted strings", "tests": [ - { - "name": "parses string arrays inline", - "input": "tags[3]: reading,gaming,coding", - "expected": { - "tags": ["reading", "gaming", "coding"] - }, - "specSection": "9.1" - }, - { - "name": "parses number arrays inline", - "input": "nums[3]: 1,2,3", - "expected": { - "nums": [1, 2, 3] - }, - "specSection": "9.1" - }, { "name": "parses mixed primitive arrays inline", "input": "data[4]: x,y,true,10", @@ -99,14 +83,6 @@ }, "specSection": "9.1" }, - { - "name": "parses quoted key containing brackets with inline array", - "input": "\"key[test]\"[3]: 1,2,3", - "expected": { - "key[test]": [1, 2, 3] - }, - "specSection": "9.1" - }, { "name": "parses quoted key containing a colon with inline array", "input": "\"a:b\"[2]: 1,2", @@ -115,22 +91,6 @@ }, "specSection": "9.1" }, - { - "name": "parses quoted key with empty array", - "input": "\"x-custom\"[0]:", - "expected": { - "x-custom": [] - }, - "specSection": "9.1" - }, - { - "name": "parses quoted empty string key with empty array", - "input": "\"\"[0]:", - "expected": { - "": [] - }, - "specSection": "9.1" - }, { "name": "decodes canonical empty array key: []", "input": "items: []", @@ -139,22 +99,6 @@ }, "specSection": "9.1" }, - { - "name": "decodes canonical empty array with quoted key", - "input": "\"x-custom\": []", - "expected": { - "x-custom": [] - }, - "specSection": "9.1" - }, - { - "name": "decodes canonical empty array with empty-string key", - "input": "\"\": []", - "expected": { - "": [] - }, - "specSection": "9.1" - }, { "name": "parses quoted key containing escaped quotes and brackets with inline array", "input": "\"has \\\"[2]\\\" in it\"[3]: 3,4,5", diff --git a/tests/fixtures/decode/arrays-tabular.json b/tests/fixtures/decode/arrays-tabular.json index 175feb5..ac9ed00 100644 --- a/tests/fixtures/decode/arrays-tabular.json +++ b/tests/fixtures/decode/arrays-tabular.json @@ -25,17 +25,6 @@ }, "specSection": "9.3" }, - { - "name": "parses quoted colon in tabular row as data", - "input": "items[2]{id,note}:\n 1,\"a:b\"\n 2,\"c:d\"", - "expected": { - "items": [ - { "id": 1, "note": "a:b" }, - { "id": 2, "note": "c:d" } - ] - }, - "specSection": "9.3" - }, { "name": "parses quoted header keys in tabular arrays", "input": "items[2]{\"order:id\",\"full name\"}:\n 1,Ada\n 2,Bob", diff --git a/tests/fixtures/decode/blank-lines.json b/tests/fixtures/decode/blank-lines.json index 1765ad0..fa2afa9 100644 --- a/tests/fixtures/decode/blank-lines.json +++ b/tests/fixtures/decode/blank-lines.json @@ -179,13 +179,6 @@ "specSection": "14.2", "note": "The blank is after the inner scope's last row but inside the outer array's span (\u00a712)" }, - { - "name": "throws on blank line between a list item's fields", - "input": "items[2]:\n - a: 1\n\n b: 2\n - x", - "expected": null, - "shouldError": true, - "specSection": "14.2" - }, { "name": "throws on blank line inside the last list item's fields", "input": "items[1]:\n - a: 1\n\n b: 2", diff --git a/tests/fixtures/decode/comments.json b/tests/fixtures/decode/comments.json index e351082..0eb5ac3 100644 --- a/tests/fixtures/decode/comments.json +++ b/tests/fixtures/decode/comments.json @@ -88,12 +88,6 @@ "specSection": "5.1", "note": "Comment text is discarded without interpretation; the quote never reaches the tokenizer" }, - { - "name": "decodes document of only comments to empty object", - "input": "# only\n# comments", - "expected": {}, - "specSection": "5.1" - }, { "name": "decodes comments and blank lines only to empty object", "input": "\n# c\n\n# d\n", @@ -200,18 +194,6 @@ "msg": "hello # world" }, "specSection": "5.1" - }, - { - "name": "round-trips quoted hash-leading values in field and inline array positions", - "input": "tag: \"#x\"\nlist[2]: \"#a\",b", - "expected": { - "tag": "#x", - "list": [ - "#a", - "b" - ] - }, - "specSection": "7.2" } ] } diff --git a/tests/fixtures/decode/delimiters.json b/tests/fixtures/decode/delimiters.json index 6d9c38b..c67e681 100644 --- a/tests/fixtures/decode/delimiters.json +++ b/tests/fixtures/decode/delimiters.json @@ -177,54 +177,6 @@ }, "specSection": "11" }, - { - "name": "parses nested array values containing pipe delimiter", - "input": "pairs[1|]:\n - [2|]: a|\"b|c\"", - "expected": { - "pairs": [["a", "b|c"]] - }, - "specSection": "11" - }, - { - "name": "parses nested array values containing tab delimiter", - "input": "pairs[1\t]:\n - [2\t]: a\t\"b\\tc\"", - "expected": { - "pairs": [["a", "b\tc"]] - }, - "specSection": "11" - }, - { - "name": "preserves quoted ambiguity with pipe delimiter", - "input": "items[3|]: \"true\"|\"42\"|\"-3.14\"", - "expected": { - "items": ["true", "42", "-3.14"] - }, - "specSection": "11" - }, - { - "name": "preserves quoted ambiguity with tab delimiter", - "input": "items[3\t]: \"true\"\t\"42\"\t\"-3.14\"", - "expected": { - "items": ["true", "42", "-3.14"] - }, - "specSection": "11" - }, - { - "name": "parses structural-looking strings when quoted with pipe delimiter", - "input": "items[3|]: \"[5]\"|\"{key}\"|\"- item\"", - "expected": { - "items": ["[5]", "{key}", "- item"] - }, - "specSection": "11" - }, - { - "name": "parses structural-looking strings when quoted with tab delimiter", - "input": "items[3\t]: \"[5]\"\t\"{key}\"\t\"- item\"", - "expected": { - "items": ["[5]", "{key}", "- item"] - }, - "specSection": "11" - }, { "name": "parses tabular headers with keys containing the active delimiter", "input": "items[2|]{\"a|b\"}:\n 1\n 2", diff --git a/tests/fixtures/decode/indentation-errors.json b/tests/fixtures/decode/indentation-errors.json index 405e343..396e5cb 100644 --- a/tests/fixtures/decode/indentation-errors.json +++ b/tests/fixtures/decode/indentation-errors.json @@ -189,14 +189,6 @@ "shouldError": true, "specSection": "14.2" }, - { - "name": "throws on orphan scalar line under a primitive field", - "input": "a: 1\n hello", - "expected": null, - "shouldError": true, - "specSection": "14.2", - "note": "A scalar line is valid only at root primitive position" - }, { "name": "throws on orphan scalar line under a primitive field in non-strict mode", "input": "a: 1\n hello", diff --git a/tests/fixtures/decode/numbers.json b/tests/fixtures/decode/numbers.json index 22ac3f1..d13c843 100644 --- a/tests/fixtures/decode/numbers.json +++ b/tests/fixtures/decode/numbers.json @@ -69,22 +69,6 @@ }, "specSection": "4" }, - { - "name": "parses very small exponent", - "input": "value: 1e-10", - "expected": { - "value": 0.0000000001 - }, - "specSection": "4" - }, - { - "name": "parses integer with positive exponent", - "input": "value: 5E+00", - "expected": { - "value": 5 - }, - "specSection": "4" - }, { "name": "parses zero with exponent as number", "input": "value: 0e1", @@ -93,50 +77,18 @@ }, "specSection": "4" }, - { - "name": "parses negative zero with exponent as number", - "input": "value: -0e1", - "expected": { - "value": 0 - }, - "specSection": "4" - }, { "name": "parses exponent notation", "input": "1e6", "expected": 1000000, "specSection": "4" }, - { - "name": "parses exponent notation with uppercase E", - "input": "1E+6", - "expected": 1000000, - "specSection": "4" - }, - { - "name": "parses negative exponent notation", - "input": "-1e-3", - "expected": -0.001, - "specSection": "4" - }, { "name": "treats unquoted leading-zero number as string", "input": "05", "expected": "05", "specSection": "4" }, - { - "name": "treats unquoted multi-leading-zero as string", - "input": "007", - "expected": "007", - "specSection": "4" - }, - { - "name": "treats unquoted octal-like as string", - "input": "0123", - "expected": "0123", - "specSection": "4" - }, { "name": "treats leading-zeros in array as strings", "input": "nums[3]: 05,007,0123", @@ -149,14 +101,6 @@ "expected": "-05", "specSection": "4" }, - { - "name": "treats negative leading-zeros in array as strings", - "input": "nums[2]: -05,-007", - "expected": { - "nums": ["-05", "-007"] - }, - "specSection": "4" - }, { "name": "treats bare fraction without integer part as string", "input": "value: .5", @@ -192,15 +136,6 @@ "specSection": "4", "note": "The grammar admits only an optional leading minus, so leading-plus cells stay strings" }, - { - "name": "parses positive exponent sign as number", - "input": "value: 1e+2", - "expected": { - "value": 100 - }, - "specSection": "4", - "note": "The plus sign is inside the exponent, which the grammar allows" - }, { "name": "treats Infinity and NaN tokens as strings", "input": "a: Infinity\nb: -Infinity\nc: NaN", diff --git a/tests/fixtures/decode/objects-keyed.json b/tests/fixtures/decode/objects-keyed.json index 3e2e05d..ee13147 100644 --- a/tests/fixtures/decode/objects-keyed.json +++ b/tests/fixtures/decode/objects-keyed.json @@ -136,16 +136,6 @@ "specSection": "9.5", "note": "Encoders never emit keyed headers for fewer than two entries; decoders accept any N with count checks" }, - { - "name": "accepts a single entry row", - "input": "m[1:]{v}:\n a: 1", - "expected": { - "m": { - "a": { "v": 1 } - } - }, - "specSection": "9.5" - }, { "name": "parses quoted cells containing the active delimiter", "input": "m[2:]{t}:\n a: \"x,y\"\n b: z", diff --git a/tests/fixtures/decode/objects.json b/tests/fixtures/decode/objects.json index dd8d36e..7fbb48d 100644 --- a/tests/fixtures/decode/objects.json +++ b/tests/fixtures/decode/objects.json @@ -89,14 +89,6 @@ }, "specSection": "8" }, - { - "name": "parses quoted object value with only spaces", - "input": "text: \" \"", - "expected": { - "text": " " - }, - "specSection": "8" - }, { "name": "parses quoted string value that looks like true", "input": "v: \"true\"", @@ -113,14 +105,6 @@ }, "specSection": "8" }, - { - "name": "parses quoted string value that looks like negative decimal", - "input": "v: \"-7.5\"", - "expected": { - "v": "-7.5" - }, - "specSection": "8" - }, { "name": "parses quoted object value shaped like an inline array header", "input": "a: \"[2]: x\"", @@ -309,22 +293,6 @@ }, "specSection": "8" }, - { - "name": "parses underscore-prefixed keys", - "input": "_private: 1", - "expected": { - "_private": 1 - }, - "specSection": "8" - }, - { - "name": "parses underscore-containing keys", - "input": "user_name: 1", - "expected": { - "user_name": 1 - }, - "specSection": "8" - }, { "name": "unescapes newline in key", "input": "\"line\\nbreak\": 1", @@ -333,14 +301,6 @@ }, "specSection": "7.1" }, - { - "name": "unescapes tab in key", - "input": "\"tab\\there\": 2", - "expected": { - "tab\there": 2 - }, - "specSection": "7.1" - }, { "name": "unescapes tab in key immediately followed by a colon", "input": "\"\\t:x\": v", diff --git a/tests/fixtures/decode/primitives.json b/tests/fixtures/decode/primitives.json index 9a89d42..f47fcf4 100644 --- a/tests/fixtures/decode/primitives.json +++ b/tests/fixtures/decode/primitives.json @@ -9,12 +9,6 @@ "expected": "hello", "specSection": "7.4" }, - { - "name": "parses unquoted string with underscore and numbers", - "input": "Ada_99", - "expected": "Ada_99", - "specSection": "7.4" - }, { "name": "parses empty quoted string", "input": "\"\"", @@ -75,12 +69,6 @@ "expected": "你好", "specSection": "7.4" }, - { - "name": "parses emoji", - "input": "🚀", - "expected": "🚀", - "specSection": "7.4" - }, { "name": "parses string with emoji and spaces", "input": "hello 👋 world", @@ -129,42 +117,12 @@ "expected": "true", "specSection": "7.4" }, - { - "name": "respects ambiguity quoting for false", - "input": "\"false\"", - "expected": "false", - "specSection": "7.4" - }, - { - "name": "respects ambiguity quoting for null", - "input": "\"null\"", - "expected": "null", - "specSection": "7.4" - }, { "name": "respects ambiguity quoting for integer", "input": "\"42\"", "expected": "42", "specSection": "7.4" }, - { - "name": "respects ambiguity quoting for negative decimal", - "input": "\"-3.14\"", - "expected": "-3.14", - "specSection": "7.4" - }, - { - "name": "respects ambiguity quoting for scientific notation", - "input": "\"1e-6\"", - "expected": "1e-6", - "specSection": "7.4" - }, - { - "name": "respects ambiguity quoting for leading-zero", - "input": "\"05\"", - "expected": "05", - "specSection": "7.4" - }, { "name": "decodes supplementary scalar (U+1F680) in quoted string as literal UTF-8", "input": "\"🚀 launch\"", diff --git a/tests/fixtures/decode/root-form.json b/tests/fixtures/decode/root-form.json index 36ee20c..081bdde 100644 --- a/tests/fixtures/decode/root-form.json +++ b/tests/fixtures/decode/root-form.json @@ -12,33 +12,6 @@ }, "specSection": "5" }, - { - "name": "parses single primitive string at root as primitive", - "input": "hello", - "expected": "hello", - "options": { - "strict": true - }, - "specSection": "5" - }, - { - "name": "parses single primitive number at root as primitive", - "input": "42", - "expected": 42, - "options": { - "strict": true - }, - "specSection": "5" - }, - { - "name": "parses single primitive boolean at root as primitive", - "input": "true", - "expected": true, - "options": { - "strict": true - }, - "specSection": "5" - }, { "name": "parses literal [] at root as empty array", "input": "[]", diff --git a/tests/fixtures/encode/arrays-nested.json b/tests/fixtures/encode/arrays-nested.json index 55e0715..be84301 100644 --- a/tests/fixtures/encode/arrays-nested.json +++ b/tests/fixtures/encode/arrays-nested.json @@ -3,14 +3,6 @@ "category": "encode", "description": "Nested and mixed array encoding – arrays of arrays, mixed type arrays, root arrays", "tests": [ - { - "name": "encodes nested arrays of primitives", - "input": { - "pairs": [["a", "b"], ["c", "d"]] - }, - "expected": "pairs[2]:\n - [2]: a,b\n - [2]: c,d", - "specSection": "9.2" - }, { "name": "quotes strings containing delimiters in nested arrays", "input": { @@ -27,14 +19,6 @@ "expected": "pairs[2]:\n - [0]:\n - [0]:", "specSection": "9.2" }, - { - "name": "encodes mixed-length inner arrays", - "input": { - "pairs": [[1], [2, 3]] - }, - "expected": "pairs[2]:\n - [1]: 1\n - [2]: 2,3", - "specSection": "9.2" - }, { "name": "encodes root-level primitive array", "input": ["x", "y", "true", true, 10], @@ -71,20 +55,6 @@ "expected": "[]", "specSection": "9.1" }, - { - "name": "encodes complex nested structure", - "input": { - "user": { - "id": 123, - "name": "Ada", - "tags": ["reading", "gaming"], - "active": true, - "prefs": [] - } - }, - "expected": "user:\n id: 123\n name: Ada\n tags[2]: reading,gaming\n active: true\n prefs: []", - "specSection": "8" - }, { "name": "uses list form for arrays mixing primitives and objects", "input": { diff --git a/tests/fixtures/encode/arrays-objects.json b/tests/fixtures/encode/arrays-objects.json index bc46a04..3bcc50c 100644 --- a/tests/fixtures/encode/arrays-objects.json +++ b/tests/fixtures/encode/arrays-objects.json @@ -14,18 +14,6 @@ "expected": "items[2]:\n - id: 1\n name: First\n - id: 2\n name: Second\n extra: true", "specSection": "9.4" }, - { - "name": "uses list form for objects with nested values", - "input": { - "items": [ - { "id": 1, "nested": { "x": 1 } }, - { "id": 2, "nested": { "y": 2 } } - ] - }, - "expected": "items[2]:\n - id: 1\n nested:\n x: 1\n - id: 2\n nested:\n y: 2", - "specSection": "9.4", - "note": "The nested column mixes key sets {x} and {y}, so it is not nested-uniform (§9.3), so the array uses list form" - }, { "name": "preserves field order in list items - array first", "input": { @@ -73,22 +61,6 @@ "expected": "items[1]:\n - users[2]:\n - id: 1\n name: Ada\n - id: 2\n status: active", "specSection": "10" }, - { - "name": "uses list form for objects with multiple array fields", - "input": { - "items": [{ "nums": [1, 2], "tags": ["a", "b"], "name": "test" }] - }, - "expected": "items[1]:\n - nums[2]: 1,2\n tags[2]: a,b\n name: test", - "specSection": "10" - }, - { - "name": "uses list form for objects with only array fields", - "input": { - "items": [{ "nums": [1, 2, 3], "tags": ["a", "b"] }] - }, - "expected": "items[1]:\n - nums[3]: 1,2,3\n tags[2]: a,b", - "specSection": "10" - }, { "name": "encodes objects with empty arrays in list form", "input": { @@ -99,15 +71,6 @@ "expected": "items[1]:\n - name: Ada\n data: []", "specSection": "10" }, - { - "name": "uses canonical encoding for multi-field list-item objects with tabular arrays", - "input": { - "items": [{ "users": [{ "id": 1 }, { "id": 2 }], "note": "x" }] - }, - "expected": "items[1]:\n - users[2]{id}:\n 1\n 2\n note: x", - "specSection": "10", - "note": "Tabular header on hyphen line with rows at depth +2 and sibling fields at depth +1" - }, { "name": "uses canonical encoding for single-field list-item tabular arrays", "input": { @@ -125,15 +88,6 @@ "expected": "items[1]:\n - data: []\n name: x", "specSection": "10" }, - { - "name": "encodes empty object list items as bare hyphen", - "input": { - "items": ["first", "second", {}] - }, - "expected": "items[3]:\n - first\n - second\n -", - "specSection": "10", - "note": "Empty object list items encode as a single \"-\" line at the list-item depth" - }, { "name": "uses field order from first object for tabular headers", "input": { diff --git a/tests/fixtures/encode/arrays-primitive.json b/tests/fixtures/encode/arrays-primitive.json index 5644091..f0f01e6 100644 --- a/tests/fixtures/encode/arrays-primitive.json +++ b/tests/fixtures/encode/arrays-primitive.json @@ -3,22 +3,6 @@ "category": "encode", "description": "Primitive array encoding – inline arrays of strings, numbers, booleans", "tests": [ - { - "name": "encodes string arrays inline", - "input": { - "tags": ["reading", "gaming"] - }, - "expected": "tags[2]: reading,gaming", - "specSection": "9.1" - }, - { - "name": "encodes number arrays inline", - "input": { - "nums": [1, 2, 3] - }, - "expected": "nums[3]: 1,2,3", - "specSection": "9.1" - }, { "name": "encodes mixed primitive arrays inline", "input": { diff --git a/tests/fixtures/encode/delimiters.json b/tests/fixtures/encode/delimiters.json index 9954553..b8fbc61 100644 --- a/tests/fixtures/encode/delimiters.json +++ b/tests/fixtures/encode/delimiters.json @@ -122,17 +122,6 @@ }, "specSection": "11" }, - { - "name": "quotes strings containing tab delimiter", - "input": { - "items": ["a", "b\tc", "d"] - }, - "expected": "items[3\t]: a\t\"b\\tc\"\td", - "options": { - "delimiter": "\t" - }, - "specSection": "11" - }, { "name": "quotes strings containing pipe delimiter", "input": { @@ -155,31 +144,6 @@ }, "specSection": "11" }, - { - "name": "does not quote commas with pipe delimiter", - "input": { - "items": ["a,b", "c,d"] - }, - "expected": "items[2|]: a,b|c,d", - "options": { - "delimiter": "|" - }, - "specSection": "11" - }, - { - "name": "quotes tabular values containing comma delimiter", - "input": { - "items": [ - { "id": 1, "note": "a,b" }, - { "id": 2, "note": "c,d" } - ] - }, - "expected": "items[2]{id,note}:\n 1,\"a,b\"\n 2,\"c,d\"", - "options": { - "delimiter": "," - }, - "specSection": "11" - }, { "name": "does not quote commas in tabular values with tab delimiter", "input": { @@ -205,17 +169,6 @@ }, "specSection": "11" }, - { - "name": "does not quote commas in object values with tab delimiter", - "input": { - "note": "a,b" - }, - "expected": "note: a,b", - "options": { - "delimiter": "\t" - }, - "specSection": "11" - }, { "name": "quotes nested array values containing pipe delimiter", "input": { @@ -227,28 +180,6 @@ }, "specSection": "11" }, - { - "name": "quotes nested array values containing tab delimiter", - "input": { - "pairs": [["a", "b\tc"]] - }, - "expected": "pairs[1\t]:\n - [2\t]: a\t\"b\\tc\"", - "options": { - "delimiter": "\t" - }, - "specSection": "11" - }, - { - "name": "preserves ambiguity quoting regardless of delimiter", - "input": { - "items": ["true", "42", "-3.14"] - }, - "expected": "items[3|]: \"true\"|\"42\"|\"-3.14\"", - "options": { - "delimiter": "|" - }, - "specSection": "11" - }, { "name": "quotes a primitive list item containing the document delimiter", "input": { diff --git a/tests/fixtures/encode/objects.json b/tests/fixtures/encode/objects.json index 4d54d97..c3cf7c2 100644 --- a/tests/fixtures/encode/objects.json +++ b/tests/fixtures/encode/objects.json @@ -259,15 +259,6 @@ "specSection": "15", "note": "No key has special meaning, so the encoder emits prototype-named own entries like any other (§8)" }, - { - "name": "encodes constructor and prototype own properties as ordinary keys", - "input": { - "constructor": 1, - "prototype": 2 - }, - "expected": "constructor: 1\nprototype: 2", - "specSection": "15" - }, { "name": "encodes __proto__ as a tabular field name", "input": { diff --git a/tests/fixtures/encode/primitives.json b/tests/fixtures/encode/primitives.json index 0f12f0c..c928dd8 100644 --- a/tests/fixtures/encode/primitives.json +++ b/tests/fixtures/encode/primitives.json @@ -149,18 +149,6 @@ "expected": "café", "specSection": "7.2" }, - { - "name": "encodes Chinese characters without quotes", - "input": "你好", - "expected": "你好", - "specSection": "7.2" - }, - { - "name": "encodes emoji without quotes", - "input": "🚀", - "expected": "🚀", - "specSection": "7.2" - }, { "name": "encodes string with emoji and spaces", "input": "hello 👋 world", diff --git a/tests/fixtures/encode/whitespace.json b/tests/fixtures/encode/whitespace.json index 1c6eae3..f2c01b7 100644 --- a/tests/fixtures/encode/whitespace.json +++ b/tests/fixtures/encode/whitespace.json @@ -3,14 +3,6 @@ "category": "encode", "description": "Whitespace and formatting invariants – no trailing spaces, no trailing newlines", "tests": [ - { - "name": "produces no trailing newline at end of output", - "input": { - "id": 123 - }, - "expected": "id: 123", - "specSection": "12" - }, { "name": "maintains proper indentation for nested structures", "input": { From 0024f0cb022ce4a496a620df98b908f958223def Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 14:05:24 +0200 Subject: [PATCH 31/36] test: correct fixture names, notes, and spec sections --- tests/fixtures/decode/arrays-nested.json | 16 +++---- tests/fixtures/decode/blank-lines.json | 2 +- tests/fixtures/decode/comments.json | 12 ++---- tests/fixtures/decode/delimiters.json | 2 +- tests/fixtures/decode/indentation-errors.json | 10 ++--- tests/fixtures/decode/numbers.json | 5 +-- tests/fixtures/decode/objects.json | 42 +++++++++---------- tests/fixtures/decode/primitives.json | 16 +++---- tests/fixtures/decode/validation-errors.json | 6 +-- tests/fixtures/decode/whitespace.json | 3 +- tests/fixtures/encode/arrays-nested.json | 13 +----- tests/fixtures/encode/arrays-primitive.json | 2 +- tests/fixtures/encode/objects-keyed.json | 8 +--- tests/fixtures/encode/objects.json | 2 +- 14 files changed, 58 insertions(+), 81 deletions(-) diff --git a/tests/fixtures/decode/arrays-nested.json b/tests/fixtures/decode/arrays-nested.json index 685deba..e8c6a95 100644 --- a/tests/fixtures/decode/arrays-nested.json +++ b/tests/fixtures/decode/arrays-nested.json @@ -84,24 +84,20 @@ "specSection": "10" }, { - "name": "parses objects containing arrays (including empty arrays) in list form", + "name": "parses a legacy [0] header as a later field of a list-item object", "input": "items[1]:\n - name: Ada\n data[0]:", "expected": { - "items": [ - { "name": "Ada", "data": [] } - ] + "items": [{ "name": "Ada", "data": [] }] }, - "specSection": "9.4" + "specSection": "10" }, { - "name": "parses arrays of arrays within objects", + "name": "parses a list-form array as the first field of a list-item object", "input": "items[1]:\n - matrix[2]:\n - [2]: 1,2\n - [2]: 3,4\n name: grid", "expected": { - "items": [ - { "matrix": [[1, 2], [3, 4]], "name": "grid" } - ] + "items": [{ "matrix": [[1, 2], [3, 4]], "name": "grid" }] }, - "specSection": "9.2" + "specSection": "10" }, { "name": "parses quoted strings and mixed lengths in nested arrays", diff --git a/tests/fixtures/decode/blank-lines.json b/tests/fixtures/decode/blank-lines.json index fa2afa9..f709078 100644 --- a/tests/fixtures/decode/blank-lines.json +++ b/tests/fixtures/decode/blank-lines.json @@ -76,7 +76,7 @@ "specSection": "12" }, { - "name": "accepts whitespace-only line at non-multiple indent as blank in strict mode", + "name": "accepts a spaces-only line at non-multiple indent as blank in strict mode", "input": "a: 1\n \nb: 2", "expected": { "a": 1, diff --git a/tests/fixtures/decode/comments.json b/tests/fixtures/decode/comments.json index 0eb5ac3..77ceb77 100644 --- a/tests/fixtures/decode/comments.json +++ b/tests/fixtures/decode/comments.json @@ -106,24 +106,20 @@ "input": "items[2]{id}:\n #1\n 2", "expected": null, "shouldError": true, - "specSection": "14.1", + "specSection": "5.1", "note": "The first cell makes the whole line a comment; only one row remains for [2]" }, { - "name": "drops hash-leading row silently in non-strict mode", + "name": "strips a hash-leading row as a comment in non-strict mode", "input": "items[2]{id}:\n #1\n 2", "expected": { - "items": [ - { - "id": 2 - } - ] + "items": [{ "id": 2 }] }, "options": { "strict": false }, "specSection": "5.1", - "note": "A hash-leading first cell makes the whole line a comment; non-strict mode drops the row silently" + "note": "Comment removal applies in both modes; the resulting count mismatch is not an error when strict=false (§14.1)" }, { "name": "throws on tab-indented hash line, which is not a comment", diff --git a/tests/fixtures/decode/delimiters.json b/tests/fixtures/decode/delimiters.json index c67e681..65cd282 100644 --- a/tests/fixtures/decode/delimiters.json +++ b/tests/fixtures/decode/delimiters.json @@ -58,7 +58,7 @@ "specSection": "11" }, { - "name": "parses nested arrays inside list items with default comma delimiter", + "name": "parses nested arrays inside list items with default comma delimiter when parent uses tab", "input": "items[1\t]:\n - tags[3]: a,b,c", "expected": { "items": [{ "tags": ["a", "b", "c"] }] diff --git a/tests/fixtures/decode/indentation-errors.json b/tests/fixtures/decode/indentation-errors.json index 396e5cb..814d15d 100644 --- a/tests/fixtures/decode/indentation-errors.json +++ b/tests/fixtures/decode/indentation-errors.json @@ -148,7 +148,7 @@ "input": "a:\n b: 1", "expected": null, "shouldError": true, - "specSection": "14.2", + "specSection": "8", "note": "Four leading spaces are a multiple of indentSize 2, but depth 2 directly under depth 0 skips a level" }, { @@ -156,7 +156,7 @@ "input": "a:\n b:\n c: 1", "expected": null, "shouldError": true, - "specSection": "14.2" + "specSection": "8" }, { "name": "throws on over-indented line after a primitive field", @@ -171,7 +171,7 @@ "input": "a: 1\n b: 2", "expected": null, "shouldError": true, - "specSection": "14.2", + "specSection": "8", "note": "Even a single extra level is over-indentation when the preceding line did not open a scope" }, { @@ -179,7 +179,7 @@ "input": "a:\n b: 1\n c: 2", "expected": null, "shouldError": true, - "specSection": "14.2", + "specSection": "8", "note": "The depth-3 line follows a primitive field at depth 1; it belongs to no scope" }, { @@ -187,7 +187,7 @@ "input": "u[1]{a}:\n 1\n junk: 9", "expected": null, "shouldError": true, - "specSection": "14.2" + "specSection": "8" }, { "name": "throws on orphan scalar line under a primitive field in non-strict mode", diff --git a/tests/fixtures/decode/numbers.json b/tests/fixtures/decode/numbers.json index d13c843..bf23d1c 100644 --- a/tests/fixtures/decode/numbers.json +++ b/tests/fixtures/decode/numbers.json @@ -42,7 +42,7 @@ "value": 0 }, "specSection": "4", - "note": "Negative zero decodes to 0; most host environments do not distinguish -0 from 0" + "note": "Most host environments do not distinguish -0 from 0" }, { "name": "parses negative zero with fractional part", @@ -58,8 +58,7 @@ "expected": { "nums": [42, -1000, 1.5, 0, 250] }, - "specSection": "4", - "note": "Decoders normalize all numeric forms to host numeric values" + "specSection": "4" }, { "name": "treats leading zero as string not number", diff --git a/tests/fixtures/decode/objects.json b/tests/fixtures/decode/objects.json index 7fbb48d..153da07 100644 --- a/tests/fixtures/decode/objects.json +++ b/tests/fixtures/decode/objects.json @@ -23,7 +23,7 @@ "specSection": "8" }, { - "name": "parses empty nested object header", + "name": "decodes a bare key: as an empty object, not an empty array", "input": "user:", "expected": { "user": {} @@ -55,7 +55,7 @@ "expected": { "note": "a:b" }, - "specSection": "8" + "specSection": "4" }, { "name": "parses quoted object value with comma", @@ -63,7 +63,7 @@ "expected": { "note": "a,b" }, - "specSection": "8" + "specSection": "4" }, { "name": "parses quoted object value with newline escape", @@ -87,7 +87,7 @@ "expected": { "text": " padded " }, - "specSection": "8" + "specSection": "4" }, { "name": "parses quoted string value that looks like true", @@ -95,7 +95,7 @@ "expected": { "v": "true" }, - "specSection": "8" + "specSection": "4" }, { "name": "parses quoted string value that looks like integer", @@ -103,7 +103,7 @@ "expected": { "v": "42" }, - "specSection": "8" + "specSection": "4" }, { "name": "parses quoted object value shaped like an inline array header", @@ -120,7 +120,7 @@ "expected": { "a": "[1]: x" }, - "specSection": "8", + "specSection": "4", "note": "Quoting makes the value opaque (§4), so it is never parsed as a header – the matching count is irrelevant because a quoted scalar is never a header candidate" }, { @@ -129,7 +129,7 @@ "expected": { "key": "foo [2]: bar" }, - "specSection": "6", + "specSection": "5.2", "note": "An unquoted key cannot contain a colon, so the first colon ends the key; the bracket-and-colon shape in the remainder is value text, not an array header" }, { @@ -138,7 +138,7 @@ "expected": { "a": "b [2]{x,y}: 1,2" }, - "specSection": "6", + "specSection": "5.2", "note": "An unquoted key cannot contain a colon, so the first colon ends the key; the tabular-header shape in the remainder is value text, not an array header" }, { @@ -147,7 +147,7 @@ "expected": { "content": "accept (process.argv[2]) and greet: name" }, - "specSection": "8" + "specSection": "5.2" }, { "name": "parses quoted key with colon", @@ -155,7 +155,7 @@ "expected": { "order:id": 7 }, - "specSection": "8" + "specSection": "7.4" }, { "name": "decodes \\uXXXX in quoted key (U+0004 control character)", @@ -179,7 +179,7 @@ "expected": { "[index]": 5 }, - "specSection": "8" + "specSection": "7.4" }, { "name": "treats extra brackets after valid array segment as literal key (non-strict)", @@ -235,7 +235,7 @@ "expected": { "{key}": 5 }, - "specSection": "8" + "specSection": "7.4" }, { "name": "parses quoted key with comma", @@ -243,7 +243,7 @@ "expected": { "a,b": 1 }, - "specSection": "8" + "specSection": "7.4" }, { "name": "parses quoted key with spaces", @@ -251,7 +251,7 @@ "expected": { "full name": "Ada" }, - "specSection": "8" + "specSection": "7.4" }, { "name": "parses quoted key with leading hyphen", @@ -259,7 +259,7 @@ "expected": { "-lead": 1 }, - "specSection": "8" + "specSection": "7.4" }, { "name": "parses quoted key with leading and trailing spaces", @@ -267,7 +267,7 @@ "expected": { " a ": 1 }, - "specSection": "8" + "specSection": "7.4" }, { "name": "parses quoted numeric key", @@ -275,7 +275,7 @@ "expected": { "123": "x" }, - "specSection": "8" + "specSection": "7.4" }, { "name": "parses quoted empty string key", @@ -283,7 +283,7 @@ "expected": { "": 1 }, - "specSection": "8" + "specSection": "7.4" }, { "name": "parses dotted keys as identifiers", @@ -447,7 +447,7 @@ "specSection": "15" }, { - "name": "falls through to a key-value line when whitespace precedes the bracket segment", + "name": "falls through to a key-value line when whitespace precedes the bracket segment (non-strict)", "input": "foo [2]: bar,baz", "expected": { "foo [2]": "bar,baz" @@ -584,7 +584,7 @@ "expected": { "a[2": "]{x}" }, - "specSection": "6", + "specSection": "5.2", "minSpecVersion": "4.2" } ] diff --git a/tests/fixtures/decode/primitives.json b/tests/fixtures/decode/primitives.json index f47fcf4..14441d4 100644 --- a/tests/fixtures/decode/primitives.json +++ b/tests/fixtures/decode/primitives.json @@ -13,7 +13,7 @@ "name": "parses empty quoted string", "input": "\"\"", "expected": "", - "specSection": "7.4" + "specSection": "4" }, { "name": "parses quoted string with newline escape", @@ -59,21 +59,21 @@ }, { "name": "parses Unicode string", - "input": "café", - "expected": "café", - "specSection": "7.4" + "input": "caf\u00e9", + "expected": "caf\u00e9", + "specSection": "16" }, { "name": "parses Chinese characters", "input": "你好", "expected": "你好", - "specSection": "7.4" + "specSection": "16" }, { "name": "parses string with emoji and spaces", "input": "hello 👋 world", "expected": "hello 👋 world", - "specSection": "7.4" + "specSection": "16" }, { "name": "parses positive integer", @@ -115,13 +115,13 @@ "name": "respects ambiguity quoting for true", "input": "\"true\"", "expected": "true", - "specSection": "7.4" + "specSection": "4" }, { "name": "respects ambiguity quoting for integer", "input": "\"42\"", "expected": "42", - "specSection": "7.4" + "specSection": "4" }, { "name": "decodes supplementary scalar (U+1F680) in quoted string as literal UTF-8", diff --git a/tests/fixtures/decode/validation-errors.json b/tests/fixtures/decode/validation-errors.json index c085dbb..0abf805 100644 --- a/tests/fixtures/decode/validation-errors.json +++ b/tests/fixtures/decode/validation-errors.json @@ -36,7 +36,7 @@ "input": "\"a\\x\"", "expected": null, "shouldError": true, - "specSection": "14.2" + "specSection": "7.1" }, { "name": "throws on truncated unicode escape \\u00b", @@ -57,7 +57,7 @@ "input": "\"unterminated", "expected": null, "shouldError": true, - "specSection": "14.2" + "specSection": "7.1" }, { "name": "throws on missing colon in key-value context", @@ -81,7 +81,7 @@ "input": "items[2\t]{a\tb}:\n 1,2\n 3,4", "expected": null, "shouldError": true, - "specSection": "14.1", + "specSection": "11.2", "note": "Active delimiter is tab; rows using comma each parse as 1 value, failing the row width check" }, { diff --git a/tests/fixtures/decode/whitespace.json b/tests/fixtures/decode/whitespace.json index f527885..3384d7e 100644 --- a/tests/fixtures/decode/whitespace.json +++ b/tests/fixtures/decode/whitespace.json @@ -9,8 +9,7 @@ "expected": { "tags": ["a", "b", "c"] }, - "specSection": "12", - "note": "Trimming removes U+0020 only, never other Unicode whitespace" + "specSection": "12" }, { "name": "tolerates spaces around pipes in inline arrays", diff --git a/tests/fixtures/encode/arrays-nested.json b/tests/fixtures/encode/arrays-nested.json index be84301..079ddfb 100644 --- a/tests/fixtures/encode/arrays-nested.json +++ b/tests/fixtures/encode/arrays-nested.json @@ -82,19 +82,10 @@ { "name": "uses list form for a tabular-eligible array in list-item position", "input": { - "a": [ - [ - { - "x": 1 - }, - { - "x": 2 - } - ] - ] + "a": [[{ "x": 1 }, { "x": 2 }]] }, "expected": "a[1]:\n - [2]:\n - x: 1\n - x: 2", - "specSection": "9.3", + "specSection": "9.4", "note": "A keyless fields-bearing header is valid only at the document root", "minSpecVersion": "4.1" }, diff --git a/tests/fixtures/encode/arrays-primitive.json b/tests/fixtures/encode/arrays-primitive.json index f0f01e6..5a2ab0e 100644 --- a/tests/fixtures/encode/arrays-primitive.json +++ b/tests/fixtures/encode/arrays-primitive.json @@ -68,7 +68,7 @@ "specSection": "9.1" }, { - "name": "quotes strings that look like booleans in arrays", + "name": "quotes strings that look like literals or numbers in arrays", "input": { "items": ["x", "true", "42", "-3.14"] }, diff --git a/tests/fixtures/encode/objects-keyed.json b/tests/fixtures/encode/objects-keyed.json index 7fd4e87..255634c 100644 --- a/tests/fixtures/encode/objects-keyed.json +++ b/tests/fixtures/encode/objects-keyed.json @@ -62,14 +62,10 @@ { "name": "quotes entry keys per key encoding", "input": { - "ids": { - "42": { "v": 1 }, - "my key": { "v": 2 } - } + "ids": { "42": { "v": 1 }, "my key": { "v": 2 } } }, "expected": "ids[2:]{v}:\n \"42\": 1\n \"my key\": 2", - "specSection": "9.5", - "note": "Entry keys follow §7.3, the same rule as object keys" + "specSection": "9.5" }, { "name": "quotes entry-row cells containing the active delimiter", diff --git a/tests/fixtures/encode/objects.json b/tests/fixtures/encode/objects.json index c3cf7c2..2410e45 100644 --- a/tests/fixtures/encode/objects.json +++ b/tests/fixtures/encode/objects.json @@ -23,7 +23,7 @@ "specSection": "8" }, { - "name": "encodes empty objects as empty string", + "name": "encodes an empty root object as an empty document", "input": {}, "expected": "", "specSection": "8" From 0dd8182b8f4a945c9d8701100c85d4c0f42681e0 Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 14:05:30 +0200 Subject: [PATCH 32/36] test: sharpen fixture inputs to the rule each case pins --- tests/fixtures/decode/blank-lines.json | 6 ++--- tests/fixtures/decode/delimiters.json | 26 +++++++++---------- tests/fixtures/decode/indentation-errors.json | 17 +++++------- tests/fixtures/decode/objects-keyed.json | 11 +++----- tests/fixtures/decode/objects.json | 17 ++++++------ tests/fixtures/decode/validation-errors.json | 17 ++++++------ tests/fixtures/encode/arrays-tabular.json | 2 +- tests/fixtures/encode/objects.json | 15 ++++++----- tests/fixtures/encode/primitives.json | 25 ++++++++++-------- 9 files changed, 66 insertions(+), 70 deletions(-) diff --git a/tests/fixtures/decode/blank-lines.json b/tests/fixtures/decode/blank-lines.json index f709078..cdb8074 100644 --- a/tests/fixtures/decode/blank-lines.json +++ b/tests/fixtures/decode/blank-lines.json @@ -5,7 +5,7 @@ "tests": [ { "name": "throws on blank line inside list array", - "input": "items[3]:\n - a\n\n - b\n - c", + "input": "items[2]:\n - a\n\n - b", "expected": null, "shouldError": true, "options": { @@ -137,9 +137,9 @@ }, { "name": "ignores blank lines inside list array when strict=false", - "input": "items[3]:\n - a\n\n - b\n - c", + "input": "items[2]:\n - a\n\n - b", "expected": { - "items": ["a", "b", "c"] + "items": ["a", "b"] }, "options": { "strict": false diff --git a/tests/fixtures/decode/delimiters.json b/tests/fixtures/decode/delimiters.json index 65cd282..d2e6bcd 100644 --- a/tests/fixtures/decode/delimiters.json +++ b/tests/fixtures/decode/delimiters.json @@ -66,10 +66,10 @@ "specSection": "11" }, { - "name": "parses nested arrays inside list items with default comma delimiter when parent uses pipe", - "input": "items[1|]:\n - tags[3]: a,b,c", + "name": "parses a keyless inner array without a delimiter symbol as comma when parent uses pipe", + "input": "pairs[1|]:\n - [2]: a,b", "expected": { - "items": [{ "tags": ["a", "b", "c"] }] + "pairs": [["a", "b"]] }, "specSection": "11" }, @@ -87,19 +87,19 @@ }, { "name": "parses root-level array of objects with tab delimiter", - "input": "[2\t]{id}:\n 1\n 2", - "expected": [{ "id": 1 }, { "id": 2 }], + "input": "[1\t]{a\tb}:\n 1\t2", + "expected": [{ "a": 1, "b": 2 }], "specSection": "11" }, { "name": "parses root-level array of objects with pipe delimiter", - "input": "[2|]{id}:\n 1\n 2", - "expected": [{ "id": 1 }, { "id": 2 }], + "input": "[1|]{a|b}:\n 1|2", + "expected": [{ "a": 1, "b": 2 }], "specSection": "11" }, { "name": "parses values containing tab delimiter when quoted", - "input": "items[3\t]: a\t\"b\\tc\"\td", + "input": "items[3\t]: a\t\"b\tc\"\td", "expected": { "items": ["a", "b\tc", "d"] }, @@ -161,10 +161,10 @@ "note": "Object field values are parsed as the whole post-colon token, never split on a delimiter" }, { - "name": "does not split object values on commas inside a tab-delimited scope", - "input": "items[2\t]:\n - status: a,b\n - status: c,d", + "name": "does not split a list-item field value on the active delimiter", + "input": "items[1|]:\n - status: a|b", "expected": { - "items": [{ "status": "a,b" }, { "status": "c,d" }] + "items": [{ "status": "a|b" }] }, "specSection": "11.2", "note": "The active delimiter splits row and inline cells, never object field values" @@ -187,9 +187,9 @@ }, { "name": "falls through to a key-value line on a header delimiter mismatch", - "input": "a[2|]{x,y}: 1|2", + "input": "a[0|]{x,y}:", "expected": { - "a[2|]{x,y}": "1|2" + "a[0|]{x,y}": {} }, "options": { "strict": false diff --git a/tests/fixtures/decode/indentation-errors.json b/tests/fixtures/decode/indentation-errors.json index 814d15d..e9529bc 100644 --- a/tests/fixtures/decode/indentation-errors.json +++ b/tests/fixtures/decode/indentation-errors.json @@ -26,8 +26,8 @@ "specSection": "14.2" }, { - "name": "throws on non-multiple indentation with custom indentSize 4 (3 spaces)", - "input": "a:\n b: 1", + "name": "throws on 2-space indentation with custom indentSize 4", + "input": "a:\n b: 1", "expected": null, "shouldError": true, "options": { @@ -115,11 +115,9 @@ }, { "name": "accepts non-multiple indentation when strict=false", - "input": "a:\n b: 1", + "input": "a:\n b: 1\n c: 2", "expected": { - "a": { - "b": 1 - } + "a": { "b": 1, "c": 2 } }, "options": { "indentSize": 2, @@ -159,12 +157,11 @@ "specSection": "8" }, { - "name": "throws on over-indented line after a primitive field", - "input": "a: 1\n b: 2", + "name": "throws on an over-indented line under a primitive list item", + "input": "items[2]:\n - a\n b: 1\n - c", "expected": null, "shouldError": true, - "specSection": "14.2", - "note": "A primitive field opens no scope, so the depth-2 line belongs to none and must not be silently discarded" + "specSection": "8" }, { "name": "throws on a line indented one level under a primitive field", diff --git a/tests/fixtures/decode/objects-keyed.json b/tests/fixtures/decode/objects-keyed.json index ee13147..d62cfb2 100644 --- a/tests/fixtures/decode/objects-keyed.json +++ b/tests/fixtures/decode/objects-keyed.json @@ -93,16 +93,13 @@ "note": "A keyed scope ends only by dedent or end of input" }, { - "name": "treats key-value-shaped lines at entry depth as entry rows", - "input": "m[2:]{x}:\n total: 5\n other: 6", + "name": "splits an entry row at its first unquoted colon when a delimiter precedes it", + "input": "m[1:]{x}:\n a,b: 5", "expected": { - "m": { - "total": { "x": 5 }, - "other": { "x": 6 } - } + "m": { "a,b": { "x": 5 } } }, "specSection": "9.5", - "note": "Opposite polarity to §9.3: in a tabular array, a colon-before-delimiter line at row depth ends the rows; in a keyed scope it is an entry" + "note": "The colon split comes before the delimiter split, so the delimiter before the colon is part of the entry key" }, { "name": "splits an entry row at its first unquoted colon before delimiter splitting", diff --git a/tests/fixtures/decode/objects.json b/tests/fixtures/decode/objects.json index 153da07..5f00452 100644 --- a/tests/fixtures/decode/objects.json +++ b/tests/fixtures/decode/objects.json @@ -15,9 +15,8 @@ }, { "name": "parses null values in objects", - "input": "id: 123\nvalue: null", + "input": "value: null", "expected": { - "id": 123, "value": null }, "specSection": "8" @@ -31,12 +30,13 @@ "specSection": "8" }, { - "name": "bare key: with no children decodes as empty object, not array", - "input": "matches:", + "name": "decodes a bare key followed by trailing spaces as an empty object", + "input": "matches: ", "expected": { "matches": {} }, - "specSection": "8" + "specSection": "12", + "minSpecVersion": "4.1" }, { "name": "applies last-write-wins for duplicate sibling keys in non-strict mode", @@ -106,13 +106,12 @@ "specSection": "4" }, { - "name": "parses quoted object value shaped like an inline array header", - "input": "a: \"[2]: x\"", + "name": "parses an unquoted field value starting with a bracket segment as a string", + "input": "a: [2]: x", "expected": { "a": "[2]: x" }, - "specSection": "8", - "note": "Quoted value is opaque (§4); the bracket-and-colon shape inside the quotes is data, not an array header (§6)" + "specSection": "5.2" }, { "name": "parses quoted object value shaped like a count-matching inline array header", diff --git a/tests/fixtures/decode/validation-errors.json b/tests/fixtures/decode/validation-errors.json index 0abf805..68796e6 100644 --- a/tests/fixtures/decode/validation-errors.json +++ b/tests/fixtures/decode/validation-errors.json @@ -213,7 +213,7 @@ }, { "name": "throws on whitespace between bracket segment and colon in strict mode", - "input": "items[2] :\n 1,2", + "input": "items[2] : 1,2", "expected": null, "shouldError": true, "options": { @@ -224,7 +224,7 @@ }, { "name": "throws on whitespace between bracket segment and field list in strict mode", - "input": "items[2] {a,b}:\n 1,2\n 3,4", + "input": "items[0] {a,b}:", "expected": null, "shouldError": true, "options": { @@ -271,14 +271,14 @@ }, { "name": "throws on empty field list in strict mode", - "input": "items[1]{}:\n 1", + "input": "items[0]{}:", "expected": null, "shouldError": true, "specSection": "14.2" }, { "name": "throws on empty nested field group in strict mode", - "input": "items[1]{id,meta{}}:\n 1", + "input": "items[0]{id,meta{}}:", "expected": null, "shouldError": true, "specSection": "14.2", @@ -286,21 +286,20 @@ }, { "name": "throws on unmatched brace in field list in strict mode", - "input": "items[1]{id,customer{name:\n 1,Ada", + "input": "items[0]{id,customer{name:", "expected": null, "shouldError": true, "specSection": "14.2" }, { "name": "throws on duplicate field names at the same field-list level in strict mode", - "input": "items[1]{a,a{x}}:\n 1,2", + "input": "items[0]{a,a{x}}:", "expected": null, "shouldError": true, "options": { "strict": true }, - "specSection": "14.3", - "note": "A bare field and a nested field group sharing one name yield duplicate sibling keys in every decoded row" + "specSection": "9.3" }, { "name": "throws on entry row count mismatch with keyed header length", @@ -443,7 +442,7 @@ }, { "name": "throws on keyless fields-bearing header as list item", - "input": "items[1]:\n - [2]{x}:\n 1\n 2", + "input": "items[1]:\n - [0]{x}:", "expected": null, "shouldError": true, "specSection": "6" diff --git a/tests/fixtures/encode/arrays-tabular.json b/tests/fixtures/encode/arrays-tabular.json index 2365042..f013733 100644 --- a/tests/fixtures/encode/arrays-tabular.json +++ b/tests/fixtures/encode/arrays-tabular.json @@ -83,7 +83,7 @@ "input": { "orders": [ { "id": 1, "customer": { "name": "Ada", "country": "DK" }, "total": 99 }, - { "id": 2, "customer": { "name": "Bob", "country": "UK" }, "total": 149 } + { "id": 2, "customer": { "country": "UK", "name": "Bob" }, "total": 149 } ] }, "expected": "orders[2]{id,customer{name,country},total}:\n 1,Ada,DK,99\n 2,Bob,UK,149", diff --git a/tests/fixtures/encode/objects.json b/tests/fixtures/encode/objects.json index 2410e45..00ce231 100644 --- a/tests/fixtures/encode/objects.json +++ b/tests/fixtures/encode/objects.json @@ -93,11 +93,11 @@ "specSection": "7.2" }, { - "name": "quotes string value that looks like negative decimal", + "name": "quotes string value that looks like a number with an uppercase exponent", "input": { - "v": "-7.5" + "v": "1E6" }, - "expected": "v: \"-7.5\"", + "expected": "v: \"1E6\"", "specSection": "7.2" }, { @@ -182,12 +182,13 @@ "note": "The unquoted-key pattern is ASCII-only, so every non-ASCII key is quoted" }, { - "name": "quotes CJK key", + "name": "keeps a dotted key unquoted", "input": { - "名前": "x" + "a.b": 1 }, - "expected": "\"名前\": x", - "specSection": "7.3" + "expected": "a.b: 1", + "specSection": "7.3", + "minSpecVersion": "4.2" }, { "name": "escapes newline in key", diff --git a/tests/fixtures/encode/primitives.json b/tests/fixtures/encode/primitives.json index c928dd8..edaa4d0 100644 --- a/tests/fixtures/encode/primitives.json +++ b/tests/fixtures/encode/primitives.json @@ -46,9 +46,9 @@ "specSection": "7.2" }, { - "name": "quotes string that looks like negative decimal", - "input": "-3.14", - "expected": "\"-3.14\"", + "name": "quotes string that looks like a decimal", + "input": "3.14", + "expected": "\"3.14\"", "specSection": "7.2" }, { @@ -95,10 +95,11 @@ "specSection": "7.1" }, { - "name": "quotes string with array-like syntax", - "input": "[3]: x,y", - "expected": "\"[3]: x,y\"", - "specSection": "7.2" + "name": "keeps a string with an interior hash unquoted", + "input": "a #b", + "expected": "a #b", + "specSection": "7.2", + "minSpecVersion": "4.2" }, { "name": "quotes string starting with hyphen-space", @@ -127,8 +128,10 @@ }, { "name": "quotes leading-hyphen string in array", - "input": { "tags": ["a", "- item", "b"] }, - "expected": "tags[3]: a,\"- item\",b", + "input": { + "tags": ["-x"] + }, + "expected": "tags[1]: \"-x\"", "specSection": "7.2" }, { @@ -188,13 +191,13 @@ }, { "name": "encodes negative zero as zero", - "input": -0, + "input": -0.0, "expected": "0", "specSection": "2" }, { "name": "encodes large integer without exponent notation", - "input": 1000000, + "input": 1e6, "expected": "1000000", "specSection": "2", "note": "Within the canonical decimal range" From 02b8c1495c96ce7ace45b1dd64696452574d760a Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 14:05:43 +0200 Subject: [PATCH 33/36] test: cover port mistakes the fixture audit found --- tests/fixtures/decode/arrays-tabular.json | 8 +++ tests/fixtures/decode/blank-lines.json | 13 ++++ tests/fixtures/decode/delimiters.json | 8 +++ tests/fixtures/decode/indentation-errors.json | 7 ++ tests/fixtures/decode/numbers.json | 8 +++ tests/fixtures/decode/objects-keyed.json | 12 ++++ tests/fixtures/decode/validation-errors.json | 65 +++++++++++++++++++ tests/fixtures/decode/whitespace.json | 9 +++ tests/fixtures/encode/objects-keyed.json | 8 +++ tests/fixtures/encode/whitespace.json | 11 ++++ 10 files changed, 149 insertions(+) diff --git a/tests/fixtures/decode/arrays-tabular.json b/tests/fixtures/decode/arrays-tabular.json index ac9ed00..3730f6c 100644 --- a/tests/fixtures/decode/arrays-tabular.json +++ b/tests/fixtures/decode/arrays-tabular.json @@ -306,6 +306,14 @@ ], "specSection": "6", "minSpecVersion": "4.2" + }, + { + "name": "unescapes a quoted field name", + "input": "a[1]{\"x\\\"y\"}:\n 1", + "expected": { + "a": [{ "x\"y": 1 }] + }, + "specSection": "6" } ] } diff --git a/tests/fixtures/decode/blank-lines.json b/tests/fixtures/decode/blank-lines.json index cdb8074..c456cca 100644 --- a/tests/fixtures/decode/blank-lines.json +++ b/tests/fixtures/decode/blank-lines.json @@ -228,6 +228,19 @@ "shouldError": true, "specSection": "14.2", "minSpecVersion": "4.2" + }, + { + "name": "treats a tab-only line as blank in non-strict mode", + "input": "a: 1\n\t\nb: 2", + "expected": { + "a": 1, + "b": 2 + }, + "options": { + "strict": false + }, + "specSection": "12", + "minSpecVersion": "4.2" } ] } diff --git a/tests/fixtures/decode/delimiters.json b/tests/fixtures/decode/delimiters.json index d2e6bcd..45b01d7 100644 --- a/tests/fixtures/decode/delimiters.json +++ b/tests/fixtures/decode/delimiters.json @@ -208,6 +208,14 @@ }, "specSection": "5.2", "minSpecVersion": "4.2" + }, + { + "name": "parses a nested tabular header with its own comma delimiter inside a pipe list", + "input": "items[1|]:\n - t[1]{a,b}:\n 1,2", + "expected": { + "items": [{ "t": [{ "a": 1, "b": 2 }] }] + }, + "specSection": "11.2" } ] } diff --git a/tests/fixtures/decode/indentation-errors.json b/tests/fixtures/decode/indentation-errors.json index e9529bc..bf42b5c 100644 --- a/tests/fixtures/decode/indentation-errors.json +++ b/tests/fixtures/decode/indentation-errors.json @@ -298,6 +298,13 @@ }, "specSection": "8", "minSpecVersion": "4.2" + }, + { + "name": "throws on a depth jump to the first tabular row", + "input": "rows[1]{a}:\n 1", + "expected": null, + "shouldError": true, + "specSection": "8" } ] } diff --git a/tests/fixtures/decode/numbers.json b/tests/fixtures/decode/numbers.json index bf23d1c..df749d9 100644 --- a/tests/fixtures/decode/numbers.json +++ b/tests/fixtures/decode/numbers.json @@ -154,6 +154,14 @@ }, "specSection": "4", "note": "Host parsers that accept these forms have a wider grammar than the spec permits" + }, + { + "name": "treats non-ASCII digits as a string", + "input": "value: ٤٢", + "expected": { + "value": "٤٢" + }, + "specSection": "4" } ] } diff --git a/tests/fixtures/decode/objects-keyed.json b/tests/fixtures/decode/objects-keyed.json index d62cfb2..1819405 100644 --- a/tests/fixtures/decode/objects-keyed.json +++ b/tests/fixtures/decode/objects-keyed.json @@ -212,6 +212,18 @@ }, "specSection": "9.5", "minSpecVersion": "4.1" + }, + { + "name": "decodes every entry row beyond the declared count in non-strict mode", + "input": "m[1:]{v}:\n a: 1\n b: 2", + "expected": { + "m": { "a": { "v": 1 }, "b": { "v": 2 } } + }, + "options": { + "strict": false + }, + "specSection": "14.1", + "minSpecVersion": "4.1" } ] } diff --git a/tests/fixtures/decode/validation-errors.json b/tests/fixtures/decode/validation-errors.json index 68796e6..48055b5 100644 --- a/tests/fixtures/decode/validation-errors.json +++ b/tests/fixtures/decode/validation-errors.json @@ -663,6 +663,71 @@ "shouldError": true, "specSection": "6", "minSpecVersion": "4.2" + }, + { + "name": "throws on a \\u escape with a sign before its hex digits", + "input": "val: \"\\u+0ab\"", + "expected": null, + "shouldError": true, + "specSection": "7.1" + }, + { + "name": "throws on an escaped lone low surrogate", + "input": "val: \"\\udc00\"", + "expected": null, + "shouldError": true, + "specSection": "7.1" + }, + { + "name": "throws on characters after a closing quote in an inline array cell", + "input": "k[2]: \"a\"x,b", + "expected": null, + "shouldError": true, + "specSection": "7.4", + "minSpecVersion": "4.1" + }, + { + "name": "throws on a key-value line among tabular rows that breaks the declared count", + "input": "t[2]{a}:\n 1\n x: 2", + "expected": null, + "shouldError": true, + "specSection": "9.3" + }, + { + "name": "throws on a tabular row with more cells than fields", + "input": "items[1]{a}:\n 1,2", + "expected": null, + "shouldError": true, + "specSection": "14.1" + }, + { + "name": "throws on more keyed entry rows than declared", + "input": "m[1:]{v}:\n a: 1\n b: 2", + "expected": null, + "shouldError": true, + "specSection": "14.1" + }, + { + "name": "throws on an invalid escape in non-strict mode", + "input": "k: \"a\\x\"", + "expected": null, + "shouldError": true, + "options": { + "strict": false + }, + "specSection": "7.1", + "minSpecVersion": "4.1" + }, + { + "name": "throws on an unterminated string in non-strict mode", + "input": "k: \"abc", + "expected": null, + "shouldError": true, + "options": { + "strict": false + }, + "specSection": "7.1", + "minSpecVersion": "4.1" } ] } diff --git a/tests/fixtures/decode/whitespace.json b/tests/fixtures/decode/whitespace.json index 3384d7e..06eed2f 100644 --- a/tests/fixtures/decode/whitespace.json +++ b/tests/fixtures/decode/whitespace.json @@ -160,6 +160,15 @@ }, "specSection": "12", "minSpecVersion": "4.1" + }, + { + "name": "removes a byte-order mark before a comment line", + "input": "\ufeff# c\na: 1", + "expected": { + "a": 1 + }, + "specSection": "12", + "minSpecVersion": "4.1" } ] } diff --git a/tests/fixtures/encode/objects-keyed.json b/tests/fixtures/encode/objects-keyed.json index 255634c..d8841f5 100644 --- a/tests/fixtures/encode/objects-keyed.json +++ b/tests/fixtures/encode/objects-keyed.json @@ -146,6 +146,14 @@ "expected": "items[2]:\n - a:\n x: 1\n b:\n x: 2\n - 5", "specSection": "9.5", "note": "The element object is keyed-eligible but has no key; the keyless form is root-only, so it encodes per §10" + }, + { + "name": "keeps an object of empty objects out of keyed tabular form", + "input": { + "m": { "a": {}, "b": {} } + }, + "expected": "m:\n a:\n b:", + "specSection": "9.5" } ] } diff --git a/tests/fixtures/encode/whitespace.json b/tests/fixtures/encode/whitespace.json index f2c01b7..07dda0a 100644 --- a/tests/fixtures/encode/whitespace.json +++ b/tests/fixtures/encode/whitespace.json @@ -38,6 +38,17 @@ "specSection": "7.2", "note": "The whitespace trigger is U+0020 and U+0009 only", "minSpecVersion": "4.1" + }, + { + "name": "indents list-item sibling fields by indentSize", + "input": { + "a": [{ "b": 1, "c": 2 }, 1] + }, + "expected": "a[2]:\n - b: 1\n c: 2\n - 1", + "options": { + "indentSize": 4 + }, + "specSection": "10" } ] } From 97667bc6a9fd7fa97436afce867a13f94355cc81 Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 14:05:56 +0200 Subject: [PATCH 34/36] docs: state that nested headers in a list declare their own delimiter --- SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SPEC.md b/SPEC.md index b1e9f90..96ebd9e 100644 --- a/SPEC.md +++ b/SPEC.md @@ -542,7 +542,7 @@ When tabular requirements are not met (encoding; including any column that is ne - Object: formatted per §10 (objects as list items). Decoding: -- Header declares list length N and the active delimiter for any nested inline arrays. +- Header declares list length N; nested headers declare their own delimiter (§6). - Each list item is a list-item line (§5.2) starting with "- " at depth +1 (or the bare marker "-" for an empty object list item, §10) and is parsed as: - Primitive (no colon and no array header), - Inline primitive array (`- [M]: …`) or the empty-array item `- []` (§9.2), From 1d0e69d40a294b03927e599c1225cd90f6b272fc Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 14:05:56 +0200 Subject: [PATCH 35/36] docs: require literal UTF-8 and lowercase hex in encoder escapes --- SPEC.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/SPEC.md b/SPEC.md index 96ebd9e..0ed3a3c 100644 --- a/SPEC.md +++ b/SPEC.md @@ -394,9 +394,9 @@ In quoted strings and keys, codepoints are encoded according to the following ta | LF (U+000A) | MUST emit `\n` | MUST decode `\n` → LF | | CR (U+000D) | MUST emit `\r` | MUST decode `\r` → CR | | HTAB (U+0009) | MUST emit `\t` | MUST decode `\t` → HTAB | -| Other U+0000–U+001F controls | MUST emit `\uXXXX` (lowercase hex SHOULD) | MUST decode `\uXXXX` (case-insensitive hex) | +| Other U+0000–U+001F controls | MUST emit `\uXXXX` with lowercase hex | MUST decode `\uXXXX` (case-insensitive hex) | | U+D800–U+DFFF surrogates | (not produced by valid encoders) | MUST reject when decoded from `\uXXXX`, lone or paired | -| Other BMP codepoints (U+0020–U+D7FF, U+E000–U+FFFF) | SHOULD emit literal UTF-8; MAY emit `\uXXXX` | MUST accept either form | +| Other BMP codepoints (U+0020–U+D7FF, U+E000–U+FFFF) | MUST emit literal UTF-8 | MUST accept either form | | Supplementary scalar values (U+10000–U+10FFFF) | MUST emit as literal UTF-8 | MUST accept literal UTF-8; surrogate `\uXXXX` escapes MUST be rejected (see row above) | Decoders MUST reject any escape sequence not listed above, MUST reject `\u` followed by fewer than four hex digits, and MUST reject unterminated strings. From 66fade89cdd3f9145be6399cadc731622b19e763 Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Mon, 5 Oct 2026 14:06:08 +0200 Subject: [PATCH 36/36] docs: rewrite the 4.2 changelog for implementers --- CHANGELOG.md | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6b2077b..ef163da 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,20 +6,25 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). ## [4.2] - Unreleased -### Added - -- §5.2: a `"` opens a quoted span wherever it occurs in a line, so a colon, delimiter, bracket, or brace between it and the next unescaped `"` (or the line end) is quoted. -- §14: a non-strict decoder applies the leniency the specification names, or errors where that leniency is optional; 4.2 adds three – a depth jump's first line sets the scope's content depth (§8), ill-formed UTF-8 MAY become U+FFFD (§4), and unmatched braces MAY fall through to key-value parsing (§6). -- Conformance fixtures for the new decoder rules, the shortest digits, and list-item quoting. - ### Changed -- Encoder output is narrowed: numbers use the shortest round-trip digits (§2), strings and keys stay unquoted unless §7.2 or §7.3 requires quotes, and primitive list items are quoted by the document delimiter like object field values (§7.2, §11.1). -- Decoder edge cases are settled: a line without an unquoted colon is never a header (§6); an indented first line is over-indented (§5, §8); a hyphen marks a list item only at item depth (§5.2); an empty key token is the empty key and an empty field entry or nameless nested group is a header syntax error (§6, §7.4); whitespace between a field name and its nested field group is a header syntax error (§6); a length beyond the integer range still forms a header (§6); raw control characters inside quoted strings decode as themselves (§7.1); and a line of tabs is blank only under the non-strict tab leniency (§12). +- Encoders: numbers use the shortest round-trip digits (§2); non-ASCII text is emitted as literal UTF-8 and `\uXXXX` escapes use lowercase hex (§7.1); strings and keys are quoted only where §7.2 or §7.3 requires it; primitive list items are quoted against the document delimiter, like object field values (§11.1). +- Decoders – edge cases that 4.1 left open: + - a `"` opens a quoted span anywhere in a line (§5.2) + - a hyphen marks a list item only at item depth (§5.2) + - a line without an unquoted colon is never a header (§6) + - an indented first line is over-indented (§8) + - `: 1` decodes to the empty key; `{a,}` and `{a,{b}}` are header errors (§6, §7.4) + - whitespace before a nested field group is a header error (§6) + - a length beyond the integer range still forms a header (§6) + - raw control characters inside quotes decode as themselves (§7.1) + - a line of tabs is not blank, except under the non-strict tab leniency (§12) +- Non-strict mode: decoders apply only the leniencies the spec names, or error (§14). New in 4.2: a jumped first line sets the scope's depth (§8), ill-formed UTF-8 may become U+FFFD (§4), and unmatched braces may fall through to key-value parsing (§6). +- Conformance fixtures: cases for every change above; redundant cases removed, so fixture indices shift – runners that skip cases by index need new skip lists. ### Compatibility -Every decoder change concerns input no conforming 4.0 or 4.1 encoder emits: §7.1–§7.3 escape every control character and quote any string or key that is empty, starts with `-`, or contains `"`, a bracket, or a brace, and encoders never indent the first line, indent with tabs, put a space before a nested field group, or trigger a §14 condition. Each is MINOR under the encoder-unreachability rule in [VERSIONING.md](./VERSIONING.md). The encoder changes are tightenings: decoders read the output of every 4.0 or 4.1 encoder unchanged. +Every decoder change concerns input no conforming 4.0 or 4.1 encoder emits, so no round-trip is affected. Encoders that quote more than required, print more digits than the shortest round-trip form, escape non-ASCII text, or quote list items against the active delimiter need an update; their output still decodes unchanged. ## [4.1] - 2026-07-26