diff --git a/CHANGELOG.md b/CHANGELOG.md index dc7e431..ef163da 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,28 @@ All notable changes to the TOON specification will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). The project follows the MAJOR.MINOR versioning policy described in [VERSIONING.md](./VERSIONING.md). +## [4.2] - Unreleased + +### Changed + +- Encoders: numbers use the shortest round-trip digits (§2); non-ASCII text is emitted as literal UTF-8 and `\uXXXX` escapes use lowercase hex (§7.1); strings and keys are quoted only where §7.2 or §7.3 requires it; primitive list items are quoted against the document delimiter, like object field values (§11.1). +- Decoders – edge cases that 4.1 left open: + - a `"` opens a quoted span anywhere in a line (§5.2) + - a hyphen marks a list item only at item depth (§5.2) + - a line without an unquoted colon is never a header (§6) + - an indented first line is over-indented (§8) + - `: 1` decodes to the empty key; `{a,}` and `{a,{b}}` are header errors (§6, §7.4) + - whitespace before a nested field group is a header error (§6) + - a length beyond the integer range still forms a header (§6) + - raw control characters inside quotes decode as themselves (§7.1) + - a line of tabs is not blank, except under the non-strict tab leniency (§12) +- Non-strict mode: decoders apply only the leniencies the spec names, or error (§14). New in 4.2: a jumped first line sets the scope's depth (§8), ill-formed UTF-8 may become U+FFFD (§4), and unmatched braces may fall through to key-value parsing (§6). +- Conformance fixtures: cases for every change above; redundant cases removed, so fixture indices shift – runners that skip cases by index need new skip lists. + +### Compatibility + +Every decoder change concerns input no conforming 4.0 or 4.1 encoder emits, so no round-trip is affected. Encoders that quote more than required, print more digits than the shortest round-trip form, escape non-ASCII text, or quote list items against the active delimiter need an update; their output still decodes unchanged. + ## [4.1] - 2026-07-26 ### Added diff --git a/SPEC.md b/SPEC.md index 007377a..0ed3a3c 100644 --- a/SPEC.md +++ b/SPEC.md @@ -2,7 +2,7 @@ ## Token-Oriented Object Notation -**Version:** 4.1 +**Version:** 4.2 **Date:** 2026-07-26 @@ -138,8 +138,8 @@ All normative text is in Sections 1–16; the appendices and all examples are in - Indentation level (depth): Leading indentation measured in fixed-size space units (indentSize). Depth 0 has no indentation. - Indentation unit (indentSize): A fixed number of spaces per level (default 2). Tabs MUST NOT be used for indentation. -- Content depth: The depth at which a scope's immediate content appears – 0 for the root scope, otherwise one level deeper than the depth at which the scope's opening line stands (see §10 for first fields carried on a list-item hyphen line). -- Row depth, entry depth: The content depth of a tabular array's or keyed tabular object's scope, at which its rows or entry rows appear (§9.3, §9.5). +- Content depth: The depth at which a scope's immediate content appears – 0 for the root scope, otherwise one level deeper than the depth at which the scope's opening line stands (see §10 for first fields carried on a list-item hyphen line, and §8 for the non-strict depth-jump leniency). +- Row depth, entry depth, item depth: The content depth of a tabular array's, keyed tabular object's, or list-form array's scope, at which its rows, entry rows, or list items appear (§9.3, §9.4, §9.5). ### 1.4 Array and Tabular Terms @@ -206,7 +206,7 @@ Row, entry, and item terms: - If the fractional part is zero after normalization, emit as an integer (e.g., 1.0 → 1). - -0 MUST be normalized to 0. - For finite numbers outside the canonical range above (non-zero |n| < 1e-6, or |n| ≥ 1e21), encoders MAY emit exponent notation conforming to the JSON number grammar [RFC8259] §6 (e.g., 1e-7, 1e+21). Encoders SHOULD use lowercase `e` and an explicit exponent sign for byte-for-byte determinism. - - Encoders MUST emit sufficient precision so that, after any §3 host-type normalization, decode(encode(x)) equals x under the JSON-model equality defined below. + - Encoders MUST emit the fewest significant digits for which, after any §3 host-type normalization, decode(encode(x)) equals x under the JSON-model equality defined below; when several candidates qualify, they MUST emit the one closest to x, ties to even. - If a source value is outside the implementation's documented numeric domain (e.g., arbitrary-precision decimals or integers exceeding that domain), the encoder MAY: - Emit a quoted string containing a lossless decimal representation (plain decimal or JSON exponent form); the chosen form MUST be documented. - Emit a number that round-trips to the host's numeric approximation (losing precision), provided it conforms to the rules above. @@ -238,7 +238,7 @@ See Appendix E for non-normative language-specific examples. Decoders map text tokens to host values: -- Byte input: decoders that accept bytes MUST decode them as UTF-8. In strict mode, ill-formed UTF-8 (invalid or truncated sequences, or bytes encoding surrogate code points) MUST error; it MUST NOT be silently replaced with U+FFFD. Decoders that accept host strings (already decoded from bytes) are outside this rule. +- Byte input: decoders that accept bytes MUST decode them as UTF-8. In strict mode, ill-formed UTF-8 (invalid or truncated sequences, or bytes encoding surrogate code points) MUST error; it MUST NOT be silently replaced with U+FFFD. Non-strict decoders MAY instead replace each ill-formed sequence with U+FFFD. Decoders that accept host strings (already decoded from bytes) are outside this rule. - Quoted tokens (strings and keys): - MUST be unescaped per §7.1. Any other escape or an unterminated string MUST error. - Quoted primitives remain strings even if they look like numbers/booleans/null. @@ -276,8 +276,8 @@ TOON is a deterministic, line-oriented, indentation-based notation. - Root form discovery (applied to the comment-stripped line sequence, §5.1; line classes per §5.2): - If the first non-blank depth-0 line is a valid root array header per §6, decode a root array. - Else if the first non-blank depth-0 line is a valid keyless keyed header per §6 ([N:]{…}:), decode a root object in keyed tabular form (§9.5). - - Else if the first non-blank line is the literal token `[]`, decode an empty root array (§9.1). - - Else if the document has exactly one non-blank line and it is neither a valid array header nor a key-value line (quoted or unquoted key), decode a single primitive (examples: `hello`, `42`, `true`). + - Else if the first non-blank depth-0 line is the literal token `[]`, decode an empty root array (§9.1). + - Else if the document has exactly one non-blank line, at depth 0, and it is neither a valid array header nor a key-value line (quoted or unquoted key), decode a single primitive (examples: `hello`, `42`, `true`). - Otherwise, decode an object. - An empty document (no non-blank lines after comment removal, §5.1) decodes to an empty object `{}`. A document consisting only of comment and blank lines is therefore `{}`. - The root form spans the whole document: once a root array, an empty root array (`[]`), or a keyed tabular root object is complete, no further non-comment, non-blank line may follow. In strict mode, decoders MUST error on such trailing content (§14.2) – it MUST NOT be silently discarded. In non-strict mode, decoders MAY ignore it, except scalar lines, which are an error in any mode (§5.2). (A root object extends to the last line of the document, so this case does not arise for object roots.) @@ -300,10 +300,12 @@ Quoting keeps "#"-leading data out of the comment rule: string values that equal ### 5.2 Line Classification -Decoders classify each line of the comment-stripped sequence (§5.1) by its content after the leading indentation. The first matching class applies. Classification is lexical; whether a class is admissible at a given depth and position is determined by the enclosing construct (root form above, §8–§10). Within a tabular array's scope, lines at row depth are divided between the row and key-value classes by the disambiguation rules of §9.3, which are authoritative for that position and take precedence over the order below. Within a keyed tabular object's scope, every line at entry depth containing an unquoted colon is an entry row; §9.5 is likewise authoritative for that position. +Decoders classify each line of the comment-stripped sequence (§5.1) by its content after the leading indentation. The first matching class applies. Classification is lexical except for class 2's item-depth condition; whether a class is admissible at a given depth and position is determined by the enclosing construct (root form above, §8–§10). Within a tabular array's scope, lines at row depth are divided between the row and key-value classes by the disambiguation rules of §9.3, which are authoritative for that position and take precedence over the order below. Within a keyed tabular object's scope, every line at entry depth containing an unquoted colon is an entry row; §9.5 is likewise authoritative for that position. -1. Blank line – the content trims to empty. Handled per §12; blank lines never create or close structure. -2. List-item line – the content is the bare marker "-" or begins with "- " (hyphen, space). The remainder after the marker is parsed per §9.2, §9.4, and §10. Outside the scope of an array in list form, a leading hyphen has no structural meaning and the line is classified by the remaining classes. +Throughout this specification, a `"` opens a quoted span wherever it occurs in a line, and the next unescaped `"` closes it; without one, the span runs to the end of the line. A colon, delimiter, bracket, or brace inside a quoted span is quoted; any other is unquoted. + +1. Blank line – blank per §12, which governs its handling; blank lines never create or close structure. +2. List-item line – the content is the bare marker "-" or begins with "- " (hyphen, space). The remainder after the marker is parsed per §9.2, §9.4, and §10. A leading hyphen marks a list item only at the item depth of an array in list form; elsewhere – including a list-item object's field depth (§10) – the line is classified by the remaining classes. 3. Array-header line – the content matches the header or keyed-header grammar of §6. A line whose first unquoted colon precedes its first unquoted "[" is never a header; it is a key-value line. Only unquoted occurrences count: a quoted key containing a colon can still open a header (e.g., `"a:b"[2]: 1,2` is a header), while `a:b[2]: x` is a key-value line with key `a`. 4. Key-value line – the content contains an unquoted colon and no earlier class applies. The key token precedes the first unquoted colon and is decoded per §7.4; the remainder after the colon is the value (§8). A line that contains an unquoted colon but fails the §6 header grammar falls through to this class (e.g., `foo [2]: bar`); the strict-mode header errors enumerated in §6 and §14.2 are unaffected by this fall-through. 5. Row line – within a tabular array's scope, a delimiter-separated value line at row depth (§9.3); within a keyed tabular object's scope, an entry row at entry depth (§9.5). @@ -363,17 +365,17 @@ The ABNF does not express delimiter equality between the `bracket-seg` and `fiel Note: The grammar above specifies header syntax only. Tabular row disambiguation is defined in §9.3. -Whitespace MUST NOT appear between a key and its bracket segment, and content MUST NOT appear between `]` and `{`/`:` (e.g., `foo [2]:`, `[1][bar]:`, `[2]extra:`, `[2] :`). Such intervening whitespace or content, and the malformed bracket segments enumerated in the decoding requirements below, are strict-mode errors; non-strict decoders MAY parse the line as a key-value line, with the key treated as a literal token. +Whitespace MUST NOT appear between a key and its bracket segment or between a field name and its nested field group, and content MUST NOT appear between `]` and `{`/`:` (e.g., `foo [2]:`, `{b {c}}`, `[1][bar]:`, `[2]extra:`, `[2] :`). Such intervening whitespace or content, and the malformed bracket segments enumerated in the decoding requirements below, are strict-mode errors; non-strict decoders MAY parse the line as a key-value line, with the key treated as a literal token. Decoding requirements: -- The bracket segment MUST parse as a non-negative integer length N with no leading zeros (the single digit `0` is the only canonical form for length zero). Tokens like `[03]` or `[-1]` MUST NOT be interpreted as bracket segments. +- The bracket segment MUST parse as a non-negative integer length N with no leading zeros (the single digit `0` is the only canonical form for length zero). Tokens like `[03]` or `[-1]` MUST NOT be interpreted as bracket segments. N has no upper bound: a length the implementation cannot represent still forms a header, and its count is unmet (§14.1). - A bracket segment without a length token (`key[]:`) is not a header: strict mode MUST error; non-strict decoders MAY fall through to key-value parsing. This does not affect the empty-array value form `key: []` (§9.1), where `[]` follows the colon. - A colon immediately after the length and before the optional delimiter symbol marks a keyed header (§9.5): `[N:]` declares comma, `[N:]` tab, `[N:|]` pipe. The colon MUST occupy exactly that position – tokens such as `[2|:]`, `[2 :]`, or `[2:,]` are malformed bracket segments, and the length rules above apply unchanged (`[03:]` is malformed). - A keyed header MUST carry a field list: `key[2:]:` without braces is a header syntax error in strict mode; non-strict decoders MAY fall through to key-value parsing (§14.2). - If a trailing tab or pipe appears inside the brackets, it selects the active delimiter; otherwise comma is active. -- If a field list occurs between the bracket and the colon, parse field entries recursively using the active delimiter at every nesting level; quoted names MUST be unescaped per §7.1. Brace matching MUST ignore `{` and `}` inside quoted names. -- A field list MUST contain at least one field entry at every nesting level: an empty field list (`{}`, including a nested `field{}`) is a header syntax error in strict mode; non-strict decoders MAY fall through to key-value parsing (§14.2). Unmatched braces in a field list are likewise header syntax errors. -- A colon MUST follow the bracket segment and optional field list; missing colon MUST error. +- If a field list occurs between the bracket and the colon, parse field entries recursively using the active delimiter at every nesting level; quoted names MUST be unescaped per §7.1. Brace matching MUST ignore `{` and `}` inside quoted spans (§5.2). +- A field list MUST contain at least one field entry at every nesting level: an empty field list (`{}`, including a nested `field{}`), an empty field entry (`{a,}`), or a nameless nested group (`{a,{b}}`) is a header syntax error in strict mode; non-strict decoders MAY fall through to key-value parsing (§14.2). Unmatched braces in a field list are likewise header syntax errors, with the same non-strict fall-through. +- A line without any unquoted colon is neither a header nor a key context (§5.2): `items[2]` alone is a scalar line, and `- [2]` is a list item carrying the string `[2]`. - A non-keyed header without a field list: content after its colon is an inline primitive array (§9.1); nothing after the colon opens a block scope (§9.2, §9.4). A fields-bearing header – keyed or not – carries no inline content: in strict mode, non-whitespace content after its colon MUST error (§14.2); non-strict decoders MAY fall through to key-value parsing. - Keyless header positions: a keyless non-keyed header without a field list is valid only as the document's root header (§5) or as a list item after the `- ` marker (§9.2, §9.4); a keyless header with a field list – keyed or not – is valid only as the document's root header. In any other position, strict decoders MUST error (§14.2); non-strict decoders MAY parse the line as a key-value line, with the key treated as a literal token. @@ -392,9 +394,9 @@ In quoted strings and keys, codepoints are encoded according to the following ta | LF (U+000A) | MUST emit `\n` | MUST decode `\n` → LF | | CR (U+000D) | MUST emit `\r` | MUST decode `\r` → CR | | HTAB (U+0009) | MUST emit `\t` | MUST decode `\t` → HTAB | -| Other U+0000–U+001F controls | MUST emit `\uXXXX` (lowercase hex SHOULD) | MUST decode `\uXXXX` (case-insensitive hex) | +| Other U+0000–U+001F controls | MUST emit `\uXXXX` with lowercase hex | MUST decode `\uXXXX` (case-insensitive hex) | | U+D800–U+DFFF surrogates | (not produced by valid encoders) | MUST reject when decoded from `\uXXXX`, lone or paired | -| Other BMP codepoints (U+0020–U+D7FF, U+E000–U+FFFF) | SHOULD emit literal UTF-8; MAY emit `\uXXXX` | MUST accept either form | +| Other BMP codepoints (U+0020–U+D7FF, U+E000–U+FFFF) | MUST emit literal UTF-8 | MUST accept either form | | Supplementary scalar values (U+10000–U+10FFFF) | MUST emit as literal UTF-8 | MUST accept literal UTF-8; surrogate `\uXXXX` escapes MUST be rejected (see row above) | Decoders MUST reject any escape sequence not listed above, MUST reject `\u` followed by fewer than four hex digits, and MUST reject unterminated strings. @@ -404,12 +406,12 @@ Normative escape grammar: ```abnf ; Core rules per RFC 5234 §B.1 (DIGIT, DQUOTE, HEXDIG); HEXDIG matches hex digits case-insensitively quoted-char = escaped-char / unescaped-char -unescaped-char = %x09 / %x20-21 / %x23-5B / %x5D-D7FF / %xE000-FFFF / %x10000-10FFFF +unescaped-char = %x00-09 / %x0B-21 / %x23-5B / %x5D-D7FF / %xE000-FFFF / %x10000-10FFFF escaped-char = %x5C ( %x5C / DQUOTE / %x6E / %x72 / %x74 / unicode-escape ) unicode-escape = %x75 4HEXDIG ``` -Tabs are allowed inside quoted strings and as a declared delimiter; they MUST NOT be used for indentation (§12). Within quoted strings, encoders MUST emit HTAB as `\t` per the escape table above; the literal HTAB in `unescaped-char` expresses decoder leniency only. +Tabs are allowed inside quoted strings and as a declared delimiter; they MUST NOT be used for indentation (§12). Within quoted strings, decoders accept the literal controls in `unescaped-char` in any mode. ### 7.2 Quoting Rules for String Values @@ -421,17 +423,17 @@ Encoders MUST quote a string value if any of the following is true: - It contains a colon (:), double quote ("), or backslash (\\). - It contains brackets or braces ([, ], {, }). - It contains control characters in U+0000 through U+001F. -- It contains the relevant delimiter – the active delimiter for inline array values, tabular row cells, and keyed entry-row cells; the document delimiter for object field values (§11.1, which is authoritative for delimiter-aware quoting). +- It contains the relevant delimiter – the active delimiter for inline array values, tabular row cells, and keyed entry-row cells; the document delimiter for object field values, primitive list items, and root primitives (§11.1, which is authoritative for delimiter-aware quoting). - It equals "-" or starts with "-" (any hyphen at position 0). - It equals "#" or starts with "#" (any number sign at position 0). - It is a root primitive (§5) and starts with U+FEFF (§12). -Otherwise, the string MAY be emitted without quotes. Unicode, emoji, and strings with internal (non-leading/trailing) spaces are safe unquoted provided they do not violate the conditions. +Otherwise, encoders MUST emit the string without quotes. Unicode, emoji, and strings with internal (non-leading/trailing) spaces are safe unquoted provided they do not violate the conditions. ### 7.3 Key Encoding Object keys (including entry keys, §9.5) and the field names in a header's field list: -- MAY be unquoted only if they match: `^[A-Za-z_][A-Za-z0-9_.]*$`. +- MUST be unquoted if they match: `^[A-Za-z_][A-Za-z0-9_.]*$`. - Otherwise, they MUST be quoted and escaped per §7.1. Keys requiring quoting per the above rules MUST be quoted in all contexts, including array headers (e.g., "my-key"[N]:). @@ -442,7 +444,7 @@ Decoding of value tokens follows §4 (unquoted type inference, quoted strings, n - Quoted keys MUST be unescaped per §7.1; any other escape MUST error. - Keys (quoted or unquoted) MUST be followed by ":", optionally after spaces (§12); missing colon MUST error (see also §14.2). -- Unquoted key token (normative): an unquoted key token is the text before the first unquoted colon of a key-value line (§5.2) or entry row (§9.5), with surrounding spaces trimmed (§12); the text before a header's bracket segment; or a field name in a field list (§6). Decoders MUST accept any such token as a literal key, in strict and non-strict mode alike, even when it does not match §7.3's unquoted-key pattern: `foo-bar: 1`, `foo-bar[2]: 1,2`, and `items[1]{2key}:` are valid input. §7.3 constrains what encoders may emit unquoted, not what decoders accept. +- Unquoted key token (normative): an unquoted key token is the text before the first unquoted colon of a key-value line (§5.2) or entry row (§9.5), with surrounding spaces trimmed (§12); the text before a header's bracket segment; or a field name in a field list (§6). An empty token before a key-value or entry-row colon is the empty key: `: 1` decodes to `{"": 1}`. Decoders MUST accept any non-empty such token as a literal key, in strict and non-strict mode alike, even when it does not match §7.3's unquoted-key pattern: `foo-bar: 1`, `foo-bar[2]: 1,2`, and `items[1]{2key}:` are valid input. §7.3 governs how encoders emit keys, not what decoders accept. - Quoted-token boundary (normative): a token whose first character, after the trimming of §12, is `"` MUST be a complete quoted token – its closing `"` MUST be the token's last character. This applies wherever a token is extracted; any character after the closing quote MUST error. It overrides §4's "Otherwise → string" fallback. - Symmetrically for values: an unquoted value token that an encoder would have been required to quote (§7.2) is not an error. Decoders, strict mode included, MUST decode it per §4 – unless another rule of this specification assigns the token structural meaning (§5.2, §6, §9.1). Example: `key: -x` decodes to the string `-x`. §7.2 governs encoder output; it adds no decoder-side rejection. @@ -457,8 +459,8 @@ Decoding of value tokens follows §4 (unquoted type inference, quoted strings, n - Decoding: - Lines in an object body are classified per §5.2; the rules below cover its key-value class. - A line "key:" with nothing after the colon at depth d opens an object; subsequent lines at depth > d belong to that object until the depth decreases to ≤ d. - - In strict mode, the first line of a non-empty nested scope MUST be at exactly depth d+1; a depth increase of more than one level relative to the enclosing scope MUST error (§14.2). Conforming encoders never produce depth jumps; §10's depth model governs fields carried on a list-item hyphen line. - - A line deeper than the content depth of its enclosing scope whose preceding line did not open a scope belongs to no scope (e.g., a depth d+1 line directly under a depth-d primitive field). In strict mode, decoders MUST error (§14.2) – such lines MUST NOT be silently discarded. In non-strict mode, decoders MAY skip them, except scalar lines, which are an error in any mode (§5.2). + - In strict mode, the first line of a non-empty nested scope MUST be at exactly depth d+1; a depth increase of more than one level relative to the enclosing scope MUST error (§14.2). In non-strict mode, decoders MAY instead take the first line's depth as the scope's content depth; a later line deeper than d but shallower than that depth is then treated as over-indented (below). Conforming encoders never produce depth jumps; §10's depth model governs fields carried on a list-item hyphen line. + - A line deeper than the content depth of its enclosing scope whose preceding line did not open a scope belongs to no scope (e.g., a depth d+1 line directly under a depth-d primitive field, or the document's first line at depth 1 or more). In strict mode, decoders MUST error (§14.2) – such lines MUST NOT be silently discarded. In non-strict mode, decoders MAY skip them, except scalar lines, which are an error in any mode (§5.2). - A bare `key:` (no value after the colon) MUST decode as an empty or nested object, not an empty array. Empty arrays use the explicit `key: []` form (§9.1). - Lines "key: value" at the same depth are sibling fields. - Duplicate sibling keys at the same depth: see §14.3 for strict/non-strict behavior. @@ -540,7 +542,7 @@ When tabular requirements are not met (encoding; including any column that is ne - Object: formatted per §10 (objects as list items). Decoding: -- Header declares list length N and the active delimiter for any nested inline arrays. +- Header declares list length N; nested headers declare their own delimiter (§6). - Each list item is a list-item line (§5.2) starting with "- " at depth +1 (or the bare marker "-" for an empty object list item, §10) and is parsed as: - Primitive (no colon and no array header), - Inline primitive array (`- [M]: …`) or the empty-array item `- []` (§9.2), @@ -603,11 +605,11 @@ For an object appearing as a list item: ### 11.1 Encoding Rules -- Document delimiter: Encoders select a document delimiter (option: comma, tab, pipe; default comma). Encoders MUST declare it as the active delimiter of every header they emit (§6); it also governs delimiter-aware quoting for object field values (key: value) and root primitives. +- Document delimiter: Encoders select a document delimiter (option: comma, tab, pipe; default comma). Encoders MUST declare it as the active delimiter of every header they emit (§6); it also governs delimiter-aware quoting for object field values (key: value), primitive list items (`- value`), and root primitives. - Active delimiter: Inside a header's scope, the active delimiter governs quoting only for inline array values, tabular row cells, and keyed entry-row cells (§9.5). - Delimiter-aware quoting: - Inline array values, tabular row cells, and keyed entry-row cells: strings containing the active delimiter MUST be quoted. - - Object field values (key: value): encoders use the document delimiter to decide delimiter-aware quoting, regardless of whether the object appears within an array's scope. Entry rows (§9.5) are not object-field lines: the content after the entry key's colon is a delimiter-joined cell sequence, quoted per the active delimiter. + - Object field values (key: value) and primitive list items (`- value`): encoders use the document delimiter to decide delimiter-aware quoting, regardless of the enclosing array's scope. Entry rows (§9.5) are not object-field lines: the content after the entry key's colon is a delimiter-joined cell sequence, quoted per the active delimiter. - Strings containing non-active delimiters do not require quoting unless another condition applies (§7.2). ### 11.2 Decoding Rules @@ -635,12 +637,12 @@ For an object appearing as a list item: - Tabs used as indentation MUST error (see §7.1 for tabs in quoted strings and as the HTAB delimiter). - Non-strict mode: - Depth MAY be computed as floor(indentSpaces / indentSize). - - Implementations MAY accept tab characters in indentation. When they do, leading tabs are indentation and MUST be removed from the line's content before classification (§5.2). Depth computation for tabs is implementation-defined and MUST be documented. + - Implementations MAY accept tab characters in indentation. When they do, leading tabs are indentation and MUST be removed from the line's content before classification (§5.2), and a line of only tabs and spaces counts as blank. Depth computation for tabs is implementation-defined and MUST be documented. - Trailing spaces: trailing spaces (U+0020) at the end of a line are not part of the line's content. Decoders MUST strip them after the CR exclusion above and before line classification (§5.2); a line whose content is `-` followed only by spaces is therefore the bare marker for an empty-object list item (§9.4, §10), not a list item carrying an empty token. - - Token trimming: when a token is extracted – a key token before a key-value colon or an entry key's colon (§7.4, §9.5), a field entry in a field list (§6), or a value token after a key-value colon, after an array-header colon, or around each delimiter-separated token – decoders MUST trim surrounding spaces, exactly U+0020, no other characters. Any other whitespace (e.g., NBSP, or HTAB outside its delimiter role) is part of the token; internal semantics follow quoting rules. This trimming does not apply between a key and its bracket segment, where whitespace is a header syntax error (§6). + - Token trimming: when a token is extracted – a key token before a key-value colon or an entry key's colon (§7.4, §9.5), a field entry in a field list (§6), or a value token after a key-value colon, after an array-header colon, or around each delimiter-separated token – decoders MUST trim surrounding spaces, exactly U+0020, no other characters. Any other whitespace (e.g., NBSP, or HTAB outside its delimiter role) is part of the token; internal semantics follow quoting rules. This trimming does not apply between a key and its bracket segment or between a field name and its nested field group, where whitespace is a header syntax error (§6). - Comment lines are removed before any check in this section applies (§5.1). - Blank lines: - - A line whose content trims to empty is blank, regardless of leading-space count; the indentation checks above do not apply to blank lines. + - A line consisting of spaces only is blank, regardless of their count; the indentation checks above do not apply to blank lines. A line of tabs, or of spaces and tabs, is blank only under the non-strict tab leniency (above); otherwise its tabs are indentation. - Header span: the lines from the first item, row, or entry line in a header's scope through the last line of that scope's content (which may be a deeper line inside its last item). A blank line inside any header span: in strict mode, MUST error; in non-strict mode, MAY be ignored and not counted as a row/item/entry. - All other blank lines – including between a header and the scope's first item, row, or entry line, and after a scope's content: in strict mode, decoders MUST ignore them (they do not create or close structures and are not counted); in non-strict mode, decoders SHOULD ignore them. - Trailing newline at end-of-file: decoders SHOULD accept; validators MAY warn. @@ -661,7 +663,7 @@ Options: Strict-mode errors are enumerated in §14; validators MAY add informative diagnostics for style and encoding invariants. -Implementations SHOULD declare the specification version they target (e.g., `toon-spec: 4.1`) in their documentation. +Implementations SHOULD declare the specification version they target (e.g., `toon-spec: 4.2`) in their documentation. ### 13.1 Encoder Conformance Checklist @@ -669,8 +671,8 @@ Conforming encoders MUST: - [ ] Produce UTF-8 output with LF (U+000A) line endings (§1.2) - [ ] Use consistent indentation (default 2 spaces, no tabs) (§12) - [ ] Escape per §7.1 in quoted strings; never emit other escapes -- [ ] Quote strings per §7.2 (the relevant delimiter is governed by §11.1: document delimiter for object-field values, active delimiter for inline array values, tabular row cells, and keyed entry-row cells) -- [ ] Quote and escape object keys, entry keys, and field names that do not match §7.3's unquoted-key pattern (§7.3) +- [ ] Quote strings per §7.2, with the relevant delimiter per §11.1 +- [ ] Quote and escape exactly those object keys, entry keys, and field names that do not match §7.3's unquoted-key pattern (§7.3) - [ ] Select the form from the value's shape and position, not by preference (§1.4, §9) - [ ] Emit declared lengths [N] matching the actual inline value, list item, tabular row, or entry row count (§6, §9) - [ ] Preserve object key order as encountered, except where tabular forms reorder to the header's field order (§2) @@ -709,7 +711,7 @@ Validators SHOULD verify: ## 14. Strict Mode Errors and Diagnostics (Authoritative Checklist) -When strict mode is enabled (default), decoders MUST error on the conditions listed below. Conditions marked "(any mode)" are errors in strict and non-strict mode alike. Error type, code, and message text are implementation-defined. +When strict mode is enabled (default), decoders MUST error on the conditions listed below. Conditions marked "(any mode)" are errors in strict and non-strict mode alike. For every other condition, a non-strict decoder applies the leniency this specification names for it, or errors where that leniency is optional; it MUST NOT recover in any other way. Error type, code, and message text are implementation-defined. ### 14.1 Array Count and Width Mismatches @@ -727,14 +729,14 @@ When strict mode is enabled (default), decoders MUST error on the conditions lis - Invalid escape sequences or unterminated strings in quoted tokens, and characters after a quoted token's closing quote (any mode; §4, §7.1, §7.4). - Header delimiter mismatch (§6): MUST error as a header syntax error, independent of row width/count checks. - Malformed bracket lengths and malformed keyed markers in headers (§6). -- Malformed field lists in headers: an empty field list (`{}`, including a nested `field{}`), unmatched braces, or a field name repeated within the same field list (`{a,a}`, including inside a nested group); see §6, §9.3, §9.5. These are diagnosed from the header line alone, independent of the declared count and of any following rows or entry rows. +- Malformed field lists in headers: an empty field list (`{}`, including a nested `field{}`), an empty field entry (`{a,}`), a nameless nested group (`{a,{b}}`), unmatched braces, or a field name repeated within the same field list (`{a,a}`, including inside a nested group); see §6, §9.3, §9.5. These are diagnosed from the header line alone, independent of the declared count and of any following rows or entry rows. - Keyed headers (§9.5): a missing field list (`key[2:]:`), a keyless keyed header anywhere other than as the document's root header, or a line at entry depth without an unquoted colon. - Non-whitespace content after a fields-bearing header's colon (§6), keyed or not (e.g., `items[2]{a,b}: 1,2`). - Keyless headers outside their valid positions (§6): a keyless non-keyed header in object-field position (e.g., `[2]: x,y` under an object field, or as a non-first depth-0 line), or a keyless fields-bearing header as a list item (`- [2]{a}:`). -- Any whitespace between a key and its bracket segment, and any content between a valid bracket segment and the colon (or field list), prevents array-header interpretation; decoders MUST NOT silently discard that content. In non-strict mode, decoders MAY fall through to key-value parsing. +- Any whitespace between a key and its bracket segment or between a field name and its nested field group, and any content between a valid bracket segment and the colon (or field list), prevents array-header interpretation; decoders MUST NOT silently discard that content. In non-strict mode, decoders MAY fall through to key-value parsing. - Indentation and blank-line invariants per §12, evaluated after comment removal (§5.1). -- Indentation depth jumps (§8): a line more than one level deeper than its enclosing scope (e.g., a depth d+2 line directly under a depth-d parent). -- Over-indented lines (§8): a line deeper than the content depth of its enclosing scope when the preceding line did not open a scope (e.g., a depth d+1 line directly under a depth-d primitive field). Decoders MUST NOT silently discard such lines. +- Indentation depth jumps (§8): the first line of a nested scope standing deeper than the scope's content depth (§1.3; e.g., a depth d+2 line directly under a depth-d `key:`). +- Over-indented lines (§8): a line deeper than the content depth of its enclosing scope when the preceding line did not open a scope (e.g., a depth d+1 line directly under a depth-d primitive field, or the document's first line at depth 1 or more). Decoders MUST NOT silently discard such lines. - Trailing content after a completed root form (§5): any non-comment, non-blank line following the inline values, rows, items, or entries of a root array or keyed tabular root object, or following a root `[]`. - Ill-formed UTF-8 in byte input (§4). - A scalar line (§5.2) anywhere other than root primitive position – e.g., a bare token line inside an array or object scope (any mode; §5.2). @@ -952,11 +954,11 @@ These sketches illustrate structure and common decoding helpers. They are inform ### B.2 Array Header Parsing -- Identify the optional key prefix first (quoted: a `"…"` literal at line start; unquoted: the characters up to the first `[`, which contain no whitespace, §6). This sketch applies only to lines §5.2 has already classified as array-header lines. The bracket segment `[ … ]` begins at the first `[` after the key; parse: +- Identify the optional key prefix first (quoted: a `"…"` literal at line start; unquoted: the characters up to the first unquoted `[`, which contain no whitespace, §6). This sketch applies only to lines §5.2 has already classified as array-header lines. The bracket segment `[ … ]` begins at the first `[` after the key; parse: - Length N as decimal integer. - A colon immediately after the length marks a keyed header (§9.5); it requires a field list. Entry rows split at their first unquoted colon into entry key and cell sequence; the cells then split on the active delimiter. - Optional delimiter symbol at the end: HTAB or pipe (comma otherwise). -- If a "{ … }" field list occurs between the "]" and the ":", parse field entries recursively using the active delimiter: track brace depth, ignoring braces inside quoted names; a fieldname followed by "{" opens a nested field group. Unescape quoted names. The leaf-field list is the depth-first, pre-order walk of the resulting tree; rows assign cells to leaf fields in that order (§9.3). +- If a "{ … }" field list occurs between the "]" and the ":", parse field entries recursively using the active delimiter: track brace depth, ignoring braces inside quoted spans; a fieldname followed by "{" opens a nested field group. Unescape quoted names. The leaf-field list is the depth-first, pre-order walk of the resulting tree; rows assign cells to leaf fields in that order (§9.3). - Require a colon ":" after the bracket/field list. - Return the header (key?, length, delimiter, fields?) and any inline values after the colon. - Absence of a delimiter symbol in the bracket segment always means comma for that header (no inheritance). diff --git a/tests/fixtures/decode/arrays-nested.json b/tests/fixtures/decode/arrays-nested.json index 7a549e9..e8c6a95 100644 --- a/tests/fixtures/decode/arrays-nested.json +++ b/tests/fixtures/decode/arrays-nested.json @@ -84,32 +84,20 @@ "specSection": "10" }, { - "name": "parses objects containing arrays (including empty arrays) in list form", + "name": "parses a legacy [0] header as a later field of a list-item object", "input": "items[1]:\n - name: Ada\n data[0]:", "expected": { - "items": [ - { "name": "Ada", "data": [] } - ] + "items": [{ "name": "Ada", "data": [] }] }, - "specSection": "9.4" + "specSection": "10" }, { - "name": "parses arrays of arrays within objects", + "name": "parses a list-form array as the first field of a list-item object", "input": "items[1]:\n - matrix[2]:\n - [2]: 1,2\n - [2]: 3,4\n name: grid", "expected": { - "items": [ - { "matrix": [[1, 2], [3, 4]], "name": "grid" } - ] - }, - "specSection": "9.2" - }, - { - "name": "parses nested arrays of primitives", - "input": "pairs[2]:\n - [2]: a,b\n - [2]: c,d", - "expected": { - "pairs": [["a", "b"], ["c", "d"]] + "items": [{ "matrix": [[1, 2], [3, 4]], "name": "grid" }] }, - "specSection": "9.2" + "specSection": "10" }, { "name": "parses quoted strings and mixed lengths in nested arrays", @@ -127,14 +115,6 @@ }, "specSection": "9.2" }, - { - "name": "parses mixed-length inner arrays", - "input": "pairs[2]:\n - [1]: 1\n - [2]: 2,3", - "expected": { - "pairs": [[1], [2, 3]] - }, - "specSection": "9.2" - }, { "name": "parses root-level primitive array inline", "input": "[5]: x,y,\"true\",true,10", @@ -193,25 +173,6 @@ }, "specSection": "9.4" }, - { - "name": "parses arrays mixing objects and arrays", - "input": "items[2]:\n - a: 1\n - [2]: 1,2", - "expected": { - "items": [{ "a": 1 }, [1, 2]] - }, - "specSection": "9.4" - }, - { - "name": "parses quoted key with list-form array", - "input": "\"x-items\"[2]:\n - id: 1\n - id: 2", - "expected": { - "x-items": [ - { "id": 1 }, - { "id": 2 } - ] - }, - "specSection": "9.4" - }, { "name": "accepts bare bracket pair as empty inner array list item", "input": "items[2]:\n - []\n - [1]: x", @@ -316,6 +277,17 @@ "specSection": "14.1", "note": "A legacy [0] is a declared length, so its content is decoded (§14.1)", "minSpecVersion": "4.1" + }, + { + "name": "parses a list item shaped like a header without a colon as a string", + "input": "a[1]:\n - [2]", + "expected": { + "a": [ + "[2]" + ] + }, + "specSection": "6", + "minSpecVersion": "4.2" } ] } diff --git a/tests/fixtures/decode/arrays-primitive.json b/tests/fixtures/decode/arrays-primitive.json index f84fb8c..96978fe 100644 --- a/tests/fixtures/decode/arrays-primitive.json +++ b/tests/fixtures/decode/arrays-primitive.json @@ -3,22 +3,6 @@ "category": "decode", "description": "Primitive array decoding – inline arrays of strings, numbers, booleans, quoted strings", "tests": [ - { - "name": "parses string arrays inline", - "input": "tags[3]: reading,gaming,coding", - "expected": { - "tags": ["reading", "gaming", "coding"] - }, - "specSection": "9.1" - }, - { - "name": "parses number arrays inline", - "input": "nums[3]: 1,2,3", - "expected": { - "nums": [1, 2, 3] - }, - "specSection": "9.1" - }, { "name": "parses mixed primitive arrays inline", "input": "data[4]: x,y,true,10", @@ -99,14 +83,6 @@ }, "specSection": "9.1" }, - { - "name": "parses quoted key containing brackets with inline array", - "input": "\"key[test]\"[3]: 1,2,3", - "expected": { - "key[test]": [1, 2, 3] - }, - "specSection": "9.1" - }, { "name": "parses quoted key containing a colon with inline array", "input": "\"a:b\"[2]: 1,2", @@ -115,22 +91,6 @@ }, "specSection": "9.1" }, - { - "name": "parses quoted key with empty array", - "input": "\"x-custom\"[0]:", - "expected": { - "x-custom": [] - }, - "specSection": "9.1" - }, - { - "name": "parses quoted empty string key with empty array", - "input": "\"\"[0]:", - "expected": { - "": [] - }, - "specSection": "9.1" - }, { "name": "decodes canonical empty array key: []", "input": "items: []", @@ -139,22 +99,6 @@ }, "specSection": "9.1" }, - { - "name": "decodes canonical empty array with quoted key", - "input": "\"x-custom\": []", - "expected": { - "x-custom": [] - }, - "specSection": "9.1" - }, - { - "name": "decodes canonical empty array with empty-string key", - "input": "\"\": []", - "expected": { - "": [] - }, - "specSection": "9.1" - }, { "name": "parses quoted key containing escaped quotes and brackets with inline array", "input": "\"has \\\"[2]\\\" in it\"[3]: 3,4,5", @@ -208,6 +152,20 @@ ] }, "specSection": "9.3" + }, + { + "name": "keeps a header whose length exceeds integer range in non-strict mode", + "input": "a[99999999999999999999]: 1", + "expected": { + "a": [ + 1 + ] + }, + "options": { + "strict": false + }, + "specSection": "6", + "minSpecVersion": "4.2" } ] } diff --git a/tests/fixtures/decode/arrays-tabular.json b/tests/fixtures/decode/arrays-tabular.json index 269bffa..3730f6c 100644 --- a/tests/fixtures/decode/arrays-tabular.json +++ b/tests/fixtures/decode/arrays-tabular.json @@ -25,17 +25,6 @@ }, "specSection": "9.3" }, - { - "name": "parses quoted colon in tabular row as data", - "input": "items[2]{id,note}:\n 1,\"a:b\"\n 2,\"c:d\"", - "expected": { - "items": [ - { "id": 1, "note": "a:b" }, - { "id": 2, "note": "c:d" } - ] - }, - "specSection": "9.3" - }, { "name": "parses quoted header keys in tabular arrays", "input": "items[2]{\"order:id\",\"full name\"}:\n 1,Ada\n 2,Bob", @@ -290,6 +279,41 @@ ] }, "specSection": "9.3" + }, + { + "name": "ignores a brace inside a mid-name quote span in a field list", + "input": "a[1]{x\"}\"y}:\n 1", + "expected": { + "a": [ + { + "x\"}\"y": 1 + } + ] + }, + "specSection": "6", + "minSpecVersion": "4.2" + }, + { + "name": "parses a quoted empty field name", + "input": "[2]{\"\"}:\n 1\n 2", + "expected": [ + { + "": 1 + }, + { + "": 2 + } + ], + "specSection": "6", + "minSpecVersion": "4.2" + }, + { + "name": "unescapes a quoted field name", + "input": "a[1]{\"x\\\"y\"}:\n 1", + "expected": { + "a": [{ "x\"y": 1 }] + }, + "specSection": "6" } ] } diff --git a/tests/fixtures/decode/blank-lines.json b/tests/fixtures/decode/blank-lines.json index 74e04cf..c456cca 100644 --- a/tests/fixtures/decode/blank-lines.json +++ b/tests/fixtures/decode/blank-lines.json @@ -5,7 +5,7 @@ "tests": [ { "name": "throws on blank line inside list array", - "input": "items[3]:\n - a\n\n - b\n - c", + "input": "items[2]:\n - a\n\n - b", "expected": null, "shouldError": true, "options": { @@ -55,7 +55,7 @@ }, { "name": "throws on blank line in nested list array", - "input": "outer[2]:\n - inner[2]:\n - a\n\n - b\n - x", + "input": "outer[2]:\n - inner[2]:\n - a\n\n - b\n - x", "expected": null, "shouldError": true, "options": { @@ -76,7 +76,7 @@ "specSection": "12" }, { - "name": "accepts whitespace-only line at non-multiple indent as blank in strict mode", + "name": "accepts a spaces-only line at non-multiple indent as blank in strict mode", "input": "a: 1\n \nb: 2", "expected": { "a": 1, @@ -137,9 +137,9 @@ }, { "name": "ignores blank lines inside list array when strict=false", - "input": "items[3]:\n - a\n\n - b\n - c", + "input": "items[2]:\n - a\n\n - b", "expected": { - "items": ["a", "b", "c"] + "items": ["a", "b"] }, "options": { "strict": false @@ -179,13 +179,6 @@ "specSection": "14.2", "note": "The blank is after the inner scope's last row but inside the outer array's span (\u00a712)" }, - { - "name": "throws on blank line between a list item's fields", - "input": "items[2]:\n - a: 1\n\n b: 2\n - x", - "expected": null, - "shouldError": true, - "specSection": "14.2" - }, { "name": "throws on blank line inside the last list item's fields", "input": "items[1]:\n - a: 1\n\n b: 2", @@ -227,6 +220,27 @@ "shouldError": true, "specSection": "14.2", "note": "The blank is after the nested object's content but inside the outer array's span (§12)" + }, + { + "name": "throws on a tab-only line, which is not blank", + "input": "a: 1\n\t\nb: 2", + "expected": null, + "shouldError": true, + "specSection": "14.2", + "minSpecVersion": "4.2" + }, + { + "name": "treats a tab-only line as blank in non-strict mode", + "input": "a: 1\n\t\nb: 2", + "expected": { + "a": 1, + "b": 2 + }, + "options": { + "strict": false + }, + "specSection": "12", + "minSpecVersion": "4.2" } ] } diff --git a/tests/fixtures/decode/comments.json b/tests/fixtures/decode/comments.json index e351082..77ceb77 100644 --- a/tests/fixtures/decode/comments.json +++ b/tests/fixtures/decode/comments.json @@ -88,12 +88,6 @@ "specSection": "5.1", "note": "Comment text is discarded without interpretation; the quote never reaches the tokenizer" }, - { - "name": "decodes document of only comments to empty object", - "input": "# only\n# comments", - "expected": {}, - "specSection": "5.1" - }, { "name": "decodes comments and blank lines only to empty object", "input": "\n# c\n\n# d\n", @@ -112,24 +106,20 @@ "input": "items[2]{id}:\n #1\n 2", "expected": null, "shouldError": true, - "specSection": "14.1", + "specSection": "5.1", "note": "The first cell makes the whole line a comment; only one row remains for [2]" }, { - "name": "drops hash-leading row silently in non-strict mode", + "name": "strips a hash-leading row as a comment in non-strict mode", "input": "items[2]{id}:\n #1\n 2", "expected": { - "items": [ - { - "id": 2 - } - ] + "items": [{ "id": 2 }] }, "options": { "strict": false }, "specSection": "5.1", - "note": "A hash-leading first cell makes the whole line a comment; non-strict mode drops the row silently" + "note": "Comment removal applies in both modes; the resulting count mismatch is not an error when strict=false (§14.1)" }, { "name": "throws on tab-indented hash line, which is not a comment", @@ -200,18 +190,6 @@ "msg": "hello # world" }, "specSection": "5.1" - }, - { - "name": "round-trips quoted hash-leading values in field and inline array positions", - "input": "tag: \"#x\"\nlist[2]: \"#a\",b", - "expected": { - "tag": "#x", - "list": [ - "#a", - "b" - ] - }, - "specSection": "7.2" } ] } diff --git a/tests/fixtures/decode/delimiters.json b/tests/fixtures/decode/delimiters.json index d12802a..45b01d7 100644 --- a/tests/fixtures/decode/delimiters.json +++ b/tests/fixtures/decode/delimiters.json @@ -58,7 +58,7 @@ "specSection": "11" }, { - "name": "parses nested arrays inside list items with default comma delimiter", + "name": "parses nested arrays inside list items with default comma delimiter when parent uses tab", "input": "items[1\t]:\n - tags[3]: a,b,c", "expected": { "items": [{ "tags": ["a", "b", "c"] }] @@ -66,10 +66,10 @@ "specSection": "11" }, { - "name": "parses nested arrays inside list items with default comma delimiter when parent uses pipe", - "input": "items[1|]:\n - tags[3]: a,b,c", + "name": "parses a keyless inner array without a delimiter symbol as comma when parent uses pipe", + "input": "pairs[1|]:\n - [2]: a,b", "expected": { - "items": [{ "tags": ["a", "b", "c"] }] + "pairs": [["a", "b"]] }, "specSection": "11" }, @@ -87,19 +87,19 @@ }, { "name": "parses root-level array of objects with tab delimiter", - "input": "[2\t]{id}:\n 1\n 2", - "expected": [{ "id": 1 }, { "id": 2 }], + "input": "[1\t]{a\tb}:\n 1\t2", + "expected": [{ "a": 1, "b": 2 }], "specSection": "11" }, { "name": "parses root-level array of objects with pipe delimiter", - "input": "[2|]{id}:\n 1\n 2", - "expected": [{ "id": 1 }, { "id": 2 }], + "input": "[1|]{a|b}:\n 1|2", + "expected": [{ "a": 1, "b": 2 }], "specSection": "11" }, { "name": "parses values containing tab delimiter when quoted", - "input": "items[3\t]: a\t\"b\\tc\"\td", + "input": "items[3\t]: a\t\"b\tc\"\td", "expected": { "items": ["a", "b\tc", "d"] }, @@ -161,10 +161,10 @@ "note": "Object field values are parsed as the whole post-colon token, never split on a delimiter" }, { - "name": "does not split object values on commas inside a tab-delimited scope", - "input": "items[2\t]:\n - status: a,b\n - status: c,d", + "name": "does not split a list-item field value on the active delimiter", + "input": "items[1|]:\n - status: a|b", "expected": { - "items": [{ "status": "a,b" }, { "status": "c,d" }] + "items": [{ "status": "a|b" }] }, "specSection": "11.2", "note": "The active delimiter splits row and inline cells, never object field values" @@ -177,54 +177,6 @@ }, "specSection": "11" }, - { - "name": "parses nested array values containing pipe delimiter", - "input": "pairs[1|]:\n - [2|]: a|\"b|c\"", - "expected": { - "pairs": [["a", "b|c"]] - }, - "specSection": "11" - }, - { - "name": "parses nested array values containing tab delimiter", - "input": "pairs[1\t]:\n - [2\t]: a\t\"b\\tc\"", - "expected": { - "pairs": [["a", "b\tc"]] - }, - "specSection": "11" - }, - { - "name": "preserves quoted ambiguity with pipe delimiter", - "input": "items[3|]: \"true\"|\"42\"|\"-3.14\"", - "expected": { - "items": ["true", "42", "-3.14"] - }, - "specSection": "11" - }, - { - "name": "preserves quoted ambiguity with tab delimiter", - "input": "items[3\t]: \"true\"\t\"42\"\t\"-3.14\"", - "expected": { - "items": ["true", "42", "-3.14"] - }, - "specSection": "11" - }, - { - "name": "parses structural-looking strings when quoted with pipe delimiter", - "input": "items[3|]: \"[5]\"|\"{key}\"|\"- item\"", - "expected": { - "items": ["[5]", "{key}", "- item"] - }, - "specSection": "11" - }, - { - "name": "parses structural-looking strings when quoted with tab delimiter", - "input": "items[3\t]: \"[5]\"\t\"{key}\"\t\"- item\"", - "expected": { - "items": ["[5]", "{key}", "- item"] - }, - "specSection": "11" - }, { "name": "parses tabular headers with keys containing the active delimiter", "input": "items[2|]{\"a|b\"}:\n 1\n 2", @@ -235,15 +187,35 @@ }, { "name": "falls through to a key-value line on a header delimiter mismatch", - "input": "a[2|]{x,y}: 1|2", + "input": "a[0|]{x,y}:", "expected": { - "a[2|]{x,y}": "1|2" + "a[0|]{x,y}": {} }, "options": { "strict": false }, "specSection": "6", "minSpecVersion": "4.1" + }, + { + "name": "treats a delimiter inside a mid-token quote span as quoted", + "input": "k[2]: a\"b,c\",d", + "expected": { + "k": [ + "a\"b,c\"", + "d" + ] + }, + "specSection": "5.2", + "minSpecVersion": "4.2" + }, + { + "name": "parses a nested tabular header with its own comma delimiter inside a pipe list", + "input": "items[1|]:\n - t[1]{a,b}:\n 1,2", + "expected": { + "items": [{ "t": [{ "a": 1, "b": 2 }] }] + }, + "specSection": "11.2" } ] } diff --git a/tests/fixtures/decode/indentation-errors.json b/tests/fixtures/decode/indentation-errors.json index 2c32119..bf42b5c 100644 --- a/tests/fixtures/decode/indentation-errors.json +++ b/tests/fixtures/decode/indentation-errors.json @@ -26,8 +26,8 @@ "specSection": "14.2" }, { - "name": "throws on non-multiple indentation with custom indentSize 4 (3 spaces)", - "input": "a:\n b: 1", + "name": "throws on 2-space indentation with custom indentSize 4", + "input": "a:\n b: 1", "expected": null, "shouldError": true, "options": { @@ -115,11 +115,9 @@ }, { "name": "accepts non-multiple indentation when strict=false", - "input": "a:\n b: 1", + "input": "a:\n b: 1\n c: 2", "expected": { - "a": { - "b": 1 - } + "a": { "b": 1, "c": 2 } }, "options": { "indentSize": 2, @@ -148,7 +146,7 @@ "input": "a:\n b: 1", "expected": null, "shouldError": true, - "specSection": "14.2", + "specSection": "8", "note": "Four leading spaces are a multiple of indentSize 2, but depth 2 directly under depth 0 skips a level" }, { @@ -156,22 +154,21 @@ "input": "a:\n b:\n c: 1", "expected": null, "shouldError": true, - "specSection": "14.2" + "specSection": "8" }, { - "name": "throws on over-indented line after a primitive field", - "input": "a: 1\n b: 2", + "name": "throws on an over-indented line under a primitive list item", + "input": "items[2]:\n - a\n b: 1\n - c", "expected": null, "shouldError": true, - "specSection": "14.2", - "note": "A primitive field opens no scope, so the depth-2 line belongs to none and must not be silently discarded" + "specSection": "8" }, { "name": "throws on a line indented one level under a primitive field", "input": "a: 1\n b: 2", "expected": null, "shouldError": true, - "specSection": "14.2", + "specSection": "8", "note": "Even a single extra level is over-indentation when the preceding line did not open a scope" }, { @@ -179,7 +176,7 @@ "input": "a:\n b: 1\n c: 2", "expected": null, "shouldError": true, - "specSection": "14.2", + "specSection": "8", "note": "The depth-3 line follows a primitive field at depth 1; it belongs to no scope" }, { @@ -187,27 +184,127 @@ "input": "u[1]{a}:\n 1\n junk: 9", "expected": null, "shouldError": true, - "specSection": "14.2" + "specSection": "8" }, { - "name": "throws on orphan scalar line under a primitive field", + "name": "throws on orphan scalar line under a primitive field in non-strict mode", "input": "a: 1\n hello", "expected": null, "shouldError": true, - "specSection": "14.2", - "note": "A scalar line is valid only at root primitive position" + "options": { + "strict": false + }, + "specSection": "8", + "note": "The non-strict skip for over-indented lines excludes scalar lines (§5.2)", + "minSpecVersion": "4.1" }, { - "name": "throws on orphan scalar line under a primitive field in non-strict mode", - "input": "a: 1\n hello", + "name": "throws on an indented first line", + "input": " a: 1", + "expected": null, + "shouldError": true, + "specSection": "8", + "minSpecVersion": "4.2" + }, + { + "name": "throws on a hyphen-leading bare token under a primitive field in non-strict mode", + "input": "a: 1\n - x", "expected": null, "shouldError": true, "options": { "strict": false }, + "specSection": "5.2", + "minSpecVersion": "4.2" + }, + { + "name": "adopts the depth of a jumped first line in non-strict mode", + "input": "a:\n b: 1\n c: 2\n d: 3", + "expected": { + "a": { + "b": 1, + "d": 3 + } + }, + "options": { + "strict": false + }, "specSection": "8", - "note": "The non-strict skip for over-indented lines excludes scalar lines (§5.2)", - "minSpecVersion": "4.1" + "minSpecVersion": "4.2" + }, + { + "name": "skips a line above the adopted entry depth in non-strict mode", + "input": "m[1:]{a}:\n x: 1\n y: 2", + "expected": { + "m": { + "x": { + "a": 1 + } + } + }, + "options": { + "strict": false + }, + "specSection": "8", + "minSpecVersion": "4.2" + }, + { + "name": "skips a line above the adopted row depth in non-strict mode", + "input": "rows[2]{a}:\n 1\n x: 9\n 2", + "expected": { + "rows": [ + { + "a": 1 + }, + { + "a": 2 + } + ] + }, + "options": { + "strict": false + }, + "specSection": "8", + "minSpecVersion": "4.2" + }, + { + "name": "skips an over-indented line under a primitive list item in non-strict mode", + "input": "items[2]:\n - a\n b: 1\n - c", + "expected": { + "items": [ + "a", + "c" + ] + }, + "options": { + "strict": false + }, + "specSection": "8", + "minSpecVersion": "4.2" + }, + { + "name": "skips an over-indented line under a list-item field in non-strict mode", + "input": "items[1]:\n - a: 1\n deep: 1\n b: 2", + "expected": { + "items": [ + { + "a": 1, + "b": 2 + } + ] + }, + "options": { + "strict": false + }, + "specSection": "8", + "minSpecVersion": "4.2" + }, + { + "name": "throws on a depth jump to the first tabular row", + "input": "rows[1]{a}:\n 1", + "expected": null, + "shouldError": true, + "specSection": "8" } ] } diff --git a/tests/fixtures/decode/numbers.json b/tests/fixtures/decode/numbers.json index 22ac3f1..df749d9 100644 --- a/tests/fixtures/decode/numbers.json +++ b/tests/fixtures/decode/numbers.json @@ -42,7 +42,7 @@ "value": 0 }, "specSection": "4", - "note": "Negative zero decodes to 0; most host environments do not distinguish -0 from 0" + "note": "Most host environments do not distinguish -0 from 0" }, { "name": "parses negative zero with fractional part", @@ -58,8 +58,7 @@ "expected": { "nums": [42, -1000, 1.5, 0, 250] }, - "specSection": "4", - "note": "Decoders normalize all numeric forms to host numeric values" + "specSection": "4" }, { "name": "treats leading zero as string not number", @@ -69,22 +68,6 @@ }, "specSection": "4" }, - { - "name": "parses very small exponent", - "input": "value: 1e-10", - "expected": { - "value": 0.0000000001 - }, - "specSection": "4" - }, - { - "name": "parses integer with positive exponent", - "input": "value: 5E+00", - "expected": { - "value": 5 - }, - "specSection": "4" - }, { "name": "parses zero with exponent as number", "input": "value: 0e1", @@ -93,50 +76,18 @@ }, "specSection": "4" }, - { - "name": "parses negative zero with exponent as number", - "input": "value: -0e1", - "expected": { - "value": 0 - }, - "specSection": "4" - }, { "name": "parses exponent notation", "input": "1e6", "expected": 1000000, "specSection": "4" }, - { - "name": "parses exponent notation with uppercase E", - "input": "1E+6", - "expected": 1000000, - "specSection": "4" - }, - { - "name": "parses negative exponent notation", - "input": "-1e-3", - "expected": -0.001, - "specSection": "4" - }, { "name": "treats unquoted leading-zero number as string", "input": "05", "expected": "05", "specSection": "4" }, - { - "name": "treats unquoted multi-leading-zero as string", - "input": "007", - "expected": "007", - "specSection": "4" - }, - { - "name": "treats unquoted octal-like as string", - "input": "0123", - "expected": "0123", - "specSection": "4" - }, { "name": "treats leading-zeros in array as strings", "input": "nums[3]: 05,007,0123", @@ -149,14 +100,6 @@ "expected": "-05", "specSection": "4" }, - { - "name": "treats negative leading-zeros in array as strings", - "input": "nums[2]: -05,-007", - "expected": { - "nums": ["-05", "-007"] - }, - "specSection": "4" - }, { "name": "treats bare fraction without integer part as string", "input": "value: .5", @@ -192,15 +135,6 @@ "specSection": "4", "note": "The grammar admits only an optional leading minus, so leading-plus cells stay strings" }, - { - "name": "parses positive exponent sign as number", - "input": "value: 1e+2", - "expected": { - "value": 100 - }, - "specSection": "4", - "note": "The plus sign is inside the exponent, which the grammar allows" - }, { "name": "treats Infinity and NaN tokens as strings", "input": "a: Infinity\nb: -Infinity\nc: NaN", @@ -220,6 +154,14 @@ }, "specSection": "4", "note": "Host parsers that accept these forms have a wider grammar than the spec permits" + }, + { + "name": "treats non-ASCII digits as a string", + "input": "value: ٤٢", + "expected": { + "value": "٤٢" + }, + "specSection": "4" } ] } diff --git a/tests/fixtures/decode/objects-keyed.json b/tests/fixtures/decode/objects-keyed.json index 3e2e05d..1819405 100644 --- a/tests/fixtures/decode/objects-keyed.json +++ b/tests/fixtures/decode/objects-keyed.json @@ -93,16 +93,13 @@ "note": "A keyed scope ends only by dedent or end of input" }, { - "name": "treats key-value-shaped lines at entry depth as entry rows", - "input": "m[2:]{x}:\n total: 5\n other: 6", + "name": "splits an entry row at its first unquoted colon when a delimiter precedes it", + "input": "m[1:]{x}:\n a,b: 5", "expected": { - "m": { - "total": { "x": 5 }, - "other": { "x": 6 } - } + "m": { "a,b": { "x": 5 } } }, "specSection": "9.5", - "note": "Opposite polarity to §9.3: in a tabular array, a colon-before-delimiter line at row depth ends the rows; in a keyed scope it is an entry" + "note": "The colon split comes before the delimiter split, so the delimiter before the colon is part of the entry key" }, { "name": "splits an entry row at its first unquoted colon before delimiter splitting", @@ -136,16 +133,6 @@ "specSection": "9.5", "note": "Encoders never emit keyed headers for fewer than two entries; decoders accept any N with count checks" }, - { - "name": "accepts a single entry row", - "input": "m[1:]{v}:\n a: 1", - "expected": { - "m": { - "a": { "v": 1 } - } - }, - "specSection": "9.5" - }, { "name": "parses quoted cells containing the active delimiter", "input": "m[2:]{t}:\n a: \"x,y\"\n b: z", @@ -225,6 +212,18 @@ }, "specSection": "9.5", "minSpecVersion": "4.1" + }, + { + "name": "decodes every entry row beyond the declared count in non-strict mode", + "input": "m[1:]{v}:\n a: 1\n b: 2", + "expected": { + "m": { "a": { "v": 1 }, "b": { "v": 2 } } + }, + "options": { + "strict": false + }, + "specSection": "14.1", + "minSpecVersion": "4.1" } ] } diff --git a/tests/fixtures/decode/objects.json b/tests/fixtures/decode/objects.json index a53710c..5f00452 100644 --- a/tests/fixtures/decode/objects.json +++ b/tests/fixtures/decode/objects.json @@ -15,15 +15,14 @@ }, { "name": "parses null values in objects", - "input": "id: 123\nvalue: null", + "input": "value: null", "expected": { - "id": 123, "value": null }, "specSection": "8" }, { - "name": "parses empty nested object header", + "name": "decodes a bare key: as an empty object, not an empty array", "input": "user:", "expected": { "user": {} @@ -31,12 +30,13 @@ "specSection": "8" }, { - "name": "bare key: with no children decodes as empty object, not array", - "input": "matches:", + "name": "decodes a bare key followed by trailing spaces as an empty object", + "input": "matches: ", "expected": { "matches": {} }, - "specSection": "8" + "specSection": "12", + "minSpecVersion": "4.1" }, { "name": "applies last-write-wins for duplicate sibling keys in non-strict mode", @@ -55,7 +55,7 @@ "expected": { "note": "a:b" }, - "specSection": "8" + "specSection": "4" }, { "name": "parses quoted object value with comma", @@ -63,7 +63,7 @@ "expected": { "note": "a,b" }, - "specSection": "8" + "specSection": "4" }, { "name": "parses quoted object value with newline escape", @@ -87,15 +87,7 @@ "expected": { "text": " padded " }, - "specSection": "8" - }, - { - "name": "parses quoted object value with only spaces", - "input": "text: \" \"", - "expected": { - "text": " " - }, - "specSection": "8" + "specSection": "4" }, { "name": "parses quoted string value that looks like true", @@ -103,7 +95,7 @@ "expected": { "v": "true" }, - "specSection": "8" + "specSection": "4" }, { "name": "parses quoted string value that looks like integer", @@ -111,24 +103,15 @@ "expected": { "v": "42" }, - "specSection": "8" - }, - { - "name": "parses quoted string value that looks like negative decimal", - "input": "v: \"-7.5\"", - "expected": { - "v": "-7.5" - }, - "specSection": "8" + "specSection": "4" }, { - "name": "parses quoted object value shaped like an inline array header", - "input": "a: \"[2]: x\"", + "name": "parses an unquoted field value starting with a bracket segment as a string", + "input": "a: [2]: x", "expected": { "a": "[2]: x" }, - "specSection": "8", - "note": "Quoted value is opaque (§4); the bracket-and-colon shape inside the quotes is data, not an array header (§6)" + "specSection": "5.2" }, { "name": "parses quoted object value shaped like a count-matching inline array header", @@ -136,7 +119,7 @@ "expected": { "a": "[1]: x" }, - "specSection": "8", + "specSection": "4", "note": "Quoting makes the value opaque (§4), so it is never parsed as a header – the matching count is irrelevant because a quoted scalar is never a header candidate" }, { @@ -145,7 +128,7 @@ "expected": { "key": "foo [2]: bar" }, - "specSection": "6", + "specSection": "5.2", "note": "An unquoted key cannot contain a colon, so the first colon ends the key; the bracket-and-colon shape in the remainder is value text, not an array header" }, { @@ -154,7 +137,7 @@ "expected": { "a": "b [2]{x,y}: 1,2" }, - "specSection": "6", + "specSection": "5.2", "note": "An unquoted key cannot contain a colon, so the first colon ends the key; the tabular-header shape in the remainder is value text, not an array header" }, { @@ -163,7 +146,7 @@ "expected": { "content": "accept (process.argv[2]) and greet: name" }, - "specSection": "8" + "specSection": "5.2" }, { "name": "parses quoted key with colon", @@ -171,7 +154,7 @@ "expected": { "order:id": 7 }, - "specSection": "8" + "specSection": "7.4" }, { "name": "decodes \\uXXXX in quoted key (U+0004 control character)", @@ -195,7 +178,7 @@ "expected": { "[index]": 5 }, - "specSection": "8" + "specSection": "7.4" }, { "name": "treats extra brackets after valid array segment as literal key (non-strict)", @@ -251,7 +234,7 @@ "expected": { "{key}": 5 }, - "specSection": "8" + "specSection": "7.4" }, { "name": "parses quoted key with comma", @@ -259,7 +242,7 @@ "expected": { "a,b": 1 }, - "specSection": "8" + "specSection": "7.4" }, { "name": "parses quoted key with spaces", @@ -267,7 +250,7 @@ "expected": { "full name": "Ada" }, - "specSection": "8" + "specSection": "7.4" }, { "name": "parses quoted key with leading hyphen", @@ -275,7 +258,7 @@ "expected": { "-lead": 1 }, - "specSection": "8" + "specSection": "7.4" }, { "name": "parses quoted key with leading and trailing spaces", @@ -283,7 +266,7 @@ "expected": { " a ": 1 }, - "specSection": "8" + "specSection": "7.4" }, { "name": "parses quoted numeric key", @@ -291,7 +274,7 @@ "expected": { "123": "x" }, - "specSection": "8" + "specSection": "7.4" }, { "name": "parses quoted empty string key", @@ -299,7 +282,7 @@ "expected": { "": 1 }, - "specSection": "8" + "specSection": "7.4" }, { "name": "parses dotted keys as identifiers", @@ -309,22 +292,6 @@ }, "specSection": "8" }, - { - "name": "parses underscore-prefixed keys", - "input": "_private: 1", - "expected": { - "_private": 1 - }, - "specSection": "8" - }, - { - "name": "parses underscore-containing keys", - "input": "user_name: 1", - "expected": { - "user_name": 1 - }, - "specSection": "8" - }, { "name": "unescapes newline in key", "input": "\"line\\nbreak\": 1", @@ -333,14 +300,6 @@ }, "specSection": "7.1" }, - { - "name": "unescapes tab in key", - "input": "\"tab\\there\": 2", - "expected": { - "tab\there": 2 - }, - "specSection": "7.1" - }, { "name": "unescapes tab in key immediately followed by a colon", "input": "\"\\t:x\": v", @@ -487,7 +446,7 @@ "specSection": "15" }, { - "name": "falls through to a key-value line when whitespace precedes the bracket segment", + "name": "falls through to a key-value line when whitespace precedes the bracket segment (non-strict)", "input": "foo [2]: bar,baz", "expected": { "foo [2]": "bar,baz" @@ -585,6 +544,47 @@ }, "specSection": "16", "minSpecVersion": "4.1" + }, + { + "name": "parses an empty unquoted key as the empty key", + "input": ": 1", + "expected": { + "": 1 + }, + "specSection": "7.4", + "minSpecVersion": "4.2" + }, + { + "name": "parses a hyphen-leading line at list-item field depth as a key-value line", + "input": "items[1]:\n - a: 1\n - y: 2", + "expected": { + "items": [ + { + "a": 1, + "- y": 2 + } + ] + }, + "specSection": "5.2", + "minSpecVersion": "4.2" + }, + { + "name": "parses a key whose closed mid-token quote span hides a colon", + "input": "a\"b:c\"d: 1", + "expected": { + "a\"b:c\"d": 1 + }, + "specSection": "5.2", + "minSpecVersion": "4.2" + }, + { + "name": "reads a line whose only unquoted colon sits in the bracket segment as a key-value line", + "input": "a[2:]{x}", + "expected": { + "a[2": "]{x}" + }, + "specSection": "5.2", + "minSpecVersion": "4.2" } ] } diff --git a/tests/fixtures/decode/primitives.json b/tests/fixtures/decode/primitives.json index 2347930..14441d4 100644 --- a/tests/fixtures/decode/primitives.json +++ b/tests/fixtures/decode/primitives.json @@ -9,17 +9,11 @@ "expected": "hello", "specSection": "7.4" }, - { - "name": "parses unquoted string with underscore and numbers", - "input": "Ada_99", - "expected": "Ada_99", - "specSection": "7.4" - }, { "name": "parses empty quoted string", "input": "\"\"", "expected": "", - "specSection": "7.4" + "specSection": "4" }, { "name": "parses quoted string with newline escape", @@ -65,27 +59,21 @@ }, { "name": "parses Unicode string", - "input": "café", - "expected": "café", - "specSection": "7.4" + "input": "caf\u00e9", + "expected": "caf\u00e9", + "specSection": "16" }, { "name": "parses Chinese characters", "input": "你好", "expected": "你好", - "specSection": "7.4" - }, - { - "name": "parses emoji", - "input": "🚀", - "expected": "🚀", - "specSection": "7.4" + "specSection": "16" }, { "name": "parses string with emoji and spaces", "input": "hello 👋 world", "expected": "hello 👋 world", - "specSection": "7.4" + "specSection": "16" }, { "name": "parses positive integer", @@ -127,49 +115,28 @@ "name": "respects ambiguity quoting for true", "input": "\"true\"", "expected": "true", - "specSection": "7.4" - }, - { - "name": "respects ambiguity quoting for false", - "input": "\"false\"", - "expected": "false", - "specSection": "7.4" - }, - { - "name": "respects ambiguity quoting for null", - "input": "\"null\"", - "expected": "null", - "specSection": "7.4" + "specSection": "4" }, { "name": "respects ambiguity quoting for integer", "input": "\"42\"", "expected": "42", - "specSection": "7.4" - }, - { - "name": "respects ambiguity quoting for negative decimal", - "input": "\"-3.14\"", - "expected": "-3.14", - "specSection": "7.4" - }, - { - "name": "respects ambiguity quoting for scientific notation", - "input": "\"1e-6\"", - "expected": "1e-6", - "specSection": "7.4" - }, - { - "name": "respects ambiguity quoting for leading-zero", - "input": "\"05\"", - "expected": "05", - "specSection": "7.4" + "specSection": "4" }, { "name": "decodes supplementary scalar (U+1F680) in quoted string as literal UTF-8", "input": "\"🚀 launch\"", "expected": "🚀 launch", "specSection": "7.1" + }, + { + "name": "decodes a raw control character inside a quoted string", + "input": "val: \"x\u0001y\"", + "expected": { + "val": "x\u0001y" + }, + "specSection": "7.1", + "minSpecVersion": "4.2" } ] } diff --git a/tests/fixtures/decode/root-form.json b/tests/fixtures/decode/root-form.json index c5fc94c..081bdde 100644 --- a/tests/fixtures/decode/root-form.json +++ b/tests/fixtures/decode/root-form.json @@ -12,33 +12,6 @@ }, "specSection": "5" }, - { - "name": "parses single primitive string at root as primitive", - "input": "hello", - "expected": "hello", - "options": { - "strict": true - }, - "specSection": "5" - }, - { - "name": "parses single primitive number at root as primitive", - "input": "42", - "expected": 42, - "options": { - "strict": true - }, - "specSection": "5" - }, - { - "name": "parses single primitive boolean at root as primitive", - "input": "true", - "expected": true, - "options": { - "strict": true - }, - "specSection": "5" - }, { "name": "parses literal [] at root as empty array", "input": "[]", @@ -99,6 +72,43 @@ "specSection": "5", "note": "The non-strict leniency for trailing content excludes scalar lines (§5.2)", "minSpecVersion": "4.1" + }, + { + "name": "parses a root line shaped like a header without a colon as a string", + "input": "items[2]", + "expected": "items[2]", + "specSection": "6", + "minSpecVersion": "4.2" + }, + { + "name": "throws on an indented root primitive in non-strict mode", + "input": " hello", + "expected": null, + "shouldError": true, + "options": { + "strict": false + }, + "specSection": "5", + "note": "The non-strict skip for over-indented lines excludes scalar lines (§5.2)", + "minSpecVersion": "4.2" + }, + { + "name": "parses a root primitive whose colon follows an unclosed quote", + "input": "a\"b: 1", + "expected": "a\"b: 1", + "specSection": "5.2", + "minSpecVersion": "4.2" + }, + { + "name": "throws on a scalar line after a skipped over-indented first line in non-strict mode", + "input": " a: 1\nhello", + "expected": null, + "shouldError": true, + "options": { + "strict": false + }, + "specSection": "5", + "minSpecVersion": "4.2" } ] } diff --git a/tests/fixtures/decode/validation-errors.json b/tests/fixtures/decode/validation-errors.json index a55d2b8..48055b5 100644 --- a/tests/fixtures/decode/validation-errors.json +++ b/tests/fixtures/decode/validation-errors.json @@ -36,7 +36,7 @@ "input": "\"a\\x\"", "expected": null, "shouldError": true, - "specSection": "14.2" + "specSection": "7.1" }, { "name": "throws on truncated unicode escape \\u00b", @@ -57,7 +57,7 @@ "input": "\"unterminated", "expected": null, "shouldError": true, - "specSection": "14.2" + "specSection": "7.1" }, { "name": "throws on missing colon in key-value context", @@ -81,7 +81,7 @@ "input": "items[2\t]{a\tb}:\n 1,2\n 3,4", "expected": null, "shouldError": true, - "specSection": "14.1", + "specSection": "11.2", "note": "Active delimiter is tab; rows using comma each parse as 1 value, failing the row width check" }, { @@ -136,11 +136,11 @@ "specSection": "14.3" }, { - "name": "throws on array header missing colon", + "name": "throws on a header-shaped line without a colon above indented rows", "input": "items[2]{id,name}\n 1,Ada\n 2,Bob", "expected": null, "shouldError": true, - "specSection": "6" + "specSection": "5.2" }, { "name": "throws when inline values fall short of the declared length", @@ -213,7 +213,7 @@ }, { "name": "throws on whitespace between bracket segment and colon in strict mode", - "input": "items[2] :\n 1,2", + "input": "items[2] : 1,2", "expected": null, "shouldError": true, "options": { @@ -224,7 +224,7 @@ }, { "name": "throws on whitespace between bracket segment and field list in strict mode", - "input": "items[2] {a,b}:\n 1,2\n 3,4", + "input": "items[0] {a,b}:", "expected": null, "shouldError": true, "options": { @@ -271,14 +271,14 @@ }, { "name": "throws on empty field list in strict mode", - "input": "items[1]{}:\n 1", + "input": "items[0]{}:", "expected": null, "shouldError": true, "specSection": "14.2" }, { "name": "throws on empty nested field group in strict mode", - "input": "items[1]{id,meta{}}:\n 1", + "input": "items[0]{id,meta{}}:", "expected": null, "shouldError": true, "specSection": "14.2", @@ -286,21 +286,20 @@ }, { "name": "throws on unmatched brace in field list in strict mode", - "input": "items[1]{id,customer{name:\n 1,Ada", + "input": "items[0]{id,customer{name:", "expected": null, "shouldError": true, "specSection": "14.2" }, { "name": "throws on duplicate field names at the same field-list level in strict mode", - "input": "items[1]{a,a{x}}:\n 1,2", + "input": "items[0]{a,a{x}}:", "expected": null, "shouldError": true, "options": { "strict": true }, - "specSection": "14.3", - "note": "A bare field and a nested field group sharing one name yield duplicate sibling keys in every decoded row" + "specSection": "9.3" }, { "name": "throws on entry row count mismatch with keyed header length", @@ -443,7 +442,7 @@ }, { "name": "throws on keyless fields-bearing header as list item", - "input": "items[1]:\n - [2]{x}:\n 1\n 2", + "input": "items[1]:\n - [0]{x}:", "expected": null, "shouldError": true, "specSection": "6" @@ -631,6 +630,104 @@ "shouldError": true, "specSection": "9.1", "note": "Nothing after the colon is list form with zero items, not one empty inline value" + }, + { + "name": "throws on a length beyond the integer range that wraps to the value count", + "input": "a[18446744073709551617]: 1", + "expected": null, + "shouldError": true, + "specSection": "6", + "note": "2^64 + 1 wraps to 1 in 64-bit integers; an unrepresentable length is unmet (§14.1)", + "minSpecVersion": "4.2" + }, + { + "name": "throws on an empty field entry", + "input": "a[0]{x,}:", + "expected": null, + "shouldError": true, + "specSection": "6", + "minSpecVersion": "4.2" + }, + { + "name": "throws on a space between a field name and its nested field group", + "input": "a[0]{b {c}}:", + "expected": null, + "shouldError": true, + "specSection": "6", + "minSpecVersion": "4.2" + }, + { + "name": "throws on a nested field group without a name", + "input": "a[0]{x,{y}}:", + "expected": null, + "shouldError": true, + "specSection": "6", + "minSpecVersion": "4.2" + }, + { + "name": "throws on a \\u escape with a sign before its hex digits", + "input": "val: \"\\u+0ab\"", + "expected": null, + "shouldError": true, + "specSection": "7.1" + }, + { + "name": "throws on an escaped lone low surrogate", + "input": "val: \"\\udc00\"", + "expected": null, + "shouldError": true, + "specSection": "7.1" + }, + { + "name": "throws on characters after a closing quote in an inline array cell", + "input": "k[2]: \"a\"x,b", + "expected": null, + "shouldError": true, + "specSection": "7.4", + "minSpecVersion": "4.1" + }, + { + "name": "throws on a key-value line among tabular rows that breaks the declared count", + "input": "t[2]{a}:\n 1\n x: 2", + "expected": null, + "shouldError": true, + "specSection": "9.3" + }, + { + "name": "throws on a tabular row with more cells than fields", + "input": "items[1]{a}:\n 1,2", + "expected": null, + "shouldError": true, + "specSection": "14.1" + }, + { + "name": "throws on more keyed entry rows than declared", + "input": "m[1:]{v}:\n a: 1\n b: 2", + "expected": null, + "shouldError": true, + "specSection": "14.1" + }, + { + "name": "throws on an invalid escape in non-strict mode", + "input": "k: \"a\\x\"", + "expected": null, + "shouldError": true, + "options": { + "strict": false + }, + "specSection": "7.1", + "minSpecVersion": "4.1" + }, + { + "name": "throws on an unterminated string in non-strict mode", + "input": "k: \"abc", + "expected": null, + "shouldError": true, + "options": { + "strict": false + }, + "specSection": "7.1", + "minSpecVersion": "4.1" } ] } diff --git a/tests/fixtures/decode/whitespace.json b/tests/fixtures/decode/whitespace.json index f527885..06eed2f 100644 --- a/tests/fixtures/decode/whitespace.json +++ b/tests/fixtures/decode/whitespace.json @@ -9,8 +9,7 @@ "expected": { "tags": ["a", "b", "c"] }, - "specSection": "12", - "note": "Trimming removes U+0020 only, never other Unicode whitespace" + "specSection": "12" }, { "name": "tolerates spaces around pipes in inline arrays", @@ -161,6 +160,15 @@ }, "specSection": "12", "minSpecVersion": "4.1" + }, + { + "name": "removes a byte-order mark before a comment line", + "input": "\ufeff# c\na: 1", + "expected": { + "a": 1 + }, + "specSection": "12", + "minSpecVersion": "4.1" } ] } diff --git a/tests/fixtures/encode/arrays-nested.json b/tests/fixtures/encode/arrays-nested.json index 55e0715..079ddfb 100644 --- a/tests/fixtures/encode/arrays-nested.json +++ b/tests/fixtures/encode/arrays-nested.json @@ -3,14 +3,6 @@ "category": "encode", "description": "Nested and mixed array encoding – arrays of arrays, mixed type arrays, root arrays", "tests": [ - { - "name": "encodes nested arrays of primitives", - "input": { - "pairs": [["a", "b"], ["c", "d"]] - }, - "expected": "pairs[2]:\n - [2]: a,b\n - [2]: c,d", - "specSection": "9.2" - }, { "name": "quotes strings containing delimiters in nested arrays", "input": { @@ -27,14 +19,6 @@ "expected": "pairs[2]:\n - [0]:\n - [0]:", "specSection": "9.2" }, - { - "name": "encodes mixed-length inner arrays", - "input": { - "pairs": [[1], [2, 3]] - }, - "expected": "pairs[2]:\n - [1]: 1\n - [2]: 2,3", - "specSection": "9.2" - }, { "name": "encodes root-level primitive array", "input": ["x", "y", "true", true, 10], @@ -71,20 +55,6 @@ "expected": "[]", "specSection": "9.1" }, - { - "name": "encodes complex nested structure", - "input": { - "user": { - "id": 123, - "name": "Ada", - "tags": ["reading", "gaming"], - "active": true, - "prefs": [] - } - }, - "expected": "user:\n id: 123\n name: Ada\n tags[2]: reading,gaming\n active: true\n prefs: []", - "specSection": "8" - }, { "name": "uses list form for arrays mixing primitives and objects", "input": { @@ -112,19 +82,10 @@ { "name": "uses list form for a tabular-eligible array in list-item position", "input": { - "a": [ - [ - { - "x": 1 - }, - { - "x": 2 - } - ] - ] + "a": [[{ "x": 1 }, { "x": 2 }]] }, "expected": "a[1]:\n - [2]:\n - x: 1\n - x: 2", - "specSection": "9.3", + "specSection": "9.4", "note": "A keyless fields-bearing header is valid only at the document root", "minSpecVersion": "4.1" }, diff --git a/tests/fixtures/encode/arrays-objects.json b/tests/fixtures/encode/arrays-objects.json index bc46a04..3bcc50c 100644 --- a/tests/fixtures/encode/arrays-objects.json +++ b/tests/fixtures/encode/arrays-objects.json @@ -14,18 +14,6 @@ "expected": "items[2]:\n - id: 1\n name: First\n - id: 2\n name: Second\n extra: true", "specSection": "9.4" }, - { - "name": "uses list form for objects with nested values", - "input": { - "items": [ - { "id": 1, "nested": { "x": 1 } }, - { "id": 2, "nested": { "y": 2 } } - ] - }, - "expected": "items[2]:\n - id: 1\n nested:\n x: 1\n - id: 2\n nested:\n y: 2", - "specSection": "9.4", - "note": "The nested column mixes key sets {x} and {y}, so it is not nested-uniform (§9.3), so the array uses list form" - }, { "name": "preserves field order in list items - array first", "input": { @@ -73,22 +61,6 @@ "expected": "items[1]:\n - users[2]:\n - id: 1\n name: Ada\n - id: 2\n status: active", "specSection": "10" }, - { - "name": "uses list form for objects with multiple array fields", - "input": { - "items": [{ "nums": [1, 2], "tags": ["a", "b"], "name": "test" }] - }, - "expected": "items[1]:\n - nums[2]: 1,2\n tags[2]: a,b\n name: test", - "specSection": "10" - }, - { - "name": "uses list form for objects with only array fields", - "input": { - "items": [{ "nums": [1, 2, 3], "tags": ["a", "b"] }] - }, - "expected": "items[1]:\n - nums[3]: 1,2,3\n tags[2]: a,b", - "specSection": "10" - }, { "name": "encodes objects with empty arrays in list form", "input": { @@ -99,15 +71,6 @@ "expected": "items[1]:\n - name: Ada\n data: []", "specSection": "10" }, - { - "name": "uses canonical encoding for multi-field list-item objects with tabular arrays", - "input": { - "items": [{ "users": [{ "id": 1 }, { "id": 2 }], "note": "x" }] - }, - "expected": "items[1]:\n - users[2]{id}:\n 1\n 2\n note: x", - "specSection": "10", - "note": "Tabular header on hyphen line with rows at depth +2 and sibling fields at depth +1" - }, { "name": "uses canonical encoding for single-field list-item tabular arrays", "input": { @@ -125,15 +88,6 @@ "expected": "items[1]:\n - data: []\n name: x", "specSection": "10" }, - { - "name": "encodes empty object list items as bare hyphen", - "input": { - "items": ["first", "second", {}] - }, - "expected": "items[3]:\n - first\n - second\n -", - "specSection": "10", - "note": "Empty object list items encode as a single \"-\" line at the list-item depth" - }, { "name": "uses field order from first object for tabular headers", "input": { diff --git a/tests/fixtures/encode/arrays-primitive.json b/tests/fixtures/encode/arrays-primitive.json index 5644091..5a2ab0e 100644 --- a/tests/fixtures/encode/arrays-primitive.json +++ b/tests/fixtures/encode/arrays-primitive.json @@ -3,22 +3,6 @@ "category": "encode", "description": "Primitive array encoding – inline arrays of strings, numbers, booleans", "tests": [ - { - "name": "encodes string arrays inline", - "input": { - "tags": ["reading", "gaming"] - }, - "expected": "tags[2]: reading,gaming", - "specSection": "9.1" - }, - { - "name": "encodes number arrays inline", - "input": { - "nums": [1, 2, 3] - }, - "expected": "nums[3]: 1,2,3", - "specSection": "9.1" - }, { "name": "encodes mixed primitive arrays inline", "input": { @@ -84,7 +68,7 @@ "specSection": "9.1" }, { - "name": "quotes strings that look like booleans in arrays", + "name": "quotes strings that look like literals or numbers in arrays", "input": { "items": ["x", "true", "42", "-3.14"] }, diff --git a/tests/fixtures/encode/arrays-tabular.json b/tests/fixtures/encode/arrays-tabular.json index 2365042..f013733 100644 --- a/tests/fixtures/encode/arrays-tabular.json +++ b/tests/fixtures/encode/arrays-tabular.json @@ -83,7 +83,7 @@ "input": { "orders": [ { "id": 1, "customer": { "name": "Ada", "country": "DK" }, "total": 99 }, - { "id": 2, "customer": { "name": "Bob", "country": "UK" }, "total": 149 } + { "id": 2, "customer": { "country": "UK", "name": "Bob" }, "total": 149 } ] }, "expected": "orders[2]{id,customer{name,country},total}:\n 1,Ada,DK,99\n 2,Bob,UK,149", diff --git a/tests/fixtures/encode/delimiters.json b/tests/fixtures/encode/delimiters.json index 03c9271..b8fbc61 100644 --- a/tests/fixtures/encode/delimiters.json +++ b/tests/fixtures/encode/delimiters.json @@ -122,17 +122,6 @@ }, "specSection": "11" }, - { - "name": "quotes strings containing tab delimiter", - "input": { - "items": ["a", "b\tc", "d"] - }, - "expected": "items[3\t]: a\t\"b\\tc\"\td", - "options": { - "delimiter": "\t" - }, - "specSection": "11" - }, { "name": "quotes strings containing pipe delimiter", "input": { @@ -155,31 +144,6 @@ }, "specSection": "11" }, - { - "name": "does not quote commas with pipe delimiter", - "input": { - "items": ["a,b", "c,d"] - }, - "expected": "items[2|]: a,b|c,d", - "options": { - "delimiter": "|" - }, - "specSection": "11" - }, - { - "name": "quotes tabular values containing comma delimiter", - "input": { - "items": [ - { "id": 1, "note": "a,b" }, - { "id": 2, "note": "c,d" } - ] - }, - "expected": "items[2]{id,note}:\n 1,\"a,b\"\n 2,\"c,d\"", - "options": { - "delimiter": "," - }, - "specSection": "11" - }, { "name": "does not quote commas in tabular values with tab delimiter", "input": { @@ -205,17 +169,6 @@ }, "specSection": "11" }, - { - "name": "does not quote commas in object values with tab delimiter", - "input": { - "note": "a,b" - }, - "expected": "note: a,b", - "options": { - "delimiter": "\t" - }, - "specSection": "11" - }, { "name": "quotes nested array values containing pipe delimiter", "input": { @@ -228,26 +181,22 @@ "specSection": "11" }, { - "name": "quotes nested array values containing tab delimiter", - "input": { - "pairs": [["a", "b\tc"]] - }, - "expected": "pairs[1\t]:\n - [2\t]: a\t\"b\\tc\"", - "options": { - "delimiter": "\t" - }, - "specSection": "11" - }, - { - "name": "preserves ambiguity quoting regardless of delimiter", + "name": "quotes a primitive list item containing the document delimiter", "input": { - "items": ["true", "42", "-3.14"] + "a": [ + "x|y", + "p,q", + { + "b": 1 + } + ] }, - "expected": "items[3|]: \"true\"|\"42\"|\"-3.14\"", + "expected": "a[3|]:\n - \"x|y\"\n - p,q\n - b: 1", "options": { "delimiter": "|" }, - "specSection": "11" + "specSection": "11.1", + "minSpecVersion": "4.2" } ] } diff --git a/tests/fixtures/encode/objects-keyed.json b/tests/fixtures/encode/objects-keyed.json index 7fd4e87..d8841f5 100644 --- a/tests/fixtures/encode/objects-keyed.json +++ b/tests/fixtures/encode/objects-keyed.json @@ -62,14 +62,10 @@ { "name": "quotes entry keys per key encoding", "input": { - "ids": { - "42": { "v": 1 }, - "my key": { "v": 2 } - } + "ids": { "42": { "v": 1 }, "my key": { "v": 2 } } }, "expected": "ids[2:]{v}:\n \"42\": 1\n \"my key\": 2", - "specSection": "9.5", - "note": "Entry keys follow §7.3, the same rule as object keys" + "specSection": "9.5" }, { "name": "quotes entry-row cells containing the active delimiter", @@ -150,6 +146,14 @@ "expected": "items[2]:\n - a:\n x: 1\n b:\n x: 2\n - 5", "specSection": "9.5", "note": "The element object is keyed-eligible but has no key; the keyless form is root-only, so it encodes per §10" + }, + { + "name": "keeps an object of empty objects out of keyed tabular form", + "input": { + "m": { "a": {}, "b": {} } + }, + "expected": "m:\n a:\n b:", + "specSection": "9.5" } ] } diff --git a/tests/fixtures/encode/objects.json b/tests/fixtures/encode/objects.json index 7441aa4..00ce231 100644 --- a/tests/fixtures/encode/objects.json +++ b/tests/fixtures/encode/objects.json @@ -23,7 +23,7 @@ "specSection": "8" }, { - "name": "encodes empty objects as empty string", + "name": "encodes an empty root object as an empty document", "input": {}, "expected": "", "specSection": "8" @@ -93,11 +93,11 @@ "specSection": "7.2" }, { - "name": "quotes string value that looks like negative decimal", + "name": "quotes string value that looks like a number with an uppercase exponent", "input": { - "v": "-7.5" + "v": "1E6" }, - "expected": "v: \"-7.5\"", + "expected": "v: \"1E6\"", "specSection": "7.2" }, { @@ -182,12 +182,13 @@ "note": "The unquoted-key pattern is ASCII-only, so every non-ASCII key is quoted" }, { - "name": "quotes CJK key", + "name": "keeps a dotted key unquoted", "input": { - "名前": "x" + "a.b": 1 }, - "expected": "\"名前\": x", - "specSection": "7.3" + "expected": "a.b: 1", + "specSection": "7.3", + "minSpecVersion": "4.2" }, { "name": "escapes newline in key", @@ -259,15 +260,6 @@ "specSection": "15", "note": "No key has special meaning, so the encoder emits prototype-named own entries like any other (§8)" }, - { - "name": "encodes constructor and prototype own properties as ordinary keys", - "input": { - "constructor": 1, - "prototype": 2 - }, - "expected": "constructor: 1\nprototype: 2", - "specSection": "15" - }, { "name": "encodes __proto__ as a tabular field name", "input": { @@ -302,6 +294,17 @@ }, "expected": "\"\":\n a: 1", "specSection": "8" + }, + { + "name": "keeps strings that only resemble numbers or literals unquoted", + "input": { + "a": ".5", + "b": "NaN", + "c": "True" + }, + "expected": "a: .5\nb: NaN\nc: True", + "specSection": "7.2", + "minSpecVersion": "4.2" } ] } diff --git a/tests/fixtures/encode/primitives.json b/tests/fixtures/encode/primitives.json index 826c95e..edaa4d0 100644 --- a/tests/fixtures/encode/primitives.json +++ b/tests/fixtures/encode/primitives.json @@ -46,9 +46,9 @@ "specSection": "7.2" }, { - "name": "quotes string that looks like negative decimal", - "input": "-3.14", - "expected": "\"-3.14\"", + "name": "quotes string that looks like a decimal", + "input": "3.14", + "expected": "\"3.14\"", "specSection": "7.2" }, { @@ -95,10 +95,11 @@ "specSection": "7.1" }, { - "name": "quotes string with array-like syntax", - "input": "[3]: x,y", - "expected": "\"[3]: x,y\"", - "specSection": "7.2" + "name": "keeps a string with an interior hash unquoted", + "input": "a #b", + "expected": "a #b", + "specSection": "7.2", + "minSpecVersion": "4.2" }, { "name": "quotes string starting with hyphen-space", @@ -127,8 +128,10 @@ }, { "name": "quotes leading-hyphen string in array", - "input": { "tags": ["a", "- item", "b"] }, - "expected": "tags[3]: a,\"- item\",b", + "input": { + "tags": ["-x"] + }, + "expected": "tags[1]: \"-x\"", "specSection": "7.2" }, { @@ -149,18 +152,6 @@ "expected": "café", "specSection": "7.2" }, - { - "name": "encodes Chinese characters without quotes", - "input": "你好", - "expected": "你好", - "specSection": "7.2" - }, - { - "name": "encodes emoji without quotes", - "input": "🚀", - "expected": "🚀", - "specSection": "7.2" - }, { "name": "encodes string with emoji and spaces", "input": "hello 👋 world", @@ -200,13 +191,13 @@ }, { "name": "encodes negative zero as zero", - "input": -0, + "input": -0.0, "expected": "0", "specSection": "2" }, { "name": "encodes large integer without exponent notation", - "input": 1000000, + "input": 1e6, "expected": "1000000", "specSection": "2", "note": "Within the canonical decimal range" @@ -281,6 +272,14 @@ "specSection": "7.2", "note": "Unquoted, a decoder would remove the U+FEFF as a byte-order mark (§12) and read the number 8", "minSpecVersion": "4.1" + }, + { + "name": "encodes a large integer-valued number with its shortest round-trip digits", + "input": 123456789012345680000, + "expected": "123456789012345680000", + "specSection": "2", + "note": "The binary64 value is exactly 123456789012345683968", + "minSpecVersion": "4.2" } ] } diff --git a/tests/fixtures/encode/whitespace.json b/tests/fixtures/encode/whitespace.json index 1c6eae3..07dda0a 100644 --- a/tests/fixtures/encode/whitespace.json +++ b/tests/fixtures/encode/whitespace.json @@ -3,14 +3,6 @@ "category": "encode", "description": "Whitespace and formatting invariants – no trailing spaces, no trailing newlines", "tests": [ - { - "name": "produces no trailing newline at end of output", - "input": { - "id": 123 - }, - "expected": "id: 123", - "specSection": "12" - }, { "name": "maintains proper indentation for nested structures", "input": { @@ -46,6 +38,17 @@ "specSection": "7.2", "note": "The whitespace trigger is U+0020 and U+0009 only", "minSpecVersion": "4.1" + }, + { + "name": "indents list-item sibling fields by indentSize", + "input": { + "a": [{ "b": 1, "c": 2 }, 1] + }, + "expected": "a[2]:\n - b: 1\n c: 2\n - 1", + "options": { + "indentSize": 4 + }, + "specSection": "10" } ] }