fix(eval): parse_json(contains=True) bounds the characters its failed tries scan

dokimi/changelog.d/20261113_160000_json-contains-scan-cost.md

@@ -0,0 +1,3 @@
+### Fixed
+
+- **`parse_json(contains=True)` no longer costs starts x text length** — the scan was capped at 1000 `{` or `[` tries, but each try on a long text parsed the whole tail before it failed, so `'[' * 900 + '1,' * 100000` took seconds to read as not JSON. The tries that fail now share a budget of 1,000,000 scanned characters (`CONTAINS_JSON_MAX_SCAN`; a try that fails without a position, or on an unterminated string, is charged to the end of the text; the first parse of the whole text is not counted), and the scan reads the text as not JSON once it is spent (failed tries that have scanned exactly 1,000,000 characters have spent it; one try may cross it), as it does at the start cap. A value found after failed tries that scan under the budget in all, and a value of any length when no earlier try failed late, are found as before; a few failed tries are enough to spend it when each reads a long tail. A check whose scan gives up fails as `not JSON`, it is not left unscored as a `regex` check past its limit is: `'[' * 6 + '1,' * 100000 + ' {"a": 1}'` (six tries of 200,000 characters each) found the object before this change and fails now. The same holds for a value before a long run of brackets: a try that hits the depth limit reports no position, so it is charged the rest of the text, and `'[' + '{"a": 1},' + '[' * 1_100_000` reads as not JSON where it found the object before.

dokimi/docs/answer-formats.md

@@ -57,7 +57,7 @@ question is `<testIdx>` when the config has one prompt and
 | `fuzzy` | Normalised containment, in either direction | OpenAI `FuzzyMatch` |
 | `regex` | `regex.search`, with a 1 s limit per check; a check that runs past it is unscored, not failed | promptfoo `regex` |
 | `choice` | The letters of the last `ANSWER: ...` line, parsed the way Inspect parses them | Inspect `choice` with `multiple_choice()` |
-| `json` | Parses as JSON. With `values`, it must equal one of them. With `contains`, the answer may be any JSON value, or else it is the first JSON object or array anywhere in the answer; the scan tries the first 1000 `{` or `[`, so a value that starts after more brackets than that reads as not JSON. | OpenAI `JsonMatch`; promptfoo `is-json`, `contains-json`, `equals` with an object value |
+| `json` | Parses as JSON. With `values`, it must equal one of them. With `contains`, the answer may be any JSON value, or else it is the first JSON object or array anywhere in the answer; the scan tries the first 1000 `{` or `[` and stops once the tries that failed have scanned 1,000,000 characters in all (a try that fails on an overlong number, the depth limit or an unterminated string counts as scanning to the end of the text), so a value that starts after more brackets or more failed text than that reads as not JSON. | OpenAI `JsonMatch`; promptfoo `is-json`, `contains-json`, `equals` with an object value |
 | `judge` | One required point (`criterion`, `reference` or both), asked of the [checklist](benchmarking-and-comparison.md) judge in both polarities. 1 when the two agree that it is covered, 0 when they agree that it is not, 0.5 when they split. | Inspect `model_graded_qa`/`model_graded_fact`; a custom Inspect `@scorer`, on its docstring's first paragraph; OpenAI `ModelBasedClassify` specs; promptfoo `llm-rubric`, `g-eval`, `factuality`, `model-graded-closedqa`; every skill-creator expectation |
 | `unsupported` | Listed with the native name and never counted | promptfoo `javascript`, `python`, `similar`, `cost`, `latency`, negated judge asserts (`not-llm-rubric`) and other asserts; Inspect `pattern` (it compares an extracted group with the target) and Inspect scorers that are neither built in nor a `@scorer` of the file with a docstring; a row whose `ideal` or `target` is blank or missing |
 | `identifiers` | Every long option the answer names (`--x`, or the `pattern`, inside `within`) is in `known`. An answer that names no option passes, since the check only rules names out. Not `sufficient`: a `sufficient` identifiers check is refused. | None: built from a tool's `--help` by `skill novelty ground`, for novelty probes |

dokimi/dokimi/eval/reference_checks.py

@@ -246,6 +246,10 @@ def chosen_letters(answer: str) -> Optional[frozenset[str]]:
 #: each try on deeply nested text parses to the decoder's depth limit.
 CONTAINS_JSON_MAX_STARTS = 1000
 
+#: ...and the tries that fail may scan at most this many characters in all (each from its start to
+#: where it failed): a text of many brackets and a long tail would otherwise cost starts x length.
+CONTAINS_JSON_MAX_SCAN = 1_000_000
+
 _FENCE = re.compile(r"^\s*```(?:json)?\s*\n(.*?)\n\s*```\s*$", re.S)
 
 
@@ -254,7 +258,9 @@ def parse_json(answer: str, *, contains: bool = False) -> tuple[bool, Any]:
 
     With ``contains`` the first JSON object or array anywhere in the text counts
     (promptfoo ``contains-json``). The scan gives up after :data:`CONTAINS_JSON_MAX_STARTS`
-    brackets, and then reads the text as not JSON.
+    brackets, or once the tries that failed have scanned :data:`CONTAINS_JSON_MAX_SCAN`
+    characters in all (reaching it spends it; one try may cross it), and then reads the text as not JSON. The first parse of the whole text is
+    not counted, only the tries that start at a bracket.
     """
     text = answer.strip()
     fenced = _FENCE.match(text)
@@ -267,15 +273,21 @@ def parse_json(answer: str, *, contains: bool = False) -> tuple[bool, Any]:
     if contains:
         decoder = json.JSONDecoder()
         tried = 0
+        scanned = 0
         for i, ch in enumerate(text):
             if ch in "{[":
-                if tried == CONTAINS_JSON_MAX_STARTS:
+                if tried == CONTAINS_JSON_MAX_STARTS or scanned >= CONTAINS_JSON_MAX_SCAN:
                     break
                 tried += 1
                 try:
                     return True, decoder.raw_decode(text, i)[0]
+                except json.JSONDecodeError as err:
+                    # An unterminated string reports its opening quote, but read to the end.
+                    end = len(text) if err.msg.startswith("Unterminated string") else err.pos
+                    scanned += max(end - i, 0)
                 except (ValueError, RecursionError):
-                    continue
+                    # No position (an overlong integer, the depth limit): charge the whole tail.
+                    scanned += len(text) - i
     return False, None
 
 

dokimi/tests/test_fuzz_targets.py

@@ -165,6 +165,86 @@ def test_parse_json_contains_finds_a_value_after_a_start_too_deep_to_decode() ->
     assert parse_json('[ {"a": 1}, ' + "[1," * 10000, contains=True) == (True, {"a": 1})
 
 
+def test_parse_json_contains_bounds_the_characters_failed_tries_scan(monkeypatch: pytest.MonkeyPatch) -> None:
+    import json
+
+    from dokimi.eval import reference_checks
+    from dokimi.eval.reference_checks import CONTAINS_JSON_MAX_SCAN, parse_json
+
+    calls: list[int] = []
+    real = json.JSONDecoder.raw_decode
+
+    def spy(self: json.JSONDecoder, s: str, idx: int = 0) -> tuple[object, int]:
+        calls.append(idx)
+        return real(self, s, idx)
+
+    monkeypatch.setattr(reference_checks.json.JSONDecoder, "raw_decode", spy)
+    # 900 brackets, then a tail every try parses to its end before it fails.
+    assert parse_json("[" * 900 + "1," * 100000, contains=True) == (False, None)
+    # Each failed try scans the 200000-character tail: the budget allows 1 + CONTAINS_JSON_MAX_SCAN // 200000 + 1 of them.
+    assert len(calls) <= 1 + CONTAINS_JSON_MAX_SCAN // 200000 + 1
+    calls.clear()
+    # A value after a start that failed late, within the budget, is still found.
+    text = "[ " + "1," * 1000 + ' {"a": 1} ' + "[1,"
+    assert parse_json("[ " * 3 + '{"a": [1, 2]}' + " [1," * 100, contains=True) == (True, {"a": [1, 2]})
+    assert parse_json(text, contains=True) == (True, {"a": 1})
+
+
+@pytest.mark.parametrize(
+    "text",
+    [
+        "[" * 900 + "1" * 200000,  # int() refuses the digits: a ValueError without a position
+        "[" * 900 + '"' + "a" * 200000,  # an unterminated string reports its opening quote
+        "[" * 100 + '"' + "a" * 200000 + '",' + "[" * 10000,  # a long read, then the depth limit
+    ],
+    ids=["overlong-int", "unterminated-string", "depth-limit"],
+)
+def test_parse_json_contains_charges_failures_without_a_position(text: str, monkeypatch: pytest.MonkeyPatch) -> None:
+    import json
+
+    from dokimi.eval import reference_checks
+    from dokimi.eval.reference_checks import CONTAINS_JSON_MAX_SCAN, parse_json
+
+    calls: list[int] = []
+    real = json.JSONDecoder.raw_decode
+
+    def spy(self: json.JSONDecoder, s: str, idx: int = 0) -> tuple[object, int]:
+        calls.append(idx)
+        return real(self, s, idx)
+
+    monkeypatch.setattr(reference_checks.json.JSONDecoder, "raw_decode", spy)
+    assert parse_json(text, contains=True) == (False, None)
+    assert len(calls) <= 1 + CONTAINS_JSON_MAX_SCAN // 200000 + 1
+
+
+def test_parse_json_contains_charges_a_failed_try_from_its_start() -> None:
+    from dokimi.eval.reference_checks import CONTAINS_JSON_MAX_SCAN, parse_json
+
+    # The failed "[1,]" sits past the budget's length but scans 4 characters.
+    text = "x" * (CONTAINS_JSON_MAX_SCAN + 500000) + '[1,] {"a": 1}'
+    assert parse_json(text, contains=True) == (True, {"a": 1})
+
+
+def test_parse_json_contains_stops_when_the_failed_tries_reach_the_budget_exactly() -> None:
+    from dokimi.eval.reference_checks import CONTAINS_JSON_MAX_SCAN, parse_json
+
+    # The first try fails at the "2", position CONTAINS_JSON_MAX_SCAN (a missing comma, reported at
+    # the same place on every Python), charging exactly the budget.
+    at_budget = "[1" + " " * (CONTAINS_JSON_MAX_SCAN - 2) + '2] {"a":1}'
+    assert parse_json(at_budget, contains=True) == (False, None)
+    # One character less leaves a character of budget: the object is found.
+    under = "[1" + " " * (CONTAINS_JSON_MAX_SCAN - 3) + '2] {"a":1}'
+    assert parse_json(under, contains=True) == (True, {"a": 1})
+
+
+def test_parse_json_contains_does_not_charge_the_first_whole_text_parse() -> None:
+    from dokimi.eval.reference_checks import CONTAINS_JSON_MAX_SCAN, parse_json
+
+    # json.loads fails after reading past the budget; the first bracket still parses.
+    text = '["' + "a" * (CONTAINS_JSON_MAX_SCAN + 1) + '"] trailing'
+    assert parse_json(text, contains=True) == (True, ["a" * (CONTAINS_JSON_MAX_SCAN + 1)])
+
+
 def test_names_pin_the_byte_to_target_order() -> None:
     # A seed stores its target's index in byte 0: this order is append-only, so a new
     # target goes at the end and the seeds already saved keep their target.

dokimi/tests/test_reference_checks.py

@@ -133,6 +133,17 @@ def test_json_checks_parse_compare_and_contain():
     assert _one({"type": "json", "values": [], "contains": True}, 'Here: {"ok": true} done').passed is True
 
 
+def test_a_contains_json_scan_that_gives_up_is_failed_not_unscored():
+    # Failed tries scan past CONTAINS_JSON_MAX_SCAN before the object: the check fails as "not JSON"
+    # (the same as at the start cap, and as a deeply nested answer), where a regex past its limit is unscored.
+    answer = "[" * 6 + "1," * 100000 + ' {"a": 1}'
+    g = _one({"type": "json", "contains": True}, answer)
+    assert g.checks[0].as_dict() == {"type": "json", "passed": False, "score": 0.0, "detail": "not JSON"}
+    assert _one({"type": "json", "contains": True}, "[" * 20000).checks[0].detail == "not JSON"
+    # Within the budget the same shape is found.
+    assert _one({"type": "json", "contains": True}, "[" * 6 + "1," * 1000 + ' {"a": 1}').passed is True
+
+
 # --- judge, unsupported, weights -----------------------------------------------------------------