From a511c27f7b033b8041b534420e631b042a7043be Mon Sep 17 00:00:00 2001 From: Johann Schopplich Date: Thu, 1 Oct 2026 14:50:24 +0200 Subject: [PATCH] docs: tighten out-of-scope records --- .out-of-scope/format-conversion.md | 27 +++++------------------ .out-of-scope/framework-integrations.md | 29 ++++++------------------- .out-of-scope/json-text-helpers.md | 26 ++++++---------------- .out-of-scope/native-backend.md | 10 +++------ 4 files changed, 23 insertions(+), 69 deletions(-) diff --git a/.out-of-scope/format-conversion.md b/.out-of-scope/format-conversion.md index c99245b..b1a6b4a 100644 --- a/.out-of-scope/format-conversion.md +++ b/.out-of-scope/format-conversion.md @@ -1,30 +1,15 @@ # Conversion to Other Formats -This package converts between Python values and TOON, and nothing else. It does not parse or write CSV, YAML, or XML, does not auto-detect input formats, and does not batch-convert files or bundle them into archives. +This package converts between Python values and TOON, and nothing else. It does not read or write CSV, YAML, or XML, and does not batch-convert files. ## Why this is out of scope -The spec scopes TOON as a translation layer for the JSON data model: "produce data as JSON in code, encode to TOON for downstream consumption, and decode back to JSON if needed" (Introduction, Purpose and Scope). Python values are the hub. Any other format reaches TOON through a parser that already turns it into dicts and lists: +TOON carries the JSON data model (§2), so any other format reaches it through a parser that already yields dicts and lists – `json`, `csv`, or a mature YAML or XML library. A converter here would duplicate those parsers and turn every edge case of their formats into a bug report against this package. As Johann put it on the CSV pull request: "this is completely out of scope for the TOON Python package. There are numerous battle-tested CSV parsers available" ([#39](https://github.com/toon-format/toon-python/pull/39#issuecomment-3539108955)). -```python -import csv -import json - -encode(json.load(f)) -encode({"rows": list(csv.DictReader(f))}) - -rows = decode(text)["rows"] -writer = csv.DictWriter(out, fieldnames=rows[0].keys()) -writer.writeheader() -writer.writerows(rows) -``` - -Shipping our own CSV or YAML reader would duplicate the standard library and mature third-party parsers, and every edge case of those formats would turn into a bug report here. As Johann put it on the CSV pull request: "this is completely out of scope for the TOON Python package. There are numerous battle-tested CSV parsers available" ([#39](https://github.com/toon-format/toon-python/pull/39#issuecomment-3539108955)). - -The `toon` command follows the same line: it converts between JSON and TOON, because JSON is the data model TOON carries (§2). +The `toon` command follows the same line and converts between JSON and TOON only. ## Prior requests -- [#25](https://github.com/toon-format/toon-python/issues/25) / [#35](https://github.com/toon-format/toon-python/pull/35): "batch processing" – multi-format conversion between JSON, YAML, XML, CSV, and TOON with auto-detection -- [#38](https://github.com/toon-format/toon-python/discussions/38): "TOON to CSV python lib" – merging `tooncsv` into this package -- [#39](https://github.com/toon-format/toon-python/pull/39): "Add CSV Parsing & Writing Support to toon_format" +- [#25](https://github.com/toon-format/toon-python/issues/25) / [#35](https://github.com/toon-format/toon-python/pull/35) – "batch processing": conversion between JSON, YAML, XML, CSV, and TOON +- [#38](https://github.com/toon-format/toon-python/discussions/38) – "TOON to CSV python lib": merging `tooncsv` into this package +- [#39](https://github.com/toon-format/toon-python/pull/39) – "Add CSV Parsing & Writing Support to toon_format" diff --git a/.out-of-scope/framework-integrations.md b/.out-of-scope/framework-integrations.md index 8303168..4880890 100644 --- a/.out-of-scope/framework-integrations.md +++ b/.out-of-scope/framework-integrations.md @@ -1,31 +1,16 @@ # Framework Integrations -This package does not ship integrations for LLM or data frameworks – no LangChain serializers or output parsers, no agent-framework adapters, no pandas API, and no MCP server. The core library encodes Python values to TOON and decodes TOON back to Python values. Everything else builds on those two calls. +This package does not ship integrations for LLM or data frameworks – no LangChain serializers or output parsers, no agent-framework adapters, no pandas API, and no MCP server. ## Why this is out of scope -Every one of these integrations is a thin layer over `encode()` and `decode()`, but each one ties the package to someone else's release cycle. An integration for LangChain, pandas, or an MCP server means another optional dependency to pin, another API that breaks on its own schedule, and another test matrix – for code that is a single call on the user's side: +Each integration is a thin layer over one encode or decode call, but it ties the package to another project's dependencies, API, and release cycle. Johann closed the LangChain pull request on that line: "a LangChain integration is out of scope for the core library, which should stay focused on encoding and decoding. This would be a great fit as a separate package (e.g. `toon-langchain`)" ([#45](https://github.com/toon-format/toon-python/pull/45#issuecomment-4073109814)). The MCP server got the same answer ([#31](https://github.com/toon-format/toon-python/pull/31#issuecomment-4073055880)). -```python -# pandas -encode({"customers": df[["Customer", "Customer question"]].to_dict("records")}) - -# Pydantic models, dataclasses, agent-framework payloads -encode(model.model_dump()) -encode(dataclasses.asdict(obj)) -``` - -The DataFrame example already emits a tabular array (§9.3), `customers[3]{Customer,"Customer question"}:` with one row per record, so a dedicated pandas API wouldn't produce anything better. - -Johann closed the LangChain integration with the same reasoning: "a LangChain integration is out of scope for the core library, which should stay focused on encoding and decoding. This would be a great fit as a separate package (e.g. `toon-langchain`)" ([#45](https://github.com/toon-format/toon-python/pull/45#issuecomment-4073109814)). The MCP server got the same answer: "toon-python should stay focused on encoding/decoding. If there's appetite for this, it would be better as a separate repository" ([#31](https://github.com/toon-format/toon-python/pull/31#issuecomment-4073055880)). - -Integrations are welcome as separate packages, or on the framework's side. - -The optional `pydantic` extra ([#46](https://github.com/toon-format/toon-python/pull/46)) exists and is not part of this decision. Host-type normalization through a `JSONEncoder.default`-style hook is allowed by the spec (§3) and is a regular feature request, not an integration. +Integrations are welcome as separate packages or on the framework's side. The optional `pydantic` extra ([#46](https://github.com/toon-format/toon-python/pull/46)) is not affected. ## Prior requests -- [#31](https://github.com/toon-format/toon-python/pull/31): "Add Model Context Protocol (MCP) Server for TOON Format" -- [#44](https://github.com/toon-format/toon-python/discussions/44): "Agent framework integration" (LangChain, Pydantic AI) -- [#45](https://github.com/toon-format/toon-python/pull/45): "feat(langchain): add ToonSerializer and ToonOutputParser" -- [#56](https://github.com/toon-format/toon-python/issues/56): "An interface with Pandas" +- [#31](https://github.com/toon-format/toon-python/pull/31) – "Add Model Context Protocol (MCP) Server for TOON Format" +- [#44](https://github.com/toon-format/toon-python/discussions/44) – "Agent framework integration" (LangChain, Pydantic AI) +- [#45](https://github.com/toon-format/toon-python/pull/45) – "feat(langchain): add ToonSerializer and ToonOutputParser" +- [#56](https://github.com/toon-format/toon-python/issues/56) – "An interface with Pandas" diff --git a/.out-of-scope/json-text-helpers.md b/.out-of-scope/json-text-helpers.md index 90d0ddf..87c6f89 100644 --- a/.out-of-scope/json-text-helpers.md +++ b/.out-of-scope/json-text-helpers.md @@ -1,28 +1,16 @@ # JSON Text Helpers -`encode()` takes Python values and `decode()` returns Python values, the same way `json.dumps` and `json.loads` work. The package does not add variants that read or write JSON text: no `json_indent` option on `decode()`, no `encode_json(text)`, no `loads` alias for `json.loads`. +The encoder takes Python values and the decoder returns them, as the `json` module does. There are no variants that read or write JSON text – no JSON indentation option on decode, no `encode_json(text)`, no wrapper around `json.loads`. ## Why this is out of scope -The `json` module already does the JSON half, and each helper would only wrap one call to it: +Each helper would wrap a single call to the `json` module, and a JSON indentation option would also make decode return a string instead of the decoded value. JSON text belongs to the `toon` command, which reads JSON and writes indented JSON. -```python -import json - -encode(json.loads(text)) # JSON text → TOON -json.dumps(decode(toon_text), indent=2) # TOON → pretty-printed JSON -``` - -A `json_indent` option on `decode()` also breaks the return type: with it set, `decode()` returns a `str` instead of the decoded value, so every typed caller has to narrow a `JsonValue | str` union – for output that `json.dumps` produces without it. - -The public API also stays close to the reference implementation, where `encode()` and `decode()` work on in-memory values only. @smortezah raised this on the `encode_json` pull request: "To maintain alignment with the original TypeScript implementation of toon-format, I suggest we avoid adding excessive functions" ([#57](https://github.com/toon-format/toon-python/pull/57)). - -The `toon` command is where JSON text belongs: it reads and writes JSON files and indents its JSON output on decode. - -JSON `null` needs no helper either: `json.loads` returns `None`, which `encode()` emits as `null` (§2). +JSON `null` needs no helper either: `json.loads` returns `None`, which encodes as `null`. ## Prior requests -- [#10](https://github.com/toon-format/toon-python/issues/10): "JSON indentation option in decode method" -- [#37](https://github.com/toon-format/toon-python/pull/37): "feat: Add JSON indentation option to decode() method" -- [#57](https://github.com/toon-format/toon-python/pull/57): "feat: add encode_json and loads helpers for better JSON null support" +- [#10](https://github.com/toon-format/toon-python/issues/10) – "JSON indentation option in decode method" +- [#37](https://github.com/toon-format/toon-python/pull/37) – "feat: Add JSON indentation option to decode() method" +- [#49](https://github.com/toon-format/toon-python/issues/49) – "Support JSON null values (auto-convert to Python None) in toon-python encoding flow" +- [#57](https://github.com/toon-format/toon-python/pull/57) – "feat: add encode_json and loads helpers for better JSON null support" diff --git a/.out-of-scope/native-backend.md b/.out-of-scope/native-backend.md index 5ca6c0e..aa81007 100644 --- a/.out-of-scope/native-backend.md +++ b/.out-of-scope/native-backend.md @@ -1,15 +1,11 @@ # Native Backend -This package is pure Python. It does not bind the Rust implementation through PyO3 or ship any other compiled backend, optional or not. +`toon-format` stays pure Python. A compiled backend – Rust through PyO3 or C++ – belongs in a separate package, not in this one, optional or not. ## Why this is out of scope -A pure Python package installs from a single wheel on every interpreter and platform – CPython, PyPy, GraalPy, free-threaded builds – without a compiler toolchain. A native backend trades that for a wheel matrix per Python version, OS, and architecture, plus a fallback path for everything the matrix misses. @Justar96 laid out the same reasoning when the question first came up: "We're prioritizing pure python for maximal portability (no wheels/toolchains) interpreter coverage PyPy/GraalPy" ([#9](https://github.com/toon-format/toon-python/discussions/9#discussioncomment-14871615)). - -Sharing one implementation across languages doesn't buy correctness either. The ports validate against the same language-agnostic test suite (Appendix C), so the Python and Rust decoders agree because both pass it, not because they share code. A binding would also tie Python releases to the Rust crate's API and release schedule. - -Anyone who needs a Rust-backed binding can publish it as a separate package. +A pure Python wheel installs on every interpreter and platform, PyPy and GraalPy included, without a compiler toolchain. A native backend trades that for a wheel matrix per Python version, OS, and architecture. Speed-focused packages already exist on their own: [toons](https://github.com/alesanfra/toons) (Rust) and [ctoon](https://github.com/mohammadraziei/ctoon) (C++) are listed on the [implementations page](https://toonformat.dev/ecosystem/implementations). ## Prior requests -- [#9](https://github.com/toon-format/toon-python/discussions/9): "Using the rust library instead of a pure python implementation" +- [#9](https://github.com/toon-format/toon-python/discussions/9) – "Using the rust library instead of a pure python implementation"