Skip to content

fix: export pyarrow tables and accept JSON exporter results - #77

Merged
peter-gy merged 2 commits into
mainfrom
pgy/export-table-capture-and-json-exporters
Oct 3, 2026
Merged

peter-gy merged 2 commits into
mainfrom
pgy/export-table-capture-and-json-exporters

Conversation

@peter-gy

@peter-gy peter-gy commented Oct 2, 2026

Copy link
Copy Markdown
Collaborator

Exports failed when an output referenced a PyArrow table with string or dictionary columns. marimo's cell cache hashes Arrow data through a NumPy view, which rejects object-typed columns, and because that error happens in cache setup the failure surfaced as "failed in cell … with exception" with no type.

  • When the native hash fails, owned graphs retry it with a digest of each referenced PyArrow value's type and IPC stream. Cells that hash natively keep their keys, and untracked graphs are unchanged.
  • Exceptions from cache setup in owned graphs keep their type, so the error names it, e.g. "with TypeError".
  • kind: native exports PyArrow Table and RecordBatch values as Arrow streams, like Polars frames. They previously fell back to marimo's pickle codec and were rejected.
  • Custom exporters can return a JSON value as well as a BlobAsset. JSON results are stored as canonical marimo.json.v1, the same as kind: json, and read back through output.json().

The underlying hashing bug lives in marimo (is_data_primitive and _contiguous_tensor_bytes), so plain marimo sessions with the cell cache enabled still hit it.

Copilot AI balanced review requested due to automatic review settings October 2, 2026 20:13

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 14 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread packages/python/tests/test_marimo_cache_patch.py Outdated
Comment thread packages/python/src/marimo_export/_marimo/compat/cache/attempts.py
Comment thread packages/python/tests/test_native_output_integration.py
Comment thread packages/python/src/marimo_export/_marimo/compat/receipts.py
@peter-gy
peter-gy merged commit 7e90491 into main Oct 3, 2026
20 checks passed
@peter-gy
peter-gy deleted the pgy/export-table-capture-and-json-exporters branch October 3, 2026 06:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants