Skip to content

Add readers for the other formats supported by pandas - #89

Merged
timvink merged 4 commits into
masterfrom
42_more_pandas_readers
Sep 14, 2026
Merged

timvink merged 4 commits into
masterfrom
42_more_pandas_readers

Conversation

@timvink

@timvink timvink commented Sep 14, 2026

Copy link
Copy Markdown
Owner

Closes #42

New readers

read_parquet, read_orc, read_html, read_xml, read_hdf, read_sas, read_spss and read_stata, each with a matching pd_read_* macro. That completes the list in #42.

Two formats need a bit of handling of their own:

  • pd.read_html() returns every table it finds, so read_html() inserts the first one. Use match to select another.
  • pd.read_hdf() returns a pd.Series when a Series was stored, which is converted to a frame.

Refactor

Each reader used to be a copy of the same two functions. They are now built from a LOADERS registry by markdown_reader() and dataframe_reader(), so a new format is one line. Adding a reader to select_readers, the JSON schema and the macros happens automatically.

get_keywords() and friends now accept more than one function, because a reader can take keyword arguments from several: read_yaml() has its own encoding on top of the arguments of pd.json_normalize(). Everything a reader does not accept is still passed on to .to_markdown().

Dependencies

  • pandas>=1.3 instead of >=1.1, because pd.read_xml() was added in 1.3.
  • lxml as a dev dependency, needed by pd.read_html() and pd.read_xml(). It is the only change in uv.lock.

Build times

Adding readers used to make builds slower, because every reader ran its own regex over every page. All selected readers are now searched for in one pattern, so a page is scanned once no matter how many readers there are. On a 15KB page:

8 readers 16 readers
a regex per reader (before) 1.28 ms 2.54 ms
one regex (this PR) 0.16 ms 0.15 ms

That is about a second saved per 1000 pages compared to master, and select_readers is no longer needed to keep builds fast.

Tables are inserted with a replacement function now, instead of as a regex replacement string. A value like C:\1 path used to be expanded as a backreference and silently became C: path; it is now inserted as it is. The same goes for a value like hi\nthere, which used to break the table across two rows, as tests/fixtures/basic_setup/docs/page_carriage_return.md shows.

Tests

tests/fixtures/pandas_readers covers parquet, orc, stata, sas and xml (with the standard library parser), pandas_readers_html covers read_html() and read_xml() with the default parser.

The read_spss and read_hdf tests are skipped unless pyreadstat and tables are installed, which they are not in CI. Both pass locally with uv run --with pyreadstat --with tables pytest. I left those two out of the dev dependencies because they are heavy and have patchy wheel coverage on python 3.8.

Documented in docs/readers.md, including which readers need an extra package installed. read_spss and read_hdf have no live example there for the same reason.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JCJnK1rDt8rmDsXTFc7AHE

timvink and others added 4 commits September 14, 2026 12:46
Adds read_parquet, read_orc, read_html, read_xml, read_hdf, read_sas,
read_spss and read_stata, and the matching pd_read_* macros.

Every reader was a copy of the same two functions, so they are now built
from a registry by markdown_reader() and dataframe_reader(). The keyword
arguments a reader accepts can come from more than one function, which
read_yaml() needs for its own 'encoding' plus pd.json_normalize().

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JCJnK1rDt8rmDsXTFc7AHE
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JCJnK1rDt8rmDsXTFc7AHE
pyarrow cannot find the IANA time zone database there, so pd.read_orc()
raises. Moved to a fixture project of its own and documented.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JCJnK1rDt8rmDsXTFc7AHE
Every reader ran its own regex over every page, so adding readers made
builds slower. One pattern with all selected readers scans a page once,
which is 8x faster than the previous 8 readers and no longer depends on
how many readers are selected.

Tables are now inserted with a replacement function, so a value like
'C:\1 path' is inserted as it is. It used to be expanded as a regex
replacement, which silently changed such values.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JCJnK1rDt8rmDsXTFc7AHE
@timvink
timvink force-pushed the 42_more_pandas_readers branch from 3a256ec to aff4f5b Compare September 14, 2026 10:47
@timvink
timvink merged commit 8aad4ac into master Sep 14, 2026
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a reader for other formats supported by pandas

1 participant