Add readers for the other formats supported by pandas - #89
Merged
Merged
Conversation
Adds read_parquet, read_orc, read_html, read_xml, read_hdf, read_sas, read_spss and read_stata, and the matching pd_read_* macros. Every reader was a copy of the same two functions, so they are now built from a registry by markdown_reader() and dataframe_reader(). The keyword arguments a reader accepts can come from more than one function, which read_yaml() needs for its own 'encoding' plus pd.json_normalize(). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCJnK1rDt8rmDsXTFc7AHE
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCJnK1rDt8rmDsXTFc7AHE
pyarrow cannot find the IANA time zone database there, so pd.read_orc() raises. Moved to a fixture project of its own and documented. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCJnK1rDt8rmDsXTFc7AHE
Every reader ran its own regex over every page, so adding readers made builds slower. One pattern with all selected readers scans a page once, which is 8x faster than the previous 8 readers and no longer depends on how many readers are selected. Tables are now inserted with a replacement function, so a value like 'C:\1 path' is inserted as it is. It used to be expanded as a regex replacement, which silently changed such values. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCJnK1rDt8rmDsXTFc7AHE
timvink
force-pushed
the
42_more_pandas_readers
branch
from
September 14, 2026 10:47
3a256ec to
aff4f5b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #42
New readers
read_parquet,read_orc,read_html,read_xml,read_hdf,read_sas,read_spssandread_stata, each with a matchingpd_read_*macro. That completes the list in #42.Two formats need a bit of handling of their own:
pd.read_html()returns every table it finds, soread_html()inserts the first one. Usematchto select another.pd.read_hdf()returns apd.Serieswhen a Series was stored, which is converted to a frame.Refactor
Each reader used to be a copy of the same two functions. They are now built from a
LOADERSregistry bymarkdown_reader()anddataframe_reader(), so a new format is one line. Adding a reader toselect_readers, the JSON schema and the macros happens automatically.get_keywords()and friends now accept more than one function, because a reader can take keyword arguments from several:read_yaml()has its ownencodingon top of the arguments ofpd.json_normalize(). Everything a reader does not accept is still passed on to.to_markdown().Dependencies
pandas>=1.3instead of>=1.1, becausepd.read_xml()was added in 1.3.lxmlas a dev dependency, needed bypd.read_html()andpd.read_xml(). It is the only change inuv.lock.Build times
Adding readers used to make builds slower, because every reader ran its own regex over every page. All selected readers are now searched for in one pattern, so a page is scanned once no matter how many readers there are. On a 15KB page:
That is about a second saved per 1000 pages compared to master, and
select_readersis no longer needed to keep builds fast.Tables are inserted with a replacement function now, instead of as a regex replacement string. A value like
C:\1 pathused to be expanded as a backreference and silently becameC: path; it is now inserted as it is. The same goes for a value likehi\nthere, which used to break the table across two rows, astests/fixtures/basic_setup/docs/page_carriage_return.mdshows.Tests
tests/fixtures/pandas_readerscovers parquet, orc, stata, sas and xml (with the standard library parser),pandas_readers_htmlcoversread_html()andread_xml()with the default parser.The
read_spssandread_hdftests are skipped unlesspyreadstatandtablesare installed, which they are not in CI. Both pass locally withuv run --with pyreadstat --with tables pytest. I left those two out of the dev dependencies because they are heavy and have patchy wheel coverage on python 3.8.Documented in
docs/readers.md, including which readers need an extra package installed.read_spssandread_hdfhave no live example there for the same reason.🤖 Generated with Claude Code
https://claude.ai/code/session_01JCJnK1rDt8rmDsXTFc7AHE