Skip to content

7.0: parallel XLSX/XLSB reading, Excel tables, own byte sources, managed inflate, and a public API cleanup - #140

Merged
GabrielMarquezMatte merged 121 commits into
masterfrom
develop
Oct 12, 2026
Merged

GabrielMarquezMatte merged 121 commits into
masterfrom
develop

Conversation

@GabrielMarquezMatte

@GabrielMarquezMatte GabrielMarquezMatte commented Oct 8, 2026 •

Copy link
Copy Markdown
Owner

What does this change?

Everything on develop since v6.0.0, as the candidate for 7.0.0. It started as 6.1 (own byte
sources, managed inflate, native column buffers) and became a major release once parallel reading
replaced CsvParallel and the public API was reworked so that new settings can be added without
another break. The full v6 → v7 port is in docs/guide/migrating-to-v7.md.

The C ABI keeps every signature and XL_ABI_VERSION stays 6; the native behavior changes are listed below.

Parallel sheet reading (SheetParallel, breaking)

  • SheetParallel replaces CsvParallel: one static class with ParseAsync, ForEachAsync and
    AggregateAsync over any IExcelSheet. You open the workbook and pass the sheet, and the library
    parallelises as far as the format allows. The contracts lost their Csv prefix (IRowRecord,
    IRowAccumulator, RowAggregation, RowAction).
  • XLSX and XLSB now read on several threads. One thread inflates and cuts row-aligned 256 KB chunks,
    workers parse them in place, and an ordered merge yields rows in sheet order. When an XLSX chunk ends
    mid-markup, that part is read again sequentially, so the results always match a sequential read.
    XLS, table views and small sheets stay sequential.
  • Sheet stage on the 65K corpus (Ryzen 7 5700X, JIT harness minimum): XLSX 17.6 ms, XLSB 13.3 ms.
    NativeAOT xl_parse_typed, parallel vs sequential: XLSX 20.4 vs 53.4 ms, XLSB 15.2 vs 27.9 ms.
  • Native: degree_of_parallelism = 0 on xl_parse_typed and the Arrow exports now parallelises XLSX and
    XLSB as well as CSV. The bindings need no code change, but peak memory is higher because partitions are
    held until they merge.

Excel tables

  • Read: IExcelWorkbook.Tables and TryGetTable on every format, and ExcelTable.AsSheet() to read a
    table's rows through the normal sheet API, with column indexes relative to the table. Name binding on a
    headerless table is rejected with InvalidOperationException.
  • Write: BeginTable/EndTable on the XLSX and XLSB sheet writers (ITableSheetWriter<TRow>),
    ExcelTableOptions, and WriteTableAsync to write a record layout as a table.

Open a workbook from your own bytes

  • Core: any byte source can be read through a seekable view that fetches it in blocks, each block once.
  • C ABI: xl_open_source (random access through read_at), xl_open_stream (forward-only) and
    xl_set_source_error. xl_open_options gains source and buffering fields and still accepts the 6.0
    layout through struct_size.
  • Bindings: Python takes a Source or any readable stream, C++ an xl::Source or xl::InputStream,
    and Rust a Source or any Read.
  • Schema inference and replay work on a CSV stream that cannot seek. The Python Arrow stream releases the
    GIL during native callbacks.

Managed inflate decoder

DeflateStream is replaced in ZipIndex by InflateStream, a managed, resumable port of libdeflate's
design (the MIT notice is in THIRD-PARTY-NOTICES.md, which now ships in the package). Inflate alone is
about 1.7x faster on XLSX and 1.4x on XLSB. It has a canary after the output window, exact truncation
detection, and an inflate fuzz target that compares its output with DeflateStream.

Typed parse builds columns in native memory

ColumnBuilder writes into a native block that grows by doubling and hands it over without a copy.
Peak memory in to_record_batch on the 65K fixture fell from 26.0 to 18.1 MiB (whole sheet) and from
16.7 to 10.7 MiB (streamed). Typed parsing is 3–5% faster.

Public API changes (breaking)

v6 v7
CsvParallel and its contracts SheetParallel, IRowRecord, IRowAccumulator, RowAggregation, RowAction
ExcelParserConfig ExcelParserOptions
Converters and cell readers take (bool isDate1904, IFormatProvider provider) They take in ExcelRowContext
InferSchema(sheet, headerRow, sampleSize, parseText) returning an array InferSchema(sheet, SchemaInferenceOptions) returning IReadOnlyList<ExcelColumnSchema>
ExcelDataReader(sheet, headerRow) SheetDataReader(sheet, SheetDataReaderOptions) (the old name collided with Sylvan.Data.Excel and the ExcelDataReader package)
ToArrowRecordBatch(schema, headerRow), WriteRecordBatch(batch, sheetName, writeHeader) ArrowReadOptions, ArrowWriteOptions
XlsWorkbookWriter.Create(stream, leaveOpen, date1904) XlsWriterOptions; XlsxWriterOptions also gains Date1904
Excel.FromCsv*(…, CsvReaderOptions) Excel.FromCsv*(…, ExcelReaderOptions), with the dialect in .Csv
CsvReaderOptions.MaxCellBytes, .InternStrings On ExcelReaderOptions only
Cell.TryGetDateTime(out DateTime) (silently assumed 1900) Removed; pass isDate1904
CellType.Formula Removed; a formula cell reports its cached result's type

Behavior changes worth a release-note line

  • A ZIP entry that is cut short or corrupt throws InvalidDataException. Before, a sheet could silently
    end early (one damaged workbook stopped at 298 of 30,000 rows). Both the managed inflate and the new
    declared-size check on streamed entries catch it. CRC32 is still not verified.
  • XLSX text now decodes OOXML _xHHHH_ escapes (Excel writes _x000D_ for CR). Before, text written by
    this library's own writer did not read back unchanged.
  • Formula cells are typed by their cached result in every format. Before, an XLS formula with a date
    style lost its Date type. XL_CELL_FORMULA stays defined but is no longer produced.
  • Excel.Open with ExcelFileFormat.Csv now applies MaxCellBytes and InternStrings.
  • The CLI convert command no longer shifts dates by four years when converting a 1904 workbook.
  • xl_write_options.date1904 now applies to XLSX.

Other fixes

  • XLSX rows whose opening tag straddled an async buffer refill were dropped.
  • The generator sets required members in the factory it emits.
  • Table writer fixes: column names are escaped like header cells, a header-only table keeps its reserved
    row out of the range, and stray AutoFilter and sort state are no longer written to XLSB.
  • HexVal accepted bytes below '0' and 'a' as hex digits, which also affected &#x…; entities.
  • The sheet pipeline surfaces the source stream's own cancellation and reports a parse failure at the
    right sheet row after recovery.

Also

  • The static PGO profile was regenerated after each signature change.
  • Project icon on the NuGet packages and the binding READMEs.
  • Dependabot bumps.

Checklist

  • dotnet build ExcelReader.slnx --configuration Release builds clean (warnings are errors)
  • dotnet test tests/ExcelReader.Tests/ExcelReader.Tests.csproj --configuration Release passes (full suite)
  • Public API changes recorded in PublicAPI.Shipped.txt/PublicAPI.Unshipped.txt
  • Tests added/updated for every behavior change
  • Migration guide (docs/guide/migrating-to-v7.md)
  • One focused change: no, this is the release PR from develop

Test plan

  • Affected suites pass with 0 failures: Reader 1,767, Writer 508, Native 395, Parser 359, Arrow 27,
    CLI 26.
  • Sheet pipeline: parity tests against the sequential read for every format, plus a fuzz target comparing
    the two.
  • Inflate: rejection-path unit tests, differential tests against DeflateStream, and fuzz check
    (156,780 cases, 0 failures).
  • Corrupt-entry and _xHHHH_ tests were written failing first; the escape decoder passed 25,186 fuzz
    cases.

Before tagging 7.0.0:

  • BenchmarkDotNet rerun on an idle Ryzen 7 5700X. The parallel tables in benchmarks.md were taken
    under load and are marked pending.
  • Promote PublicAPI.Unshipped.txt into Shipped.
  • Python, Rust and C++ suites against the republished NativeAOT library.
  • A few CI fuzz cycles on the pipeline and inflate targets.

GabrielMarquezMatte and others added 30 commits October 7, 2026 13:41
…-api-v6.0.0

chore: promote public API for v6.0.0
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… 6.0 options layout

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ard stream reads

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Python callbacks now catch BaseException so Ctrl-C fails the read instead of
truncating a CSV stream. C++ open_source rejects a size above INT64_MAX before
releasing the source, and the C++ tests cover destruction on failure, parallel
reads and stream errors. The smoke test pins the 88-byte 6.0 options struct.
Source error wording no longer claims 0 is valid, StreamBuffer clamps its limit
to Array.MaxLength, and the header, docstring and READMEs are tightened.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bumps Meziantou.Analyzer from 3.0.290 to 3.0.294

---
updated-dependencies:
- dependency-name: Meziantou.Analyzer
  dependency-version: 3.0.294
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: analyzers
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps Microsoft.Testing.Extensions.CodeCoverage from 18.11.2 to 18.12.0
Bumps Microsoft.Testing.Extensions.TrxReport from 2.4.1 to 2.5.1

---
updated-dependencies:
- dependency-name: Microsoft.Testing.Extensions.CodeCoverage
  dependency-version: 18.12.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: test
- dependency-name: Microsoft.Testing.Extensions.TrxReport
  dependency-version: 2.5.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: test
...

Signed-off-by: dependabot[bot] <support@github.com>
…elop/analyzers-55e46a7a66

Bump the analyzers group with 1 update
…ts/ExcelReader.Tests/develop/test-4352e14058

Bump the test group with 2 updates
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replaces DeflateStream at all four sites in ZipIndex. A deflate entry that ends before its final block now throws InvalidDataException; DeflateStream returned the bytes decoded so far and reported end of stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GabrielMarquezMatte added 5 commits October 11, 2026 01:42
…ema>

Matches ArrowReadOptions.Schema and stops handing callers a mutable array.
…e 1900 date system

A cell does not know its workbook's date system, so the overload silently returned dates
four years early for a 1904 workbook. Callers now pass IsDate1904. The CLI convert command
had exactly that bug and now passes the sheet's date system.
ExcelDataReader and ExcelDataReaderOptions collided with Sylvan.Data.Excel's types and
with the ExcelDataReader package's namespace, so code importing either needed global::
qualification. SheetDataReaderOptions follows the same rename.
…ype.Formula

CellType.Formula meant a different thing per format: XLSX used it for a text result, XLS
for a numeric result (dropping Date for a date-styled formula), and XLSB never used it.
Every format now reports the result's own type. CellType.Error keeps its value 6, and the
native XL_CELL_FORMULA constant stays defined but is no longer produced.
@GabrielMarquezMatte GabrielMarquezMatte changed the title 6.1: open workbooks from your own sources and streams, managed inflate, leaner typed parse 7.0: parallel XLSX/XLSB reading, Excel tables, own byte sources, managed inflate, and a public API cleanup Oct 11, 2026
GabrielMarquezMatte and others added 22 commits October 11, 2026 02:23
The batch parse got fast enough (about 40 ms for a million CSV rows on CI) to fall under the
fixed 50 ms floor, which only exists so a gap is long compared with a thread switch. The test
now drops the switch interval to 1 ms for the parse and requires 20 ms, still twenty switch
intervals.
Loads the offset entry before deciding whether to refill, with libdeflate's thresholds (31 bits, 38 through a subtable), so at least 11 bits remain after the offset. The next litlen entry is then loaded from those bits before the refill instead of waiting on it. Inflate only, Ryzen 7 5700X, 65K-row sheets: XLSX 15.0 -> 12.7 ms, XLSB 9.5 -> 8.6 ms under the JIT; 14.9 -> 13.5 and 9.5 -> 8.9 ms under NativeAOT. Native libdeflate on the same streams is 13.5 and 8.8 ms.
…t-loop change

The fast-loop change and the v7 renames left ILC reporting "PGO data available, but IL did not match" for FastLoop. Recorded with NativePgoTrainer as the Native README describes, XLSB methods still excluded. AOT xl_parse_typed, 65K fixtures, min of three alternating runs: XLSX dop 0 19.1 -> 17.8 ms, dop 1 52.2 -> 51.0 ms; XLSB dop 0 14.8 -> 14.3 ms, dop 1 27.8 -> 27.1 ms; 160 MB CSV dop 1 396 -> 385 ms. CSV, XLS and the parallel 160 MB CSV are flat.
docs.rs builds offline, so build.rs failed downloading the release asset
("curl: (6) Could not resolve host: github.com") and excelreader 6.0.0 has no
documentation there. rustdoc never links, so the build script now returns early
when DOCS_RS is set.

Also drops two explicit link targets rustdoc flags as redundant.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… examples

lib.rs includes README.md as the crate-level docs, so docs.rs opens on it and
cargo test compiles every example. The examples gained hidden setup lines and
no_run, since they name files that do not exist; the HttpSource one stays
ignore because the crate has no reqwest dependency. The FFI note that used to
be the crate doc is now a plain comment.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Microsoft.CodeAnalysis.Analyzers already adds AnalyzerReleases.Shipped.md and
AnalyzerReleases.Unshipped.md as AdditionalFiles. The explicit items duplicated
them, which docfx reports as duplicate source files.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Inheriting the whole BeginTable comment and restating its params made docfx see
every param twice. CS1573 still needs the params once ct is documented, so the
inheritdoc is narrowed to /summary and /exception instead.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The heading became "Generate typed maps at compile time (Native AOT /
trimming)" and the link still pointed at the old slug.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
docfx.json builds a site from the Core and Arrow XML docs, the guides, the
performance documents and the binding READMEs, with the root README as the
landing page. The config sits at the repo root so the README's links
(docs/guide/...) resolve unchanged: docfx does not match a link that climbs out
of its own folder. docs.yml publishes the site to gh-pages next to the
benchmark dashboard.

The template sizes the navbar logo, hides the logo the binding READMEs open
with, and strips rustdoc's hidden lines from Rust examples. ARCHITECTURE,
STYLEGUIDE and docs/README link docs/README.md or #index instead of a folder,
which docfx cannot link.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
One table from each kind of failure to the C# exception, the XL_* status, and
the Python, Rust and C++ error types, plus what each status code means.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
python/scripts/api_markdown.py writes a Markdown page per public class, a
functions page and a toc.yml from excelreader's signatures and docstrings, so
the reference renders with the site's own layout, navigation and search. Types
are linked to their pages in parameter, return and field tables, and every
generated signature is checked with ast.parse. Importing excelreader loads the
native library lazily, so CI needs only the sources.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
cpp/scripts/api_markdown.py runs Doxygen over excelreader.hpp and
excelreader_arrow.hpp and turns its XML into Markdown pages for docfx: classes,
structs, enums, concepts and free functions, with @PARAM, @tparam and @return in
their own tables and library types linked. xl::detail, iterator internals and
copy/move members are left out.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Public types and members get @brief, @PARAM, @tparam and @return comments that
state the contract: ownership, how long a view stays valid, threading, how the
end of data is reported, and defaults. The OpenOptions defaults come from the
managed options the native layer forwards each field to.

Also restores, as one line inside write_sheet and write_sheet_to_memory, the
invariant a truncated comment used to state: refs borrows builders' buffers, so
both must live until the write returns.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The C++ generator runs Doxygen with WARN_NO_PARAMDOC, WARN_IF_INCOMPLETE_DOC and
FAIL_ON_WARNINGS, so an @PARAM that no longer matches its signature fails the
build. The workflow also runs on pull requests, building without publishing,
and uses a per-ref concurrency group so a pull request run cannot cancel a
publish.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Updated error handling section to include cross-references for exceptions.
- Migrating guide updated to include cross-references for new types and methods.
- Migration to v7 guide improved with cross-references for new options and types.
- Parsing guide enhanced with cross-references for interfaces and methods.
- Reading guide updated to include cross-references for workbook and sheet types.
- Writing guide improved with cross-references for options and interfaces.
- Introduced new snippets for reading and writing CSV files, including options for custom delimiters and encoding.
- Added examples for handling encrypted Excel files, including opening, verifying integrity, and writing encrypted workbooks.
- Created a project file for snippets to ensure they compile against the current API.
- Implemented global using directives for cleaner snippet code.
- Added comprehensive examples for parsing Excel files with attributes, fluent interfaces, and converters.
- Included parallel processing examples for reading large Excel files efficiently.
- Enhanced documentation with detailed snippets for various writing scenarios, including styles, shared strings, and table writing.
- Developed a synchronization script to keep snippets in sync with documentation.
@GabrielMarquezMatte
GabrielMarquezMatte merged commit bcc628f into master Oct 12, 2026
35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants