Repository navigation
7.0: parallel XLSX/XLSB reading, Excel tables, own byte sources, managed inflate, and a public API cleanup - #140
Merged
Merged
Conversation
…-api-v6.0.0 chore: promote public API for v6.0.0
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… 6.0 options layout Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ard stream reads Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Python callbacks now catch BaseException so Ctrl-C fails the read instead of truncating a CSV stream. C++ open_source rejects a size above INT64_MAX before releasing the source, and the C++ tests cover destruction on failure, parallel reads and stream errors. The smoke test pins the 88-byte 6.0 options struct. Source error wording no longer claims 0 is valid, StreamBuffer clamps its limit to Array.MaxLength, and the header, docstring and READMEs are tightened. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lay functionality
Bumps Meziantou.Analyzer from 3.0.290 to 3.0.294 --- updated-dependencies: - dependency-name: Meziantou.Analyzer dependency-version: 3.0.294 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: analyzers ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps Microsoft.Testing.Extensions.CodeCoverage from 18.11.2 to 18.12.0 Bumps Microsoft.Testing.Extensions.TrxReport from 2.4.1 to 2.5.1 --- updated-dependencies: - dependency-name: Microsoft.Testing.Extensions.CodeCoverage dependency-version: 18.12.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: test - dependency-name: Microsoft.Testing.Extensions.TrxReport dependency-version: 2.5.1 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: test ... Signed-off-by: dependabot[bot] <support@github.com>
…elop/analyzers-55e46a7a66 Bump the analyzers group with 1 update
…ts/ExcelReader.Tests/develop/test-4352e14058 Bump the test group with 2 updates
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replaces DeflateStream at all four sites in ZipIndex. A deflate entry that ends before its final block now throws InvalidDataException; DeflateStream returned the bytes decoded so far and reported end of stream. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
added 5 commits
October 11, 2026 01:42
…ema> Matches ArrowReadOptions.Schema and stops handing callers a mutable array.
…e 1900 date system A cell does not know its workbook's date system, so the overload silently returned dates four years early for a 1904 workbook. Callers now pass IsDate1904. The CLI convert command had exactly that bug and now passes the sheet's date system.
ExcelDataReader and ExcelDataReaderOptions collided with Sylvan.Data.Excel's types and with the ExcelDataReader package's namespace, so code importing either needed global:: qualification. SheetDataReaderOptions follows the same rename.
…ype.Formula CellType.Formula meant a different thing per format: XLSX used it for a text result, XLS for a numeric result (dropping Date for a date-styled formula), and XLSB never used it. Every format now reports the result's own type. CellType.Error keeps its value 6, and the native XL_CELL_FORMULA constant stays defined but is no longer produced.
The batch parse got fast enough (about 40 ms for a million CSV rows on CI) to fall under the fixed 50 ms floor, which only exists so a gap is long compared with a thread switch. The test now drops the switch interval to 1 ms for the parse and requires 20 ms, still twenty switch intervals.
Loads the offset entry before deciding whether to refill, with libdeflate's thresholds (31 bits, 38 through a subtable), so at least 11 bits remain after the offset. The next litlen entry is then loaded from those bits before the refill instead of waiting on it. Inflate only, Ryzen 7 5700X, 65K-row sheets: XLSX 15.0 -> 12.7 ms, XLSB 9.5 -> 8.6 ms under the JIT; 14.9 -> 13.5 and 9.5 -> 8.9 ms under NativeAOT. Native libdeflate on the same streams is 13.5 and 8.8 ms.
…t-loop change The fast-loop change and the v7 renames left ILC reporting "PGO data available, but IL did not match" for FastLoop. Recorded with NativePgoTrainer as the Native README describes, XLSB methods still excluded. AOT xl_parse_typed, 65K fixtures, min of three alternating runs: XLSX dop 0 19.1 -> 17.8 ms, dop 1 52.2 -> 51.0 ms; XLSB dop 0 14.8 -> 14.3 ms, dop 1 27.8 -> 27.1 ms; 160 MB CSV dop 1 396 -> 385 ms. CSV, XLS and the parallel 160 MB CSV are flat.
docs.rs builds offline, so build.rs failed downloading the release asset
("curl: (6) Could not resolve host: github.com") and excelreader 6.0.0 has no
documentation there. rustdoc never links, so the build script now returns early
when DOCS_RS is set.
Also drops two explicit link targets rustdoc flags as redundant.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… examples lib.rs includes README.md as the crate-level docs, so docs.rs opens on it and cargo test compiles every example. The examples gained hidden setup lines and no_run, since they name files that do not exist; the HttpSource one stays ignore because the crate has no reqwest dependency. The FFI note that used to be the crate doc is now a plain comment. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Microsoft.CodeAnalysis.Analyzers already adds AnalyzerReleases.Shipped.md and AnalyzerReleases.Unshipped.md as AdditionalFiles. The explicit items duplicated them, which docfx reports as duplicate source files. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Inheriting the whole BeginTable comment and restating its params made docfx see every param twice. CS1573 still needs the params once ct is documented, so the inheritdoc is narrowed to /summary and /exception instead. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The heading became "Generate typed maps at compile time (Native AOT / trimming)" and the link still pointed at the old slug. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
docfx.json builds a site from the Core and Arrow XML docs, the guides, the performance documents and the binding READMEs, with the root README as the landing page. The config sits at the repo root so the README's links (docs/guide/...) resolve unchanged: docfx does not match a link that climbs out of its own folder. docs.yml publishes the site to gh-pages next to the benchmark dashboard. The template sizes the navbar logo, hides the logo the binding READMEs open with, and strips rustdoc's hidden lines from Rust examples. ARCHITECTURE, STYLEGUIDE and docs/README link docs/README.md or #index instead of a folder, which docfx cannot link. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
One table from each kind of failure to the C# exception, the XL_* status, and the Python, Rust and C++ error types, plus what each status code means. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
python/scripts/api_markdown.py writes a Markdown page per public class, a functions page and a toc.yml from excelreader's signatures and docstrings, so the reference renders with the site's own layout, navigation and search. Types are linked to their pages in parameter, return and field tables, and every generated signature is checked with ast.parse. Importing excelreader loads the native library lazily, so CI needs only the sources. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
cpp/scripts/api_markdown.py runs Doxygen over excelreader.hpp and excelreader_arrow.hpp and turns its XML into Markdown pages for docfx: classes, structs, enums, concepts and free functions, with @PARAM, @tparam and @return in their own tables and library types linked. xl::detail, iterator internals and copy/move members are left out. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Public types and members get @brief, @PARAM, @tparam and @return comments that state the contract: ownership, how long a view stays valid, threading, how the end of data is reported, and defaults. The OpenOptions defaults come from the managed options the native layer forwards each field to. Also restores, as one line inside write_sheet and write_sheet_to_memory, the invariant a truncated comment used to state: refs borrows builders' buffers, so both must live until the write returns. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The C++ generator runs Doxygen with WARN_NO_PARAMDOC, WARN_IF_INCOMPLETE_DOC and FAIL_ON_WARNINGS, so an @PARAM that no longer matches its signature fails the build. The workflow also runs on pull requests, building without publishing, and uses a per-ref concurrency group so a pull request run cannot cancel a publish. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Updated error handling section to include cross-references for exceptions. - Migrating guide updated to include cross-references for new types and methods. - Migration to v7 guide improved with cross-references for new options and types. - Parsing guide enhanced with cross-references for interfaces and methods. - Reading guide updated to include cross-references for workbook and sheet types. - Writing guide improved with cross-references for options and interfaces.
- Introduced new snippets for reading and writing CSV files, including options for custom delimiters and encoding. - Added examples for handling encrypted Excel files, including opening, verifying integrity, and writing encrypted workbooks. - Created a project file for snippets to ensure they compile against the current API. - Implemented global using directives for cleaner snippet code. - Added comprehensive examples for parsing Excel files with attributes, fluent interfaces, and converters. - Included parallel processing examples for reading large Excel files efficiently. - Enhanced documentation with detailed snippets for various writing scenarios, including styles, shared strings, and table writing. - Developed a synchronization script to keep snippets in sync with documentation.
…g and update related code comments
…d native bindings
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this change?
Everything on
developsincev6.0.0, as the candidate for 7.0.0. It started as 6.1 (own bytesources, managed inflate, native column buffers) and became a major release once parallel reading
replaced
CsvParalleland the public API was reworked so that new settings can be added withoutanother break. The full v6 → v7 port is in
docs/guide/migrating-to-v7.md.The C ABI keeps every signature and
XL_ABI_VERSIONstays 6; the native behavior changes are listed below.Parallel sheet reading (
SheetParallel, breaking)SheetParallelreplacesCsvParallel: one static class withParseAsync,ForEachAsyncandAggregateAsyncover anyIExcelSheet. You open the workbook and pass the sheet, and the libraryparallelises as far as the format allows. The contracts lost their
Csvprefix (IRowRecord,IRowAccumulator,RowAggregation,RowAction).workers parse them in place, and an ordered merge yields rows in sheet order. When an XLSX chunk ends
mid-markup, that part is read again sequentially, so the results always match a sequential read.
XLS, table views and small sheets stay sequential.
NativeAOT
xl_parse_typed, parallel vs sequential: XLSX 20.4 vs 53.4 ms, XLSB 15.2 vs 27.9 ms.degree_of_parallelism = 0onxl_parse_typedand the Arrow exports now parallelises XLSX andXLSB as well as CSV. The bindings need no code change, but peak memory is higher because partitions are
held until they merge.
Excel tables
IExcelWorkbook.TablesandTryGetTableon every format, andExcelTable.AsSheet()to read atable's rows through the normal sheet API, with column indexes relative to the table. Name binding on a
headerless table is rejected with
InvalidOperationException.BeginTable/EndTableon the XLSX and XLSB sheet writers (ITableSheetWriter<TRow>),ExcelTableOptions, andWriteTableAsyncto write a record layout as a table.Open a workbook from your own bytes
xl_open_source(random access throughread_at),xl_open_stream(forward-only) andxl_set_source_error.xl_open_optionsgains source and buffering fields and still accepts the 6.0layout through
struct_size.Sourceor any readable stream, C++ anxl::Sourceorxl::InputStream,and Rust a
Sourceor anyRead.GIL during native callbacks.
Managed inflate decoder
DeflateStreamis replaced inZipIndexbyInflateStream, a managed, resumable port of libdeflate'sdesign (the MIT notice is in
THIRD-PARTY-NOTICES.md, which now ships in the package). Inflate alone isabout 1.7x faster on XLSX and 1.4x on XLSB. It has a canary after the output window, exact truncation
detection, and an
inflatefuzz target that compares its output withDeflateStream.Typed parse builds columns in native memory
ColumnBuilderwrites into a native block that grows by doubling and hands it over without a copy.Peak memory in
to_record_batchon the 65K fixture fell from 26.0 to 18.1 MiB (whole sheet) and from16.7 to 10.7 MiB (streamed). Typed parsing is 3–5% faster.
Public API changes (breaking)
CsvParalleland its contractsSheetParallel,IRowRecord,IRowAccumulator,RowAggregation,RowActionExcelParserConfigExcelParserOptions(bool isDate1904, IFormatProvider provider)in ExcelRowContextInferSchema(sheet, headerRow, sampleSize, parseText)returning an arrayInferSchema(sheet, SchemaInferenceOptions)returningIReadOnlyList<ExcelColumnSchema>ExcelDataReader(sheet, headerRow)SheetDataReader(sheet, SheetDataReaderOptions)(the old name collided with Sylvan.Data.Excel and the ExcelDataReader package)ToArrowRecordBatch(schema, headerRow),WriteRecordBatch(batch, sheetName, writeHeader)ArrowReadOptions,ArrowWriteOptionsXlsWorkbookWriter.Create(stream, leaveOpen, date1904)XlsWriterOptions;XlsxWriterOptionsalso gainsDate1904Excel.FromCsv*(…, CsvReaderOptions)Excel.FromCsv*(…, ExcelReaderOptions), with the dialect in.CsvCsvReaderOptions.MaxCellBytes,.InternStringsExcelReaderOptionsonlyCell.TryGetDateTime(out DateTime)(silently assumed 1900)isDate1904CellType.FormulaBehavior changes worth a release-note line
InvalidDataException. Before, a sheet could silentlyend early (one damaged workbook stopped at 298 of 30,000 rows). Both the managed inflate and the new
declared-size check on streamed entries catch it. CRC32 is still not verified.
_xHHHH_escapes (Excel writes_x000D_for CR). Before, text written bythis library's own writer did not read back unchanged.
style lost its
Datetype.XL_CELL_FORMULAstays defined but is no longer produced.Excel.OpenwithExcelFileFormat.Csvnow appliesMaxCellBytesandInternStrings.convertcommand no longer shifts dates by four years when converting a 1904 workbook.xl_write_options.date1904now applies to XLSX.Other fixes
requiredmembers in the factory it emits.row out of the range, and stray AutoFilter and sort state are no longer written to XLSB.
HexValaccepted bytes below'0'and'a'as hex digits, which also affected&#x…;entities.right sheet row after recovery.
Also
Checklist
dotnet build ExcelReader.slnx --configuration Releasebuilds clean (warnings are errors)dotnet test tests/ExcelReader.Tests/ExcelReader.Tests.csproj --configuration Releasepasses (full suite)PublicAPI.Shipped.txt/PublicAPI.Unshipped.txtdocs/guide/migrating-to-v7.md)developTest plan
CLI 26.
the two.
DeflateStream, and fuzzcheck(156,780 cases, 0 failures).
_xHHHH_tests were written failing first; the escape decoder passed 25,186 fuzzcases.
Before tagging 7.0.0:
benchmarks.mdwere takenunder load and are marked pending.
PublicAPI.Unshipped.txtintoShipped.