Repository navigation
Parallel sheet reads from one reader, positional sources and reader lifetime (v5.2.0) - #135
Merged
Merged
Conversation
… empty rows in readers
…seekable streams Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Source Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o the central directory Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…yteSource Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…e from the read path Excel.cs detects and dispatches with a ZipIndex; the Reopen* helpers, ZipEntryBytes, ZipReaderOpen and the ZipArchive helpers in WorkbookLookups are gone. Tests: deleted ZipIndexTests.ZipEntryBytesReadThrowsInvalidDataWhenEntryUnderDelivers (sole subject was ZipEntryBytes); rewrote the ReadViaZipArchive helper in ZipIndexTests to read the entry through ZipArchive directly. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…g a shared stream Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…P index ZIP64 fixtures now also run through a stream source, sync and async. The mutation differential test covers sync and async part and entry-stream reads. Entry streams are compared with the memory entry stream, since streams never verify declared sizes on any source. Reader entry points are checked for disposing the stream on failure, and the "less data than declared" guard is covered again. No product defect found. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…exact stream types Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
DecompressedByteCounter.Add and Remaining are now thread-safe via Interlocked.Add and Interlocked.Read, enabling concurrent sheet inflation under a shared decompression limit. Once any thread exceeds the limit, all subsequent Add calls throw—captured atomically in _total regardless of order. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Replace lazy-initialization with thread-safe compare-and-swap in Cell.GetString() and each reader's SharedStringCache property. Materializes cache array once per reader; filled slots never change, ensuring enumerators on different threads see consistent string references and cached values. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
OnceGate runs a load once for multiple callers. A failure is recorded and rethrown without reloading; cancellation runs reset and lets the next caller load fresh. Sync and async paths share one load. ReaderLifetime is a reference-counted lifetime tracker. The reader holds one reference, each enumerator another. Release runs when the last reference goes. Close is idempotent and prevents new acquisitions once called. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…or its enumerators Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…e offsets everywhere ByteSourceStream and ZipIndex throw ObjectDisposedException after Dispose; every ByteSource rejects a negative offset with ArgumentOutOfRangeException. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…uctor throws The constructor reads the header row before it returns. A throw there (a limit, a corrupt sheet) dropped the enumerator it had just taken, and that enumerator's lease kept the reader's source open after the caller disposed the reader. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sources Enumerators created on one thread from a single XLSX, XLSB or XLS reader (memory, stream and path sources) are drained on separate threads, sync and async, and must match a sequential read. Threads reading one sheet must get the very same shared-string instances (equal strings for XLS labels). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispose left the deferred-open flag set, so a MoveNext after Dispose opened a new sheet stream and buffers with no lease. Clear the flag in Dispose and DisposeAsync, and mark the cursor at end of data once its buffers are returned so a use after dispose reports no rows instead of waiting for a source that never comes. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…numerators CsvReader over a seekable stream wraps it in a ByteSource and gives every enumerator its own positional view, so enumerators no longer share or rewind one Stream and can run concurrently. The reader holds a ReaderLifetime: Dispose closes it to new enumerators and its file or stream is released when the last enumerator is disposed. A non-seekable stream serves a single enumerator. Rewrites the thread-safety remark of IExcelRowReader accordingly. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A FileStream opened with a non-UTF-8 encoding would hand parallel chunking the raw, untranscoded file. The only caller already filters on encoding, so this was latent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The positional stream buffered 64K of its own under a consumer that already buffers, copying every byte twice. It now takes an unbuffered mode, used for CSV when no transcoding stream sits in between. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #135 +/- ##
==========================================
+ Coverage 89.45% 89.88% +0.42%
==========================================
Files 181 185 +4
Lines 13612 13911 +299
Branches 2515 2550 +35
==========================================
+ Hits 12177 12504 +327
+ Misses 956 941 -15
+ Partials 479 466 -13 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Benchmark ResultsBaseline: master Moved by at least 10%
77 within ±10%. Full results per groupExcelReader.Benchmarks.ArrowConversionBenchmark
ExcelReader.Benchmarks.ChunkedParseBenchmark
ExcelReader.Benchmarks.ColdStartBenchmark
ExcelReader.Benchmarks.CsvParallelParseBenchmark
ExcelReader.Benchmarks.CsvParallelVsSepBenchmark
ExcelReader.Benchmarks.CsvParseBenchmark
ExcelReader.Benchmarks.CsvReadBenchmark
ExcelReader.Benchmarks.CsvWriteBenchmark
ExcelReader.Benchmarks.DataReaderBenchmark
ExcelReader.Benchmarks.EncryptedWorkbookBenchmark
ExcelReader.Benchmarks.NativeRowReadBenchmark
ExcelReader.Benchmarks.NativeTypedParseBenchmark
ExcelReader.Benchmarks.ParseBenchmark
ExcelReader.Benchmarks.ReadBenchmark
ExcelReader.Benchmarks.RealDataReadBenchmark
ExcelReader.Benchmarks.RealDataTypedParseBenchmark
ExcelReader.Benchmarks.RecordWriteBenchmark
ExcelReader.Benchmarks.StringHeavyReadBenchmark
ExcelReader.Benchmarks.WriteBenchmark
ExcelReader.Benchmarks.WritePathBenchmark
ExcelReader.Benchmarks.XlsReadBenchmark
ExcelReader.Benchmarks.XlsWriteBenchmark
ExcelReader.Benchmarks.XlsxSharedStringHotPathBenchmark
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this change?
Enumerators obtained from one reader can now be consumed on different threads, for XLSX, XLSB, XLS
and CSV, and a reader's file or stream stays open until its last enumerator is disposed. To get
there, every source is read positionally and XLSX/XLSB stop using
ZipArchiveon the read path.The public API is unchanged.
ByteSource(memory, file handle, seekable stream) andByteSourceStream.ZipIndex(formerlyZipMemoryIndex) reads its directory and entries throughany
ByteSource, so XLSX and XLSB from a path or a stream no longer go throughSystem.IO.Compression.ZipArchive. XLS loads its sectors the same way. A non-seekable stream isbuffered to memory at open for XLSX, XLSB and XLS.
are published with compare-and-swap, and the XLSX shared-string table is loaded once behind a gate
that records a failure and undoes a cancelled load.
reader and its live enumerators.
run at once. A non-seekable stream is still one streaming pass and is not buffered.
WriteRecordBatchaccepts more Arrow types; XLSB and XLS yieldrows written without cells, as XLSX does; the CSV structural scanner uses PMULL on ARM.
Behaviour changes
DisposethrowsObjectDisposedException.An enumerator that is never disposed keeps the file open.
strings on the second attempt.
ZipIndexaccepts trailing bytes after the end-of-central-directory record and rejects storedentries whose two sizes disagree.
Not in this PR
The v6 API (
IExcelWorkbook, sheet values, removal ofIExcelRowReader). "Current sheet" andMoveToSheetare still unsynchronized: select sheets and create enumerators from one thread.Checklist
dotnet build ExcelReader.slnx --configuration Releasebuilds clean (warnings are errors)dotnet test tests/ExcelReader.Tests/ExcelReader.Tests.csproj --configuration ReleasepassesPublicAPI.Unshipped.txtupdated (no public API change; the file is empty)Test plan
ByteSourceandByteSourceStreamper source kind;ZipIndexover positional sourcesincluding ZIP64 and failure paths; lifetime per format (enumerator outlives its reader, double
dispose, failed open releases the reader); concurrent sheets across XLSX/XLSB/XLS and
memory/stream/path, sync and async (27 cases, five consecutive runs); CSV with concurrent
enumerators per source kind.
check <seeds> 512with seeds 1 and 7, 349,866 cases, 0 failures.1.5% on both lowest and median (XLSX and XLSB by path,
FileStreamand memory; CSV by path,FileStreamand memory).XLSX 59 ms vs 65 ms, XLSB 67 ms vs 88 ms; sequential 225 ms and 206 ms.
CI covers them), and a long fuzz run on
ZipIndex.