diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index a627640d..09653c1f 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -246,6 +246,23 @@ against a shared bug, so the fixtures in `tests/ExcelReader.Tests/data/encrypted plaintext produced by an independent implementation) remain the only correctness oracle for decryption specifically. +XLSX and XLSB packages are written by `Writer/Zip/ZipPackageWriter`, not `ZipArchive`. It writes the +layout `ZipArchive` uses in create mode (patched local headers on seekable streams, data descriptors +otherwise, Zip64 only when a field overflows). `CompressionLevel.Fastest` entries go through +`DeflateEncoder`, a fixed-Huffman deflate with a stored-block fallback, and are checksummed by +`Crc32`, which folds 128 or 256 bits at a time with PCLMULQDQ. `Optimal` and `SmallestSize` keep +zlib through `DeflateStream`. When a write faults, `Abandon` still writes a central directory for +the entries that completed, as disposing `ZipArchive` did; a failed entry write marks the whole +archive failed, so nothing more can be added to it. + +`WriteRecordsParallelAsync` and `WriteTableParallelAsync` (`Writer/Parallel/`) cut a record source into +batches. Each batch is formatted on a pool thread by a detached copy of the sheet writer, which has +its own buffer and never touches the ZIP, then compressed into a byte-aligned deflate segment that +refers to nothing outside itself. The calling flow appends segments in order through +`ZipEntryStream.AppendSegmentAsync`, which merges CRCs with `Crc32.Combine`. Once an entry has +received a segment, everything written to it is a segment, including later serial rows, so no +compressor ever carries history across one. + ## The `excelreader` CLI `src/ExcelReader.Cli/` is a thin `dotnet tool` shell (`excelreader`) over Core's public API — it diff --git a/README.md b/README.md index 1d410915..9e48640f 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,7 @@ foreach (Row row in workbook.FirstSheet) - [Reading rows](docs/guide/reading.md) — opening a workbook, enumerating rows, auto-detecting the format, async reads, prefetch decompression, and the ADO.NET `IDataReader` bridge. - [Typed parsing](docs/guide/parsing.md) — binding rows to your own types with attributes, the compile-time generator (Native AOT / trimming), the fluent API, required columns, custom converters, and the zero-copy `ref struct` path. -- [Writing workbooks](docs/guide/writing.md) — XLSX, XLSB (BIFF12) and XLS (BIFF8) writers, cell styles, typed records, and prefetch compression. +- [Writing workbooks](docs/guide/writing.md) — XLSX, XLSB (BIFF12) and XLS (BIFF8) writers, cell styles, typed records, prefetch compression, and parallel record writes. - [CSV](docs/guide/csv.md) — reading, dialect sniffing, parallel parsing, and writing delimited text. - [Encrypted workbooks](docs/guide/encryption.md) — opening password-protected packages and encrypting written ones. - [Reading untrusted files](docs/guide/untrusted-input.md) — the resource limits that bound what one hostile file can cost, and how to reject one cleanly. @@ -85,8 +85,10 @@ errors to stderr, so `convert` is safe to pipe. - String conversion allocates only when you call `GetString()`. - The XLSX scanner accepts the SpreadsheetML shapes commonly emitted by non-Excel producers, including single-quoted attributes, comments in `sheetData`, and CDATA text runs. - Readers bound untrusted input by default: 512 MB total decompressed ZIP data, 32 MB per cell/row value buffer, and 128 MB for shared strings. Pass `ExcelReaderOptions` to the `Excel.From*`/`Excel.Open*` factories to tune these limits; set a limit to `0` to opt out and restore unlimited behavior for that limit. `CsvReader` has its own `CsvReaderOptions.MaxCellBytes` (default 32 MB) for the same purpose. +- Each ZIP entry of an XLSX or XLSB package is checked against its CRC-32 as it is read, so a damaged file throws `InvalidDataException` instead of returning wrong cells. `ExcelReaderOptions.VerifyZipChecksums = false` turns the check off. - The XLSX writer emits a compact workbook with strings, numbers, booleans, dates, and blank cells; shared strings are opt-in. - The XLSB writer emits BIFF12 workbook parts inside the standard XLSB ZIP package; shared strings are opt-in. +- `WriteRecordsParallelAsync` and `WriteTableParallelAsync` format and compress batches of records on several cores and write them in order, for XLSX and XLSB. On a Ryzen 7 5700X a 1,000,000-row sheet writes in 50 ms as XLSX with 4 batches in flight and 36 ms with 8, against 187 ms serially; XLSB takes 41 ms and 28 ms against 124 ms. Memory stays at about 3 MB per batch in flight. See [Write records in parallel](docs/guide/writing.md#write-records-in-parallel-xlsxxlsb) and the [benchmarks](docs/performance/benchmarks.md#parallel-record-writes). - The XLS writer buffers records in memory and assembles the OLE container at `EndAsync`; choose it when write throughput matters more than peak allocation. - `Excel.EncryptPackage`/`EncryptPackageAsync` wrap a written XLSX/XLSB package in agile ECMA-376 encryption; see [Encrypted workbooks](docs/guide/encryption.md). diff --git a/docs/guide/migrating-to-v7.md b/docs/guide/migrating-to-v7.md index 1a73bf9d..555605d6 100644 --- a/docs/guide/migrating-to-v7.md +++ b/docs/guide/migrating-to-v7.md @@ -133,6 +133,24 @@ Two behavior changes come with them: - The CLI `convert` command no longer shifts dates by four years when the source workbook uses the 1904 date system. - [`XlsxWriterOptions`](xref:ExcelReader.Core.Writer.Xlsx.XlsxWriterOptions) gains `Date1904`, which v6 lacked. It defaults to `false`, so existing output is unchanged. +## Stricter reading and writing + +v7 refuses several inputs that v6 accepted and then got wrong. + +- **Damaged packages.** Each ZIP entry of an XLSX or XLSB package is checked against its CRC-32. A mismatch throws `InvalidDataException` when the entry's last byte is read, so rows before the damage may already have been returned. v6 checked only the entry's size. Set [`ExcelReaderOptions.VerifyZipChecksums`](xref:ExcelReader.Core.Reader.ExcelReaderOptions.VerifyZipChecksums) to `false` to read a package whose producer wrote wrong checksums; the bindings have the same switch as `verify_zip_checksums`. The check costs 2% to 4% of a read, and under 1% with `PrefetchDecompression`, where it runs on the background thread. +- **Line breaks in XLSX text.** A cell with a line break now reads with ` +`, as XLSB and XLS already did. Excel stores the break as a CR LF pair in the XML, which v6 returned as ` +`. A carriage return stored as data (`_x000D_` or ` `) is still returned. Code that split XLSX text on ` +` should split on ` +`. +- **Sheet names.** `AddSheet` throws `ArgumentException` for a name that begins or ends with an apostrophe, or contains a control character (tab and line feed included), U+FFFE or U+FFFF. Excel refuses a workbook with such a name. +- **Column settings.** `SetColumnWidth` and `SetColumnStyle` throw `ArgumentOutOfRangeException` for a column past the format's last one (16,384 columns; 256 for XLS) and for a width above 255, Excel's maximum. v6 wrote a file Excel refused for the first, and an XLS width of 256 came out as a hidden column. +- **Number formats.** `AddStyle` throws `ExcelLimitExceededException` for the 201st distinct number format of a workbook, and `ArgumentException` for a format longer than 255 characters. Excel does not open a workbook past either. +- **Rows in XLSB.** A sheet stops at 1,048,576 rows. v6 accepted one more, which Excel does not read. +- **Values with no numeric form.** `Write` with a type that does not convert to a number, such as `char`, throws `ArgumentException` in every writer. XLSB and XLS threw `InvalidCastException`. +- **Text in XLSX.** A carriage return, U+FFFE and U+FFFF in cell text are now written escaped. v6 wrote them raw: Excel read the carriage return as a line feed and refused a file with either of the other two. +- **`ExcelLimitExceededException.Message`** no longer says "bytes" for limits counted in rows, columns or entries. `LimitName`, `Limit` and `Actual` are unchanged. + ## Native bindings The C ABI is unchanged: no signature moved, and `XL_ABI_VERSION` stays 6. Behavior changed. `degree_of_parallelism` of `0` on `xl_parse_typed` and the Arrow exports now parallelises XLSX and XLSB as well as CSV, using every processor. Pass `1` for the old sequential behavior. The Rust, Python and C++ bindings take the same value (`degree_of_parallelism`, `parallelism`) and need no code change; a `parallelism=0` call on an XLSX or XLSB sheet is now faster, and holds every partition's columns until they merge, so its peak memory is higher. diff --git a/docs/guide/untrusted-input.md b/docs/guide/untrusted-input.md index ce61e4fa..5922d20d 100644 --- a/docs/guide/untrusted-input.md +++ b/docs/guide/untrusted-input.md @@ -110,6 +110,17 @@ bytes. Anything else escaping a read of a file — an `IndexOutOfRangeException` [SECURITY.md](../../SECURITY.md). See [Errors across languages](errors.md) for how these map onto the bindings. +## Damaged packages + +An XLSX or XLSB package is a ZIP archive, and each of its entries carries a CRC-32 of its data. +The reader checks it as the entry is read and throws `InvalidDataException` on a mismatch, so a file +cut short or altered in transit does not come back as plausible but wrong cells. A sheet is checked +when its last byte is read, which means the rows before the damage may already have been returned: +treat the exception as invalidating everything read from that file. The checksum guards against +accidents, not tampering, since whoever alters the data can recompute it. +[`ExcelReaderOptions.VerifyZipChecksums`](xref:ExcelReader.Core.Reader.ExcelReaderOptions.VerifyZipChecksums) +turns the check off. + ## What the reader does not do - **It evaluates nothing.** A formula cell reports the result Excel cached when it saved the file, diff --git a/docs/guide/writing.md b/docs/guide/writing.md index 650c0236..b3e52711 100644 --- a/docs/guide/writing.md +++ b/docs/guide/writing.md @@ -263,13 +263,78 @@ Writing a 50,000-row workbook (`WriteBenchmark`, both figures from one run): | Workload | Default | `PrefetchWrite = true` | Gain | |---|---:|---:|---:| -| XLSX | 10.910 ms | 6.865 ms | 37% | -| XLSB | 7.555 ms | 5.614 ms | 26% | +| XLSX | 8.337 ms | 5.175 ms | 38% | +| XLSB | 5.572 ms | 3.419 ms | 39% | -Allocations are unchanged (4.02 MB vs. 4.03–4.04 MB) — the background writer hands over buffers the row +Allocations are unchanged (4.02 MB either way) — the background writer hands over buffers the row writer already owns rather than copying them. The same caveat as the read side applies: **do not enable it for concurrent server workloads**. A caller already writing many files in parallel is CPU-saturated, and an extra background thread per writer only doubles thread demand for no gain. It's meant for single-file batch work. +## Write records in parallel (XLSX/XLSB) + +`PrefetchWrite` moves compression off the calling thread, but the rows are still built on one thread, +and building them is most of a write. `WriteRecordsParallelAsync` and `WriteTableParallelAsync` use +several cores for both: the records are cut into batches, each batch is formatted and compressed on +a pool thread, and the batches reach the file in order. They are extensions on +[`XlsxSheetWriter`](xref:ExcelReader.Core.Writer.Xlsx.XlsxSheetWriter) and +[`XlsbSheetWriter`](xref:ExcelReader.Core.Writer.Xlsb.XlsbSheetWriter), and take an `IEnumerable` +or an `IAsyncEnumerable`. The cell-by-cell API stays single-threaded, and XLS and CSV have no +parallel form. + + +```csharp +await using var wb = XlsxWorkbookWriter.Create(stream, leaveOpen: true); +XlsxSheetWriter sheet = wb.AddSheet("S1"); + +// Header row from the layout, then one row per record. +await sheet.WriteRecordsParallelAsync(records, ExcelRecordLayout.FromAttributes()); + +// Or format each row yourself; this delegate runs on several threads at once. +await sheet.WriteRecordsParallelAsync(records, static (XlsxRowWriter row, Metric record) => +{ + row.Write(record.Name); + row.Write(record.Value); +}, new ParallelWriteOptions { DegreeOfParallelism = 4 }); + +await sheet.EndAsync(); +``` + +The rules: + +- **Your delegate runs on several threads at once**, though never twice at once on the same row + writer. Whatever it captures must be thread-safe. The library's own layouts are. +- **Order is kept.** Rows come out in the order the source yields them, and the source itself is + enumerated by one flow at a time, never concurrently. +- **Rows may be written before and after the call, not during it.** Row numbering carries on, and + the call can be made more than once on a sheet. +- **A failure or a cancellation ends the sheet and the workbook.** A serial `WriteRecordsAsync` + cancelled between rows leaves the sheet usable; a parallel call does not. The original exception is + rethrown once every batch in flight has stopped. +- **With `UseSharedStrings`, string indexes depend on thread timing**, so two runs give files that + differ byte for byte. Their content is the same. +- More than 1,048,576 rows throws `ExcelLimitExceededException`. + +[`ParallelWriteOptions`](xref:ExcelReader.Core.Writer.ParallelWriteOptions) has two knobs. +`DegreeOfParallelism` is the most batches in flight at once (0, the default, means one per core). +`BatchRows` is the rows per batch, 8,192 by default. Memory grows with their product: at the +defaults each batch in flight holds about 3 MB (its formatted rows, the compressor's window and the +compressed result), whatever the size of the sheet. + +Writing 1,000,000 rows of four columns (text, integer, date, number) to a `MemoryStream` at +`Fastest`, on a Ryzen 7 5700X, median of 9 runs: + +| Write | XLSX | XLSB | +|---|---:|---:| +| Serial | 187.3 ms | 123.6 ms | +| Serial, `PrefetchWrite` | 117.8 ms | 67.6 ms | +| Parallel, 1 batch in flight | 172.7 ms | 120.6 ms | +| Parallel, 2 | 92.1 ms | 63.2 ms | +| Parallel, 4 | 49.9 ms | 40.9 ms | +| Parallel, 8 | 35.7 ms | 27.7 ms | + +The file comes out 0.18% (XLSX) or 0.04% (XLSB) larger, because each batch is compressed on its own. +As with `PrefetchWrite`, this is for single-file batch work: a server already writing many files at +once has no idle cores to give it. diff --git a/docs/performance/benchmarks.md b/docs/performance/benchmarks.md index 665a46fd..67bce0fd 100644 --- a/docs/performance/benchmarks.md +++ b/docs/performance/benchmarks.md @@ -20,10 +20,10 @@ Compares ExcelReader against established XLSX libraries on the same generated wo | Typed row parsing | 10.668 ms, 3.87 MB | 56.150 ms, 10.48 MB | 107.805 ms, 5.70 MB | - | | Typed row parsing async | 10.833 ms, 3.87 MB | 59.036 ms, 10.48 MB | - | - | | Typed row parsing, shared strings | 9.216 ms, 2.29 MB | - | - | - | -| Workbook writing | 10.913 ms, 4.02 MB | - | 18.166 ms, 4.02 MB | 15.128 ms, 15.84 MB | -| Workbook writing, shared strings | 10.848 ms, 4.06 MB | - | - | - | +| Workbook writing | 8.337 ms, 4.02 MB | - | 18.437 ms, 4.02 MB | 15.332 ms, 15.84 MB | +| Workbook writing, shared strings | 8.358 ms, 4.06 MB | - | - | - | -ExcelReader is ~5.3x faster than Sylvan for raw XLSX reads, allocating ~543x less, and ~15.6x faster than OfficeIMO. For typed parsing it is ~5.3x faster than Sylvan (~5.4x async against async). For XLSX writing, ExcelReader is ~1.4x faster than SpreadCheetah while allocating ~3.9x less memory, and ~1.7x faster than OfficeIMO. +ExcelReader is ~5.3x faster than Sylvan for raw XLSX reads, allocating ~543x less, and ~15.6x faster than OfficeIMO. For typed parsing it is ~5.3x faster than Sylvan (~5.4x async against async). For XLSX writing, ExcelReader is ~1.8x faster than SpreadCheetah while allocating ~3.9x less memory, and ~2.2x faster than OfficeIMO. OfficeIMO.Excel's typed parsing is hand-mapped from its `OpenDataReader`, because `RowsAs` needs an `r` attribute on every row and cell, which the spec makes optional and ExcelReader's writer omits. @@ -37,10 +37,10 @@ Reading a shared-strings XLSX workbook with typed parsing is ~14% faster than th | Cell-by-cell read async | 4.845 ms, 3.89 KB | - | | Typed row parsing | 7.819 ms, 3.87 MB | - | | Typed row parsing async | 8.061 ms, 3.87 MB | - | -| Workbook writing | 7.478 ms, 4.02 MB | - | -| Workbook writing, shared strings | 7.370 ms, 4.06 MB | - | +| Workbook writing | 5.572 ms, 4.02 MB | - | +| Workbook writing, shared strings | 5.595 ms, 4.05 MB | - | -XLSB is the fastest generated Excel format in these results: raw reads are ~1.4x faster than XLSX reads, typed parsing is ~1.4x faster than XLSX parsing, and writing is ~1.5x faster than XLSX writing. Against OfficeIMO's XLSB reader, ExcelReader is ~2.3x faster while allocating ~1,800x less. The XLSB writer is also ~2.0x faster than SpreadCheetah's XLSX writer on this benchmark while allocating ~75% less memory. +XLSB is the fastest generated Excel format in these results: raw reads are ~1.4x faster than XLSX reads, typed parsing is ~1.4x faster than XLSX parsing, and writing is ~1.5x faster than XLSX writing. Against OfficeIMO's XLSB reader, ExcelReader is ~2.3x faster while allocating ~1,800x less. The XLSB writer is also ~2.8x faster than SpreadCheetah's XLSX writer on this benchmark while allocating ~75% less memory. ### XLS (BIFF8) @@ -48,9 +48,9 @@ XLSB is the fastest generated Excel format in these results: raw reads are ~1.4x |---|---:|---:|---:| | Cell-by-cell read | 3.916 ms, 2.84 KB | 5.173 ms, 1,717.73 KB | 6.856 ms, 10,499.25 KB | | Cell-by-cell read async | 3.236 ms, 2.91 KB | - | - | -| Workbook writing | 5.296 ms, 16.03 MB | - | - | +| Workbook writing | 4.859 ms, 16.31 MB | - | - | -ExcelReader is ~1.3x faster than Sylvan for generated XLS reads while allocating ~605x less memory, and ~1.8x faster than OfficeIMO. The XLS writer is ~2.1x faster than the XLSX writer in this benchmark (`XlsWriteBenchmark`, 5.296 ms vs 10.964 ms in the same run). Most of its 16.03 MB is the benchmark's pre-sized 16 MB destination `MemoryStream`; the typed-record table below measures it at 4.03 MB like the other formats. +ExcelReader is ~1.3x faster than Sylvan for generated XLS reads while allocating ~605x less memory, and ~1.8x faster than OfficeIMO. The XLS writer is ~1.7x faster than the XLSX writer in this benchmark (`XlsWriteBenchmark`, 4.859 ms vs 8.375 ms in the same run). Most of its 16.31 MB is the benchmark's pre-sized 16 MB destination `MemoryStream`; the typed-record table below measures it at 4.31 MB, close to the other formats. ### CSV @@ -174,6 +174,52 @@ About 40% of the default read is inflate: without it the whole read drops from 3 Without prefetch a read is inflate plus parse, so the faster inflate converts directly; with prefetch the two overlap and the read is bounded by the slower stage, which is why the prefetch columns above moved little. +### Write path: managed deflate and CRC-32 + +XLSX and XLSB packages are written by the library's own ZIP writer. `CompressionLevel.Fastest`, the default, compresses with a managed fixed-Huffman deflate encoder and checksums with a managed CRC-32 that folds with PCLMULQDQ. Each piece alone on the same Ryzen 7 5700X: the deflate rows against zlib-ng at `Fastest`, which `ZipArchive` used, and the CRC row against `System.IO.Hashing.Crc32`, the fastest managed alternative (`ZipArchive` itself calls zlib's native CRC, which was not measured): + +| | zlib-ng `Fastest` / System.IO.Hashing | Managed | Ratio | +|---|---:|---:|---:| +| Deflate, writer XLSX sheet (6.0 MB) | 4.75 ms | 2.98 ms | 1.59x, same size | +| Deflate, writer XLSB sheet (4.6 MB) | 4.60 ms | 2.52 ms | 1.82x, same size | +| Deflate, 65K real XLSX sheet (34.4 MB) | 60.3 ms | 38.4 ms | 1.57x, same size | +| CRC-32, 64 KB chunks in cache | 16.4 GB/s | 35.8 GB/s | 2.19x | + +Whole writes of 50,000 rows by four columns (text, integer, date, number) into a `MemoryStream`, before and after the switch. Each figure is the median of three interleaved runs of 41 writes: + +| Write | `ZipArchive` | Managed ZIP writer | Change | File size | +|---|---:|---:|---:|---:| +| XLSX, `Fastest` | 10.43 ms | 9.11 ms | -12.7% | 773,487 B → 774,344 B | +| XLSX, `Fastest` + `PrefetchWrite` | 6.15 ms | 5.84 ms | -5.0% | same as above | +| XLSX, `Optimal` | 30.90 ms | 31.38 ms | within noise | 483,613 B → 483,607 B | +| XLSB, `Fastest` | 7.66 ms | 5.42 ms | -29.2% | 861,060 B → 861,568 B | +| XLSB, `Fastest` + `PrefetchWrite` | 5.37 ms | 3.08 ms | -42.6% | same as above | +| XLSB, `Optimal` | 38.17 ms | 37.98 ms | within noise | 466,917 B → 466,917 B | + +XLSB gains more because deflate was a larger share of its write; with `PrefetchWrite` an XLSX write is bounded by building the XML, so the faster deflate moves it little. + +The encoder's limits are deliberate: + +- It emits fixed-Huffman blocks only, so `Optimal` and `SmallestSize` still go through zlib, inside the same container. +- A block covers at most 65,535 input bytes, so any block that would grow is written stored instead. Incompressible input grows by at most 5 bytes per block. + +### Parallel record writes + +`WriteRecordsParallelAsync` formats and compresses batches of 8,192 records on pool threads and appends them to the sheet in order. Writing 1,000,000 rows by four columns (text, integer, date, number) into a pre-sized `MemoryStream` at `Fastest`, on the same Ryzen 7 5700X. Each figure is the median of 9 writes after 3 warm-up writes, from a console harness rather than BenchmarkDotNet: + +| Write | XLSX | vs serial | XLSB | vs serial | +|---|---:|---:|---:|---:| +| Serial | 187.3 ms | 1.0x | 123.6 ms | 1.0x | +| Serial, `PrefetchWrite` | 117.8 ms | 1.6x | 67.6 ms | 1.8x | +| Parallel, 1 batch in flight | 172.7 ms | 1.1x | 120.6 ms | 1.0x | +| Parallel, 2 | 92.1 ms | 2.0x | 63.2 ms | 2.0x | +| Parallel, 4 | 49.9 ms | 3.8x | 40.9 ms | 3.0x | +| Parallel, 8 | 35.7 ms | 5.2x | 27.7 ms | 4.5x | + +It scales because a write is bound by building rows, not by deflate: `PrefetchWrite` already takes deflate off the calling thread and stops at 1.6x to 1.8x, so compressing chunks in parallel would add nothing on top of it, while formatting the rows in parallel does. + +Each batch is compressed with no reference to the one before it, which costs 0.18% of file size for XLSX (15,481,372 B to 15,508,760 B) and 0.04% for XLSB (16,863,131 B to 16,869,306 B). The process's peak working set rose by 12 MB over the serial write with 4 batches in flight and by 23 MB with 8, about 3 MB per batch, and does not depend on the number of rows. + ### Matched-work reads (`*_Materialized`) Every read benchmark class has a `*_Materialized` sibling that calls `cell.GetString()` per text cell — the same UTF-16 materialization Sylvan's ADO.NET-style API is forced to pay — while keeping `TryParse`/`TryGetDateTime` for numeric and date cells, exactly mirroring the competitor accumulator. These are the matched-work counterparts to the zero-copy rows above. @@ -217,12 +263,12 @@ The takeaway is not that one column beats the other: it is that ExcelReader's he | Format | Mean | Allocated | |---|---:|---:| -| XLSX | 12.099 ms | 4.02 MB | -| XLSB | 7.597 ms | 4.02 MB | -| XLS | 5.021 ms | 4.03 MB | -| CSV | 4.762 ms | 4.00 MB | +| XLSX | 8.922 ms | 4.02 MB | +| XLSB | 6.113 ms | 4.02 MB | +| XLS | 6.292 ms | 4.31 MB | +| CSV | 4.824 ms | 4.00 MB | -Relative ordering: CSV and XLS close at the front, then XLSB, then XLSX — the record-mapping layer adds little overhead over hand-written cell-by-cell writes (XLSB 7.597 ms vs. 7.478 ms, XLSX 12.099 ms vs. 10.913 ms). +Relative ordering: CSV at the front, XLSB and XLS close behind it, then XLSX — the record-mapping layer adds little overhead over hand-written cell-by-cell writes (XLSB 6.113 ms vs. 5.572 ms, XLSX 8.922 ms vs. 8.337 ms). ### Native and Arrow string writes @@ -230,12 +276,12 @@ Relative ordering: CSV and XLS close at the front, then XLSB, then XLSX — the | Scenario | Mean | Allocated | |---|---:|---:| -| Native string columns → XLSX | 20.487 ms | 18.02 KB | +| Native string columns → XLSX | 14.559 ms | 15.74 KB | | Native string columns → CSV | 5.098 ms | 416 B | -| Arrow `StringArray` → XLSX | 26.644 ms | 18.06 KB | -| Styled rows → XLSX | 7.068 ms | 18.20 KB | +| Arrow `StringArray` → XLSX | 20.343 ms | 15.79 KB | +| Styled rows → XLSX | 5.411 ms | 15.92 KB | -Text arrives through `IRowWriter.WriteUtf8`, which the XLSX and CSV writers copy through as bytes whenever the text needs no escaping, so none of these allocate per cell: the ~18 KB is per-workbook ZIP and part setup, which CSV does not have. +Text arrives through `IRowWriter.WriteUtf8`, which the XLSX and CSV writers copy through as bytes whenever the text needs no escaping, so none of these allocate per cell: the ~16 KB is per-workbook ZIP and part setup, which CSV does not have. ### Ref struct typed parsing (zero-copy) diff --git a/docs/snippets/Writing.cs b/docs/snippets/Writing.cs index c69772fe..b2f8b115 100644 --- a/docs/snippets/Writing.cs +++ b/docs/snippets/Writing.cs @@ -278,3 +278,31 @@ internal static async Task Run(Stream stream, Metric[] records) } } } + +namespace Snippets.Writing.Parallel +{ + public sealed record Metric(string Name, double Value); + + internal static class Snippet + { + internal static async Task Run(Stream stream, IEnumerable records) + { + #region writing-parallel + await using var wb = XlsxWorkbookWriter.Create(stream, leaveOpen: true); + XlsxSheetWriter sheet = wb.AddSheet("S1"); + + // Header row from the layout, then one row per record. + await sheet.WriteRecordsParallelAsync(records, ExcelRecordLayout.FromAttributes()); + + // Or format each row yourself; this delegate runs on several threads at once. + await sheet.WriteRecordsParallelAsync(records, static (XlsxRowWriter row, Metric record) => + { + row.Write(record.Name); + row.Write(record.Value); + }, new ParallelWriteOptions { DegreeOfParallelism = 4 }); + + await sheet.EndAsync(); + #endregion + } + } +} diff --git a/python/README.md b/python/README.md index 16bf8f9f..d694c42f 100644 --- a/python/README.md +++ b/python/README.md @@ -463,6 +463,10 @@ options = OpenOptions( single-file batch work, not for a server already reading many files in parallel. See the root README for the measured trade. +Each ZIP entry of an XLSX or XLSB is checked against its CRC-32 as it is read, so a damaged file +raises `ExcelReaderError` instead of returning wrong cells. `verify_zip_checksums=False` turns the +check off, for a file whose producer wrote wrong checksums. + ### Encrypted workbooks `open_workbook()`/`open_bytes()` take a `password` keyword to open a password-protected OOXML diff --git a/python/src/excelreader/_native.py b/python/src/excelreader/_native.py index c67def55..cf8f5b25 100644 --- a/python/src/excelreader/_native.py +++ b/python/src/excelreader/_native.py @@ -266,6 +266,7 @@ class NativeOpenOptions(ctypes.Structure): ("source_cache_bytes", ctypes.c_int64), ("max_buffered_bytes", ctypes.c_int64), ("source_block_size", ctypes.c_int32), + ("verify_zip_checksums", ctypes.c_int32), ] @@ -334,6 +335,7 @@ def to_native_open_options(options: OpenOptions) -> NativeOpenOptions: raw.csv_intern_strings = _opt_state(options.csv_intern_strings) raw.prefetch_decompression = _opt_state(options.prefetch_decompression) raw.intern_strings = _opt_state(options.intern_strings) + raw.verify_zip_checksums = _opt_state(options.verify_zip_checksums) raw.csv_delimiter = _opt_number(options.csv_delimiter) raw.csv_quote = _opt_number(options.csv_quote) raw.csv_max_cell_bytes = _opt_number(options.csv_max_cell_bytes) diff --git a/python/src/excelreader/types.py b/python/src/excelreader/types.py index c562a61a..5a143485 100644 --- a/python/src/excelreader/types.py +++ b/python/src/excelreader/types.py @@ -31,6 +31,10 @@ class SheetVisibility(IntEnum): VERY_HIDDEN = _native.XL_SHEET_VERY_HIDDEN +# One past 9999-12-31, the last day Excel shows as a date. +_MAX_SERIAL = 2_958_466 + + class Cell(NamedTuple): """One cell. `value` is the raw text as stored, so DATE cells are Excel serial numbers.""" @@ -42,11 +46,21 @@ def as_date(self, date1904: bool = False) -> datetime.date | None: """Converts a DATE cell's serial value to a date. Returns None for any other cell type. `date1904` should come from `Workbook.is_date1904` for the workbook this cell came from. + Raises `ValueError` for a serial outside what Excel can show as a date. """ if self.type is not CellType.DATE: return None - epoch = datetime.date(1904, 1, 1) if date1904 else datetime.date(1899, 12, 30) - return epoch + datetime.timedelta(days=int(float(self.value))) + serial = float(self.value) + if not 0 <= serial < _MAX_SERIAL: + raise ValueError(f"{self.value} is not an Excel date serial") + days = int(serial) + if date1904: + return datetime.date(1904, 1, 1) + datetime.timedelta(days=days) + # Excel counts a 29 February 1900 that never existed, so serials before it sit one day later + # against this epoch than those after. Serial 60, that day itself, reads as 28 February. + if days < 60: + days += 1 + return datetime.date(1899, 12, 30) + datetime.timedelta(days=days) class ColumnarSheet(NamedTuple): @@ -121,6 +135,10 @@ class OpenOptions(NamedTuple): in 64 MiB by default; a negative block size turns the cache off). `max_buffered_bytes` caps how much `open_stream()` reads before an XLSX, XLSB or XLS opens. + `verify_zip_checksums` (on by default) checks each ZIP entry of an XLSX or XLSB against its + CRC-32, so a damaged file raises instead of returning wrong cells. Set it to `False` to read a + file whose producer wrote wrong checksums. + Validation happens on the native side, which owns the real limits; an out-of-range value raises `ExcelReaderError` with the reason. Fields mirror xl_open_options in src/ExcelReader.Native/include/excelreader.h. @@ -141,6 +159,7 @@ class OpenOptions(NamedTuple): source_block_size: int | None = None source_cache_bytes: int | None = None max_buffered_bytes: int | None = None + verify_zip_checksums: bool | None = None class WriteOptions(NamedTuple): diff --git a/python/src/excelreader/writer_handle.py b/python/src/excelreader/writer_handle.py index afa171c2..589f9dc6 100644 --- a/python/src/excelreader/writer_handle.py +++ b/python/src/excelreader/writer_handle.py @@ -18,6 +18,8 @@ _EPOCH_DATE = datetime.date(1970, 1, 1) _EPOCH_DATETIME = datetime.datetime(1970, 1, 1) +_INT64_MIN = -(2**63) +_INT64_MAX = 2**63 - 1 def _encode(value: str) -> tuple[Any, int]: @@ -74,9 +76,13 @@ def write_str(self, value: str | None) -> None: _check(self._lib.xl_write_string(handle, pointer, length)) def write_i64(self, value: int | None) -> None: + """Writes the next cell as a whole number. Raises `OverflowError` outside the signed 64-bit range.""" if value is None: self.write_null(ColumnType.I64) return + # ctypes would keep the low 64 bits of a larger value without complaint. + if not _INT64_MIN <= value <= _INT64_MAX: + raise OverflowError(f"{value} does not fit a 64-bit integer cell; write it with write_f64 or write_str") _check(self._lib.xl_write_int64(self._require_handle(), value)) def write_f64(self, value: float | None) -> None: @@ -126,7 +132,8 @@ def write_row(self, values: Sequence[Any]) -> None: The type of each value picks the write method. `None` becomes a blank *string* cell — it is the one case inference cannot resolve; use `write_null(ColumnType.X)` for a typed blank. - Raises `TypeError`, before writing anything, for a value whose type has no mapping. + Raises, before writing anything, `TypeError` for a value whose type has no mapping and + `OverflowError` for an `int` outside the signed 64-bit range. """ for position, value in enumerate(values): if not _is_writable(value): @@ -134,6 +141,8 @@ def write_row(self, values: Sequence[Any]) -> None: f"write_row cannot infer a cell type for {type(value).__name__} " f"at position {position}" ) + if type(value) is int and not _INT64_MIN <= value <= _INT64_MAX: + raise OverflowError(f"{value} at position {position} does not fit a 64-bit integer cell") self.start_row() for value in values: diff --git a/python/tests/test_reader.py b/python/tests/test_reader.py index 1f9888ba..896dbe7f 100644 --- a/python/tests/test_reader.py +++ b/python/tests/test_reader.py @@ -823,3 +823,45 @@ def test_infer_schema_without_parse_text_keeps_csv_columns_as_strings(): schema = workbook.sheets[0].infer_schema() assert {spec.type for spec in schema} == {ColumnType.STRING} + + +@pytest.mark.parametrize( + ("serial", "date1904", "expected"), + [ + ("1", False, (1900, 1, 1)), + ("59", False, (1900, 2, 28)), + ("61", False, (1900, 3, 1)), + ("45000.75", False, (2023, 3, 15)), + ("2958465", False, (9999, 12, 31)), + ("0", True, (1904, 1, 1)), + ("43538", True, (2023, 3, 15)), + ], +) +def test_as_date_follows_excels_calendar(serial, date1904, expected): + from datetime import date + + assert Cell(0, CellType.DATE, serial).as_date(date1904) == date(*expected) + + +@pytest.mark.parametrize("serial", ["-1", "2958466", "1e400", "nan"]) +def test_as_date_rejects_a_serial_excel_cannot_show(serial): + with pytest.raises(ValueError, match="not an Excel date serial"): + Cell(0, CellType.DATE, serial).as_date() + + +@pytest.mark.parametrize( + ("text", "expected_micros"), + [ + ("2024-01-15T13:30:05.0000000Z", 1_705_325_405_000_000), + ("2024-01-15T13:30:05+03:00", 1_705_314_605_000_000), + ("2024-01-15T13:30:05", 1_705_325_405_000_000), + ], +) +def test_parse_typed_timestamp_text_does_not_depend_on_the_local_time_zone(tmp_path, text, expected_micros): + path = tmp_path / "stamps.csv" + path.write_text(f"at\n{text}\n", encoding="utf-8") + + with open_workbook(path) as workbook: + table = workbook.sheets[0].parse_typed([ColumnSpec(ColumnType.TIMESTAMP, name="at")]) + + assert list(table.columns[0]) == [expected_micros] diff --git a/python/tests/test_writer_handle.py b/python/tests/test_writer_handle.py index 9f09469a..7b3b6042 100644 --- a/python/tests/test_writer_handle.py +++ b/python/tests/test_writer_handle.py @@ -219,3 +219,18 @@ def test_write_workbook_to_bytes_matches_write_workbook(tmp_path, xlsx_path): from_memory = [[cell.value for cell in row] for row in workbook.sheets[0].rows()] assert from_file == from_memory + + +@pytest.mark.parametrize("value", [2**63, -(2**63) - 1, 2**64 + 5]) +def test_an_integer_beyond_64_bits_is_refused_not_wrapped(tmp_path, value): + with open_writer(tmp_path / "big.xlsx") as writer: + writer.start_sheet("Data") + with pytest.raises(OverflowError): + writer.write_row(["fine", value]) + writer.start_row() + with pytest.raises(OverflowError): + writer.write_i64(value) + writer.write_i64(2**63 - 1) + writer.write_i64(-(2**63)) + writer.end_row() + writer.end_sheet() diff --git a/rust/excelreader/src/lib.rs b/rust/excelreader/src/lib.rs index 82e7f00c..64e81bc5 100644 --- a/rust/excelreader/src/lib.rs +++ b/rust/excelreader/src/lib.rs @@ -128,6 +128,7 @@ pub struct XlOpenOptions { pub source_cache_bytes: i64, pub max_buffered_bytes: i64, pub source_block_size: i32, + pub verify_zip_checksums: i32, } #[repr(C)] diff --git a/rust/excelreader/src/options.rs b/rust/excelreader/src/options.rs index a14c58ae..8f9342ec 100644 --- a/rust/excelreader/src/options.rs +++ b/rust/excelreader/src/options.rs @@ -37,6 +37,8 @@ pub struct OpenOptions { pub max_zip_entries: Option, pub prefetch_decompression: Option, pub intern_strings: Option, + /// XLSX/XLSB only: check each ZIP entry against its CRC-32. `None` is `true`. + pub verify_zip_checksums: Option, /// `open_source` only: bytes fetched per `read_at` call. `None` is 4 MiB; a negative value turns /// the cache off. @@ -98,6 +100,9 @@ impl OpenOptions { prefetch_decompression: bool, /// XLS/XLSX/XLSB only. Library default is `false`. intern_strings: bool, + /// XLSX/XLSB only: check each ZIP entry against its CRC-32, so a damaged file fails the read + /// instead of returning wrong cells. Library default is `true`. + verify_zip_checksums: bool, /// `open_source` only: bytes fetched per `read_at` call. Library default is 4 MiB; a negative /// value turns the cache off. source_block_size: i32, @@ -148,6 +153,7 @@ impl OpenOptions { source_cache_bytes: opt_number(self.source_cache_bytes), max_buffered_bytes: opt_number(self.max_buffered_bytes), source_block_size: opt_number(self.source_block_size), + verify_zip_checksums: opt_state(self.verify_zip_checksums), }; if let Some(bytes) = password { raw.password = bytes.as_ptr(); @@ -304,11 +310,13 @@ mod tests { fn booleans_lower_to_distinct_true_and_false_states() { let options = OpenOptions::new() .prefetch_decompression(true) - .intern_strings(false); + .intern_strings(false) + .verify_zip_checksums(false); let guard = options.to_raw(); let raw = guard.raw; assert_eq!(raw.prefetch_decompression, XL_OPT_TRUE); assert_eq!(raw.intern_strings, XL_OPT_FALSE); + assert_eq!(raw.verify_zip_checksums, XL_OPT_FALSE); assert_eq!(raw.csv_detect_bom, XL_OPT_DEFAULT); } @@ -393,6 +401,7 @@ mod tests { assert_eq!(std::mem::offset_of!(XlOpenOptions, source_cache_bytes), 88); assert_eq!(std::mem::offset_of!(XlOpenOptions, max_buffered_bytes), 96); assert_eq!(std::mem::offset_of!(XlOpenOptions, source_block_size), 104); + assert_eq!(std::mem::offset_of!(XlOpenOptions, verify_zip_checksums), 108); assert_eq!(std::mem::size_of::(), 40); assert_eq!(std::mem::size_of::(), 32); } diff --git a/src/ExcelReader.Core/Parser/ExcelCellReaders.cs b/src/ExcelReader.Core/Parser/ExcelCellReaders.cs index b186ce01..da0833dd 100644 --- a/src/ExcelReader.Core/Parser/ExcelCellReaders.cs +++ b/src/ExcelReader.Core/Parser/ExcelCellReaders.cs @@ -76,8 +76,7 @@ private static bool ReadDateOnlyAuto(in Cell cell, in ExcelRowContext context, o private static bool ReadTimeOnlyAuto(in Cell cell, in ExcelRowContext context, out TimeOnly value) { - return ColumnParserFactory.ReadTimeOnly(in cell, in context, out value) - || ColumnParserFactory.ReadTextTimeOnly(in cell, in context, out value); + return ColumnParserFactory.ReadTimeOnlyAuto(in cell, in context, out value); } /// diff --git a/src/ExcelReader.Core/Parser/Internal/ColumnParserFactory.cs b/src/ExcelReader.Core/Parser/Internal/ColumnParserFactory.cs index 0af63c74..055cd8bb 100644 --- a/src/ExcelReader.Core/Parser/Internal/ColumnParserFactory.cs +++ b/src/ExcelReader.Core/Parser/Internal/ColumnParserFactory.cs @@ -119,7 +119,7 @@ internal static ColumnParser BuildConverter(PropertyInfo prop, Type conver } if (propType == typeof(TimeOnly)) { - return BuildValue(prop, ReadTimeOnly); + return BuildValue(prop, ReadTimeOnlyAuto); } if (propType.IsEnum) { @@ -161,7 +161,7 @@ internal static ColumnParser BuildConverter(PropertyInfo prop, Type conver } if (innerType == typeof(TimeOnly)) { - return BuildNullableValue(prop, ReadTimeOnly); + return BuildNullableValue(prop, ReadTimeOnlyAuto); } if (innerType.IsEnum) { @@ -272,6 +272,11 @@ internal static bool ReadTimeOnly(in Cell cell, in ExcelRowContext context, out return true; } + internal static bool ReadTimeOnlyAuto(in Cell cell, in ExcelRowContext context, out TimeOnly value) + { + return ReadTimeOnly(in cell, in context, out value) || ReadTextTimeOnly(in cell, in context, out value); + } + internal static bool ReadTextDateTime(in Cell cell, in ExcelRowContext context, out DateTime value) { return TryParseDateTimeText(in cell, context.FormatProvider, out value); @@ -292,6 +297,11 @@ internal static bool ReadTextTimeOnly(in Cell cell, in ExcelRowContext context, internal static bool TryParseSpanParsable(in Cell cell, IFormatProvider provider, [MaybeNullWhen(false)] out TValue value) where TValue : ISpanParsable { + // XLSB and XLS keep a number in binary, with no text in Value to parse. + if (cell.Value.IsEmpty && cell.Type is CellType.Number or CellType.Date) + { + return TValue.TryParse(cell.GetString(), provider, out value); + } Span stack = stackalloc char[Utf8Text.StackChars]; ReadOnlySpan chars = Utf8Text.Decode(cell.Value, stack, out char[]? rented); try @@ -480,7 +490,7 @@ private static FrozenDictionary BuildValueMap() [SkipLocalsInit] public static bool TryParse(in Cell cell, out TEnum value) { - if (cell.Type == CellType.Number && cell.TryGetDouble(out double d)) + if (cell.Type is CellType.Number or CellType.Date && cell.TryGetDouble(out double d)) { if (d != Math.Truncate(d) || d < long.MinValue || d > long.MaxValue) { @@ -585,6 +595,12 @@ private static bool TryParseBool(in Cell cell, out bool value) } if (Ascii.EqualsIgnoreCase(v, "true"u8)) { value = true; return true; } if (Ascii.EqualsIgnoreCase(v, "false"u8)) { value = false; return true; } + // XLSB and XLS keep a number in binary, with no text in Value to compare. + if (cell.Type is CellType.Number or CellType.Date && cell.TryGetDouble(out double number) && number is 0 or 1) + { + value = number == 1; + return true; + } value = false; return false; } diff --git a/src/ExcelReader.Core/PublicAPI/PublicAPI.Unshipped.txt b/src/ExcelReader.Core/PublicAPI/PublicAPI.Unshipped.txt index 772cc5ed..5d3628e0 100644 --- a/src/ExcelReader.Core/PublicAPI/PublicAPI.Unshipped.txt +++ b/src/ExcelReader.Core/PublicAPI/PublicAPI.Unshipped.txt @@ -56,6 +56,26 @@ ExcelReader.Core.Writer.Xlsb.XlsbSheetWriter.BeginTableAsync(string! name, Syste ExcelReader.Core.Writer.Xlsb.XlsbSheetWriter.EndTable() -> void static ExcelReader.Core.Writer.SheetWriterExtensions.WriteTableAsync(this ExcelReader.Core.Writer.ITableSheetWriter! sheet, System.Collections.Generic.IAsyncEnumerable! records, string! name, ExcelReader.Core.Writer.ExcelRecordLayout! layout, ExcelReader.Core.Writer.ExcelTableOptions? options = null, System.Threading.CancellationToken ct = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.ValueTask static ExcelReader.Core.Writer.SheetWriterExtensions.WriteTableAsync(this ExcelReader.Core.Writer.ITableSheetWriter! sheet, System.Collections.Generic.IEnumerable! records, string! name, ExcelReader.Core.Writer.ExcelRecordLayout! layout, ExcelReader.Core.Writer.ExcelTableOptions? options = null, System.Threading.CancellationToken ct = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.ValueTask +ExcelReader.Core.Writer.ParallelWriteOptions +ExcelReader.Core.Writer.ParallelWriteOptions.BatchRows.get -> int +ExcelReader.Core.Writer.ParallelWriteOptions.BatchRows.init -> void +ExcelReader.Core.Writer.ParallelWriteOptions.DegreeOfParallelism.get -> int +ExcelReader.Core.Writer.ParallelWriteOptions.DegreeOfParallelism.init -> void +ExcelReader.Core.Writer.ParallelWriteOptions.ParallelWriteOptions() -> void +static ExcelReader.Core.Writer.ParallelWriteOptions.Default.get -> ExcelReader.Core.Writer.ParallelWriteOptions! +ExcelReader.Core.Writer.ParallelSheetWriterExtensions +static ExcelReader.Core.Writer.ParallelSheetWriterExtensions.WriteRecordsParallelAsync(this ExcelReader.Core.Writer.Xlsx.XlsxSheetWriter! sheet, System.Collections.Generic.IAsyncEnumerable! records, ExcelReader.Core.Writer.ExcelRecordLayout! layout, ExcelReader.Core.Writer.ParallelWriteOptions? options = null, System.Threading.CancellationToken ct = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.ValueTask +static ExcelReader.Core.Writer.ParallelSheetWriterExtensions.WriteRecordsParallelAsync(this ExcelReader.Core.Writer.Xlsx.XlsxSheetWriter! sheet, System.Collections.Generic.IAsyncEnumerable! records, System.Action! writeRow, ExcelReader.Core.Writer.ParallelWriteOptions? options = null, System.Threading.CancellationToken ct = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.ValueTask +static ExcelReader.Core.Writer.ParallelSheetWriterExtensions.WriteRecordsParallelAsync(this ExcelReader.Core.Writer.Xlsx.XlsxSheetWriter! sheet, System.Collections.Generic.IEnumerable! records, ExcelReader.Core.Writer.ExcelRecordLayout! layout, ExcelReader.Core.Writer.ParallelWriteOptions? options = null, System.Threading.CancellationToken ct = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.ValueTask +static ExcelReader.Core.Writer.ParallelSheetWriterExtensions.WriteRecordsParallelAsync(this ExcelReader.Core.Writer.Xlsx.XlsxSheetWriter! sheet, System.Collections.Generic.IEnumerable! records, System.Action! writeRow, ExcelReader.Core.Writer.ParallelWriteOptions? options = null, System.Threading.CancellationToken ct = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.ValueTask +static ExcelReader.Core.Writer.ParallelSheetWriterExtensions.WriteTableParallelAsync(this ExcelReader.Core.Writer.Xlsx.XlsxSheetWriter! sheet, System.Collections.Generic.IAsyncEnumerable! records, string! name, ExcelReader.Core.Writer.ExcelRecordLayout! layout, ExcelReader.Core.Writer.ExcelTableOptions? tableOptions = null, ExcelReader.Core.Writer.ParallelWriteOptions? options = null, System.Threading.CancellationToken ct = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.ValueTask +static ExcelReader.Core.Writer.ParallelSheetWriterExtensions.WriteTableParallelAsync(this ExcelReader.Core.Writer.Xlsx.XlsxSheetWriter! sheet, System.Collections.Generic.IEnumerable! records, string! name, ExcelReader.Core.Writer.ExcelRecordLayout! layout, ExcelReader.Core.Writer.ExcelTableOptions? tableOptions = null, ExcelReader.Core.Writer.ParallelWriteOptions? options = null, System.Threading.CancellationToken ct = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.ValueTask +static ExcelReader.Core.Writer.ParallelSheetWriterExtensions.WriteRecordsParallelAsync(this ExcelReader.Core.Writer.Xlsb.XlsbSheetWriter! sheet, System.Collections.Generic.IAsyncEnumerable! records, ExcelReader.Core.Writer.ExcelRecordLayout! layout, ExcelReader.Core.Writer.ParallelWriteOptions? options = null, System.Threading.CancellationToken ct = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.ValueTask +static ExcelReader.Core.Writer.ParallelSheetWriterExtensions.WriteRecordsParallelAsync(this ExcelReader.Core.Writer.Xlsb.XlsbSheetWriter! sheet, System.Collections.Generic.IAsyncEnumerable! records, System.Action! writeRow, ExcelReader.Core.Writer.ParallelWriteOptions? options = null, System.Threading.CancellationToken ct = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.ValueTask +static ExcelReader.Core.Writer.ParallelSheetWriterExtensions.WriteRecordsParallelAsync(this ExcelReader.Core.Writer.Xlsb.XlsbSheetWriter! sheet, System.Collections.Generic.IEnumerable! records, ExcelReader.Core.Writer.ExcelRecordLayout! layout, ExcelReader.Core.Writer.ParallelWriteOptions? options = null, System.Threading.CancellationToken ct = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.ValueTask +static ExcelReader.Core.Writer.ParallelSheetWriterExtensions.WriteRecordsParallelAsync(this ExcelReader.Core.Writer.Xlsb.XlsbSheetWriter! sheet, System.Collections.Generic.IEnumerable! records, System.Action! writeRow, ExcelReader.Core.Writer.ParallelWriteOptions? options = null, System.Threading.CancellationToken ct = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.ValueTask +static ExcelReader.Core.Writer.ParallelSheetWriterExtensions.WriteTableParallelAsync(this ExcelReader.Core.Writer.Xlsb.XlsbSheetWriter! sheet, System.Collections.Generic.IAsyncEnumerable! records, string! name, ExcelReader.Core.Writer.ExcelRecordLayout! layout, ExcelReader.Core.Writer.ExcelTableOptions? tableOptions = null, ExcelReader.Core.Writer.ParallelWriteOptions? options = null, System.Threading.CancellationToken ct = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.ValueTask +static ExcelReader.Core.Writer.ParallelSheetWriterExtensions.WriteTableParallelAsync(this ExcelReader.Core.Writer.Xlsb.XlsbSheetWriter! sheet, System.Collections.Generic.IEnumerable! records, string! name, ExcelReader.Core.Writer.ExcelRecordLayout! layout, ExcelReader.Core.Writer.ExcelTableOptions? tableOptions = null, ExcelReader.Core.Writer.ParallelWriteOptions? options = null, System.Threading.CancellationToken ct = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.ValueTask ExcelReader.Core.Reader.SheetParallel ExcelReader.Core.Reader.SheetParallelOptions ExcelReader.Core.Reader.SheetParallelOptions.DegreeOfParallelism.get -> int @@ -140,3 +160,5 @@ ExcelReader.Core.Reader.SheetDataReaderOptions.HeaderRow.get -> int ExcelReader.Core.Reader.SheetDataReaderOptions.HeaderRow.init -> void ExcelReader.Core.Reader.SheetDataReaderOptions.SheetDataReaderOptions() -> void static ExcelReader.Core.Reader.SheetDataReaderOptions.Default.get -> ExcelReader.Core.Reader.SheetDataReaderOptions! +ExcelReader.Core.Reader.ExcelReaderOptions.VerifyZipChecksums.get -> bool +ExcelReader.Core.Reader.ExcelReaderOptions.VerifyZipChecksums.init -> void diff --git a/src/ExcelReader.Core/Reader/ExcelLimitExceededException.cs b/src/ExcelReader.Core/Reader/ExcelLimitExceededException.cs index f8dc22c6..ade3e303 100644 --- a/src/ExcelReader.Core/Reader/ExcelLimitExceededException.cs +++ b/src/ExcelReader.Core/Reader/ExcelLimitExceededException.cs @@ -29,10 +29,10 @@ public ExcelLimitExceededException(string message, Exception innerException) /// Initializes a new instance of the class for a specific exceeded limit, building the message from the given values and populating , , and . /// The name of the exceeded limit (e.g. the corresponding property name). - /// The configured limit, in bytes. - /// The actual size, in bytes, that exceeded the limit. + /// The limit, in the unit the named limit is expressed in (bytes, rows, columns, entries, ...). + /// The value, in the same unit, that exceeded the limit. public ExcelLimitExceededException(string limitName, long limit, long actual) - : base($"{limitName} limit exceeded: {actual} bytes exceeds {limit} bytes.") + : base($"{limitName} limit exceeded: {actual} exceeds {limit}.") { LimitName = limitName; Limit = limit; @@ -42,10 +42,10 @@ public ExcelLimitExceededException(string limitName, long limit, long actual) /// Gets the name of the limit that was exceeded. public string LimitName { get; } = string.Empty; - /// Gets the configured limit, in bytes, that was exceeded. + /// Gets the limit that was exceeded, in the unit is expressed in. public long Limit { get; } - /// Gets the actual size, in bytes, that exceeded . + /// Gets the value, in the same unit, that exceeded . public long Actual { get; } } } diff --git a/src/ExcelReader.Core/Reader/ExcelReaderOptions.cs b/src/ExcelReader.Core/Reader/ExcelReaderOptions.cs index 3e4f5470..be69e168 100644 --- a/src/ExcelReader.Core/Reader/ExcelReaderOptions.cs +++ b/src/ExcelReader.Core/Reader/ExcelReaderOptions.cs @@ -35,6 +35,13 @@ public sealed record ExcelReaderOptions /// with work already saturating the CPU. public bool PrefetchDecompression { get; init; } + /// Gets a value indicating whether each ZIP entry read from an XLSX or XLSB package is + /// checked against the CRC-32 its central directory records. Defaults to . + /// A mismatch throws when the entry's last bytes are read, so + /// rows before the damage may already have been returned. Turn it off to read a package whose + /// producer wrote wrong checksums; damaged data then surfaces only if it also breaks parsing. + public bool VerifyZipChecksums { get; init; } = true; + /// /// Gets a value indicating whether text cells not served by a shared-string table (inline and /// formula-string cells, and every CSV field) are deduplicated through a small content-keyed cache diff --git a/src/ExcelReader.Core/Reader/Internal/FastDouble.cs b/src/ExcelReader.Core/Reader/Internal/FastDouble.cs index 1051fb16..f602ec77 100644 --- a/src/ExcelReader.Core/Reader/Internal/FastDouble.cs +++ b/src/ExcelReader.Core/Reader/Internal/FastDouble.cs @@ -12,10 +12,16 @@ internal static class FastDouble private const int MaxPowerOfFive = 64; [MethodImpl(MethodImplOptions.AggressiveInlining)] - public static bool TryParse(ReadOnlySpan s, out double value) => TryParse(s, MaxMantissaDigits, out value); + public static bool TryParse(ReadOnlySpan s, out double value) + { + return TryParse(s, MaxMantissaDigits, out value); + } [MethodImpl(MethodImplOptions.AggressiveInlining)] - public static bool TryParseLossless(ReadOnlySpan s, out double value) => TryParse(s, MaxLosslessDigits, out value); + public static bool TryParseLossless(ReadOnlySpan s, out double value) + { + return TryParse(s, MaxLosslessDigits, out value); + } [MethodImpl(MethodImplOptions.AggressiveInlining)] private static bool TryParse(ReadOnlySpan s, int maxDigits, out double value) diff --git a/src/ExcelReader.Core/Reader/Internal/WorkbookLookups.cs b/src/ExcelReader.Core/Reader/Internal/WorkbookLookups.cs index 8d13eb95..9df6af38 100644 --- a/src/ExcelReader.Core/Reader/Internal/WorkbookLookups.cs +++ b/src/ExcelReader.Core/Reader/Internal/WorkbookLookups.cs @@ -81,13 +81,14 @@ internal static ZipEntryRef GetWorksheetEntry(ZipIndex memZip, string path) internal static LimitedReadStream Wrap( Stream opened, DecompressedByteCounter counter, ExcelReaderOptions options, - string entryLimitName, long entryLimit, long uncompressedSize) + string entryLimitName, long entryLimit, long uncompressedSize, uint? expectedCrc = null) { if (!options.PrefetchDecompression || uncompressedSize < PrefetchMinUncompressedSize) { - return new LimitedReadStream(opened, uncompressedSize, counter, entryLimitName, entryLimit); + return new LimitedReadStream(opened, uncompressedSize, counter, entryLimitName, entryLimit, expectedCrc); } - return new LimitedReadStream(new PrefetchStream(opened), uncompressedSize, counter, entryLimitName, entryLimit); + return new LimitedReadStream( + new PrefetchStream(opened, uncompressedSize, expectedCrc), uncompressedSize, counter, entryLimitName, entryLimit); } internal static int InitialBufferCapacity(long entryLength) diff --git a/src/ExcelReader.Core/Reader/Xls/XlsWorkbook.Enumerator.cs b/src/ExcelReader.Core/Reader/Xls/XlsWorkbook.Enumerator.cs index 84beb389..59815c39 100644 --- a/src/ExcelReader.Core/Reader/Xls/XlsWorkbook.Enumerator.cs +++ b/src/ExcelReader.Core/Reader/Xls/XlsWorkbook.Enumerator.cs @@ -294,6 +294,12 @@ private void ParseFormula(ReadOnlySpan data) _acc.AddError(col, style, result[2]); break; case 0: + // The first cell of a shared or array formula, or of a data table, has that + // definition's record between its FORMULA and its STRING. + if (_cursor.PeekId() is Rec.ShrFmla or Rec.Array or Rec.Table) + { + _cursor.TryReadRecord(out _, out _); + } if (_cursor.PeekId() == Rec.StringRec && _cursor.TryReadRecord(out _, out ReadOnlySpan str) && str.Length >= 3) { int start = _acc.ValueLength; diff --git a/src/ExcelReader.Core/Reader/Xls/XlsWorkbook.cs b/src/ExcelReader.Core/Reader/Xls/XlsWorkbook.cs index 0953fcc8..cb70b660 100644 --- a/src/ExcelReader.Core/Reader/Xls/XlsWorkbook.cs +++ b/src/ExcelReader.Core/Reader/Xls/XlsWorkbook.cs @@ -507,6 +507,9 @@ private static class Rec internal const int Blank = 0x0201; internal const int MulBlank = 0x00BE; internal const int StringRec = 0x0207; + internal const int ShrFmla = 0x04BC; + internal const int Array = 0x0221; + internal const int Table = 0x0236; } } } diff --git a/src/ExcelReader.Core/Reader/Xlsx/XlsxWorkbook.Enumerator.cs b/src/ExcelReader.Core/Reader/Xlsx/XlsxWorkbook.Enumerator.cs index 23b97aea..a36b7ac1 100644 --- a/src/ExcelReader.Core/Reader/Xlsx/XlsxWorkbook.Enumerator.cs +++ b/src/ExcelReader.Core/Reader/Xlsx/XlsxWorkbook.Enumerator.cs @@ -878,7 +878,7 @@ private void EmitScalarValue(Kind kind, ReadOnlySpan v, int col, int style private void EmitScalarValueFast(Kind kind, ReadOnlySpan v, int valueStart, int col, int style) { - if (kind == Kind.IsoDate || (kind == Kind.Formula && (v.IndexOf((byte)'&') >= 0 || v.IndexOf("_x"u8) >= 0))) + if (kind == Kind.IsoDate || (kind == Kind.Formula && (v.IndexOfAny((byte)'&', (byte)'\r') >= 0 || v.IndexOf("_x"u8) >= 0))) { EmitScalarValue(kind, v, col, style); return; @@ -950,13 +950,42 @@ private static ReadOnlySpan ElementText(ReadOnlySpan inner, ReadOnly int s = inner.IndexOf(openTag); if (s < 0) { - return default; + return AttributedElementText(inner, openTag, closeTag); } s += openTag.Length; int e = inner[s..].IndexOf(closeTag); return e < 0 ? default : inner.Slice(s, e); } + // Excel writes for a formula's text result with whitespace at either end. + [MethodImpl(MethodImplOptions.NoInlining)] + private static ReadOnlySpan AttributedElementText(ReadOnlySpan inner, ReadOnlySpan openTag, ReadOnlySpan closeTag) + { + ReadOnlySpan name = openTag[..^1]; + int from = 0; + while (true) + { + int s = inner[from..].IndexOf(name); + if (s < 0) + { + return default; + } + int afterName = from + s + name.Length; + if (afterName < inner.Length && inner[afterName] is (byte)' ' or (byte)'\t' or (byte)'\r' or (byte)'\n') + { + int gt = inner[afterName..].IndexOf((byte)'>'); + if (gt < 0 || inner[afterName + gt - 1] == (byte)'/') + { + return default; + } + int textStart = afterName + gt + 1; + int e = inner[textStart..].IndexOf(closeTag); + return e < 0 ? default : inner.Slice(textStart, e); + } + from = afterName; + } + } + private void AppendDecoded(ReadOnlySpan src) { if (src.IsEmpty) diff --git a/src/ExcelReader.Core/Reader/Xlsx/XlsxXml.cs b/src/ExcelReader.Core/Reader/Xlsx/XlsxXml.cs index 08465db1..4b89e551 100644 --- a/src/ExcelReader.Core/Reader/Xlsx/XlsxXml.cs +++ b/src/ExcelReader.Core/Reader/Xlsx/XlsxXml.cs @@ -8,6 +8,8 @@ namespace ExcelReader.Core.Reader.Xlsx { internal static class XlsxXml { + private static readonly SearchValues MarkupOrEscape = SearchValues.Create("<&_\r"u8); + internal static int ParseIntOr(ReadOnlySpan source, int fallback) { return Utf8Parser.TryParse(source, out int value, out _) ? value : fallback; @@ -77,7 +79,7 @@ public static int Decode(ReadOnlySpan src, Span dest) int w = 0; while (!src.IsEmpty) { - int special = src.IndexOfAny((byte)'&', (byte)'<'); + int special = src.IndexOfAny((byte)'&', (byte)'<', (byte)'\r'); if (special < 0) { src.CopyTo(dest[w..]); @@ -87,6 +89,15 @@ public static int Decode(ReadOnlySpan src, Span dest) w += special; src = src[special..]; + // XML end-of-line handling: a literal CR LF pair, or a CR alone, reads as one LF. A carriage + // return meant as data arrives as or _x000D_, which this does not touch. + if (src[0] == (byte)'\r') + { + dest[w++] = (byte)'\n'; + src = src[(src.Length > 1 && src[1] == (byte)'\n' ? 2 : 1)..]; + continue; + } + if (src[0] == (byte)'<') { if (!src.StartsWith(" body, Span dest, ReadO return raw.Length; } - // Finds the first '<' or '&'. Only text with a '_' ahead of it can hold an _xHHHH_ escape, so the - // usual underscore-free text costs one scan instead of a second pass for "_x". + // Finds the first '<', '&' or carriage return. Only text with a '_' ahead of it can hold an _xHHHH_ + // escape, so the usual underscore-free text costs one scan instead of a second pass for "_x". + // A carriage return stops the scan because text holding one needs Decode's end-of-line handling; + // callers that expect a closing tag at the returned index then fall back to it. internal static int IndexOfMarkup(ReadOnlySpan text, out bool mayEscape) { - int i = text.IndexOfAny((byte)'<', (byte)'&', (byte)'_'); + int i = text.IndexOfAny(MarkupOrEscape); mayEscape = i >= 0 && text[i] == (byte)'_'; if (!mayEscape) { return i; } - int rest = text[i..].IndexOfAny((byte)'<', (byte)'&'); + int rest = text[i..].IndexOfAny((byte)'<', (byte)'&', (byte)'\r'); return rest < 0 ? -1 : i + rest; } diff --git a/src/ExcelReader.Core/Reader/Zip/LimitedReadStream.cs b/src/ExcelReader.Core/Reader/Zip/LimitedReadStream.cs index b805477c..fff4ad26 100644 --- a/src/ExcelReader.Core/Reader/Zip/LimitedReadStream.cs +++ b/src/ExcelReader.Core/Reader/Zip/LimitedReadStream.cs @@ -1,3 +1,5 @@ +using ExcelReader.Core.Writer.Zip; + namespace ExcelReader.Core.Reader.Zip { internal sealed class DecompressedByteCounter @@ -37,24 +39,31 @@ internal void Add(long bytes) internal sealed class LimitedReadStream : Stream { + internal const string ChecksumMismatch = "The ZIP entry's data does not match its CRC-32."; + private readonly Stream _inner; private readonly DecompressedByteCounter? _totalCounter; private readonly DecompressedByteCounter? _entryCounter; private readonly long _declaredLength; + private readonly uint? _expectedCrc; private long _produced; + private uint _crc; // declaredLength is the entry's size from the central directory. Neither raw deflate nor a stored // entry can tell a cut-short stream from a complete one, so the length is the only end-of-data check. + // expectedCrc, when given, is compared once the declared length has been produced. internal LimitedReadStream( Stream inner, long declaredLength, DecompressedByteCounter? totalCounter, string entryLimitName = "", - long entryLimit = 0) + long entryLimit = 0, + uint? expectedCrc = null) { _inner = inner; _declaredLength = declaredLength; _totalCounter = totalCounter; + _expectedCrc = expectedCrc; _entryCounter = entryLimit > 0 ? new DecompressedByteCounter(entryLimit, entryLimitName) : null; } @@ -76,17 +85,17 @@ public override void Flush() public override int Read(byte[] buffer, int offset, int count) { - return Count(_inner.Read(buffer, offset, count), count); + return Count(_inner.Read(buffer, offset, count), buffer.AsSpan(offset, count)); } public override int Read(Span buffer) { - return Count(_inner.Read(buffer), buffer.Length); + return Count(_inner.Read(buffer), buffer); } public override async ValueTask ReadAsync(Memory buffer, CancellationToken cancellationToken = default) { - return Count(await _inner.ReadAsync(buffer, cancellationToken).ConfigureAwait(false), buffer.Length); + return Count(await _inner.ReadAsync(buffer, cancellationToken).ConfigureAwait(false), buffer.Span); } public override Task ReadAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken) @@ -124,11 +133,11 @@ public override async ValueTask DisposeAsync() await base.DisposeAsync().ConfigureAwait(false); } - private int Count(int read, int requested) + private int Count(int read, ReadOnlySpan buffer) { if (read <= 0) { - if (requested > 0 && _produced != _declaredLength) + if (buffer.Length > 0 && _produced != _declaredLength) { throw new InvalidDataException("The ZIP entry produced less data than its declared uncompressed size."); } @@ -141,6 +150,15 @@ private int Count(int read, int requested) } _totalCounter?.Add(read); _entryCounter?.Add(read); + if (_expectedCrc is not uint expected) + { + return read; + } + _crc = Crc32.Append(_crc, buffer[..read]); + if (_produced == _declaredLength && _crc != expected) + { + throw new InvalidDataException(ChecksumMismatch); + } return read; } } diff --git a/src/ExcelReader.Core/Reader/Zip/PrefetchStream.cs b/src/ExcelReader.Core/Reader/Zip/PrefetchStream.cs index b7f7469f..5d8eea70 100644 --- a/src/ExcelReader.Core/Reader/Zip/PrefetchStream.cs +++ b/src/ExcelReader.Core/Reader/Zip/PrefetchStream.cs @@ -2,6 +2,7 @@ using System.Diagnostics.CodeAnalysis; using System.Runtime.ExceptionServices; using System.Threading.Channels; +using ExcelReader.Core.Writer.Zip; namespace ExcelReader.Core.Reader.Zip { @@ -14,15 +15,21 @@ internal sealed class PrefetchStream : Stream private readonly Channel<(byte[] Buffer, int Length)> _channel; private readonly CancellationTokenSource _cts; private readonly Task _producer; + private readonly long _declaredLength; + private readonly uint? _expectedCrc; private byte[]? _currentBuffer; private int _currentLength; private int _currentOffset; private ExceptionDispatchInfo? _producerException; private bool _disposed; - internal PrefetchStream(Stream inner) + // With expectedCrc, the producer checksums what it reads and fails instead of handing over the + // chunk that completes declaredLength, so the check runs off the consumer's thread. + internal PrefetchStream(Stream inner, long declaredLength = 0, uint? expectedCrc = null) { _inner = inner; + _declaredLength = declaredLength; + _expectedCrc = expectedCrc; _channel = Channel.CreateBounded<(byte[] Buffer, int Length)>(new BoundedChannelOptions(ChannelCapacity) { SingleReader = true, @@ -190,6 +197,8 @@ private int ConsumeAvailableChunks(Span destination) private async Task ProduceAsync(CancellationToken token) { byte[]? pending = null; + long produced = 0; + uint crc = 0; try { while (true) @@ -200,6 +209,15 @@ private async Task ProduceAsync(CancellationToken token) { break; } + if (_expectedCrc is uint expected) + { + crc = Crc32.Append(crc, pending.AsSpan(0, read)); + produced += read; + if (produced == _declaredLength && crc != expected) + { + throw new InvalidDataException(LimitedReadStream.ChecksumMismatch); + } + } await _channel.Writer.WriteAsync((pending, read), token).ConfigureAwait(false); pending = null; } diff --git a/src/ExcelReader.Core/Reader/Zip/ZipIndex.cs b/src/ExcelReader.Core/Reader/Zip/ZipIndex.cs index a796ef9d..e1215363 100644 --- a/src/ExcelReader.Core/Reader/Zip/ZipIndex.cs +++ b/src/ExcelReader.Core/Reader/Zip/ZipIndex.cs @@ -1,9 +1,11 @@ using System.Buffers; using System.Buffers.Binary; +using System.Diagnostics.CodeAnalysis; using System.Runtime.InteropServices; using ExcelReader.Core.Reader.Internal; using ExcelReader.Core.Reader.Sources; using ExcelReader.Core.Reader.Zip.Inflate; +using ExcelReader.Core.Writer.Zip; namespace ExcelReader.Core.Reader.Zip { @@ -15,6 +17,7 @@ internal readonly struct ZipEntryRef internal long LocalHeaderOffset { get; init; } internal long CompressedSize { get; init; } internal long UncompressedSize { get; init; } + internal uint Crc { get; init; } internal ushort Method { get; init; } internal ushort Flags { get; init; } } @@ -65,9 +68,11 @@ internal sealed class ZipIndex : IDisposable private readonly ReadOnlyMemory _directory; private readonly byte[]? _directoryRented; private readonly ZipEntryRef[] _entries; + private readonly bool _verifyChecksums; private int _disposed; - private ZipIndex(ByteSource source, ReadOnlyMemory directory, byte[]? directoryRented, ZipEntryRef[] entries, int count) + private ZipIndex( + ByteSource source, ReadOnlyMemory directory, byte[]? directoryRented, ZipEntryRef[] entries, int count, bool verifyChecksums) { _source = source; HasMemory = source.TryGetMemory(out _file); @@ -75,6 +80,7 @@ private ZipIndex(ByteSource source, ReadOnlyMemory directory, byte[]? dire _directoryRented = directoryRented; _entries = entries; Count = count; + _verifyChecksums = verifyChecksums; } internal int Count { get; } @@ -237,7 +243,7 @@ private static ZipIndex Build(ByteSource source, ReadOnlyMemory directory, { int count = WalkCentralDirectory(directory.Span, ref entries, options); ThrowIfDuplicateEntryNames(directory, entries, count); - return new ZipIndex(source, directory, directoryRented, entries, count); + return new ZipIndex(source, directory, directoryRented, entries, count, options.VerifyZipChecksums); } catch { @@ -335,6 +341,11 @@ internal ZipPart OpenPart(in ZipEntryRef entry, DecompressedByteCounter counter, { part = ReadPart(dataOffset, entry); } + if (_verifyChecksums && Crc32.Append(0, part.Memory.Span) != entry.Crc) + { + part.Dispose(); + ThrowChecksumMismatch(); + } counter.Add(entry.UncompressedSize); return part; } @@ -371,6 +382,10 @@ internal async ValueTask OpenPartAsync( throw new InvalidDataException("The ZIP entry produced more data than its declared uncompressed size."); } } + if (_verifyChecksums && Crc32.Append(0, rented.AsSpan(0, size)) != entry.Crc) + { + ThrowChecksumMismatch(); + } } catch { @@ -416,7 +431,8 @@ private LimitedReadStream WrapEntryStream( ? ToReadableMemoryStream(CompressedSlice(dataOffset, entry.CompressedSize)) : new ByteSourceStream(_source, dataOffset, entry.CompressedSize); Stream opened = entry.Method == 0 ? raw : new InflateStream(raw); - return WorkbookLookups.Wrap(opened, counter, options, entryLimitName, entryLimit, entry.UncompressedSize); + return WorkbookLookups.Wrap( + opened, counter, options, entryLimitName, entryLimit, entry.UncompressedSize, _verifyChecksums ? entry.Crc : null); } private static void ThrowIfPartTooLarge(in ZipEntryRef entry, DecompressedByteCounter counter, string entryLimitName, long entryLimit) @@ -694,6 +710,7 @@ private static int WalkCentralDirectory(ReadOnlySpan directory, ref ZipEnt LocalHeaderOffset = localOffset, CompressedSize = compressed, UncompressedSize = uncompressed, + Crc = fields.Crc, Method = fields.Method, Flags = fields.Flags, }; @@ -712,6 +729,7 @@ private readonly struct CdFixedFields internal ushort NameLength { get; init; } internal ushort ExtraLength { get; init; } internal ushort CommentLength { get; init; } + internal uint Crc { get; init; } internal uint CompressedSize { get; init; } internal uint UncompressedSize { get; init; } internal uint LocalHeaderOffset { get; init; } @@ -723,6 +741,7 @@ private static CdFixedFields ParseCentralDirectoryFixed(ReadOnlySpan recor { Flags = BinaryPrimitives.ReadUInt16LittleEndian(record[8..]), Method = BinaryPrimitives.ReadUInt16LittleEndian(record[10..]), + Crc = BinaryPrimitives.ReadUInt32LittleEndian(record[16..]), CompressedSize = BinaryPrimitives.ReadUInt32LittleEndian(record[20..]), UncompressedSize = BinaryPrimitives.ReadUInt32LittleEndian(record[24..]), NameLength = BinaryPrimitives.ReadUInt16LittleEndian(record[28..]), @@ -831,6 +850,12 @@ private static ZipPart InflateToPart(ReadOnlyMemory compressed, long uncom } } + [DoesNotReturn] + private static void ThrowChecksumMismatch() + { + throw new InvalidDataException(LimitedReadStream.ChecksumMismatch); + } + private static MemoryStream ToReadableMemoryStream(ReadOnlyMemory compressed) { if (MemoryMarshal.TryGetArray(compressed, out ArraySegment segment)) diff --git a/src/ExcelReader.Core/Writer/IWorkbookWriter.cs b/src/ExcelReader.Core/Writer/IWorkbookWriter.cs index 176a7d42..f39559bf 100644 --- a/src/ExcelReader.Core/Writer/IWorkbookWriter.cs +++ b/src/ExcelReader.Core/Writer/IWorkbookWriter.cs @@ -77,6 +77,8 @@ public interface IWorkbookWriter : IDisposable, IAsyncDisposable /// style; index 1 is always the builtin date style every date cell already uses. /// /// The number format/bold/italic combination to register. + /// The style's number format is longer than 255 characters. + /// The workbook already has 200 distinct number formats, the most Excel opens. /// The style's index, for use with and . int AddStyle(CellStyle style); } @@ -131,7 +133,7 @@ public interface ISheetWriter : IDisposable, IAsyncDisposable /// /// The 0-based column index. /// The style to apply, from . - /// is negative, or is negative or was never returned by . + /// is negative or beyond the format's last column, or is negative or was never returned by . /// The first row has already been started. void SetColumnStyle(int columnIndex, int styleId); @@ -141,7 +143,7 @@ public interface ISheetWriter : IDisposable, IAsyncDisposable /// /// The 0-based column index. /// The column width, in characters. - /// or is negative. + /// is negative or beyond the format's last column, or is negative or greater than 255. /// The first row has already been started. void SetColumnWidth(int columnIndex, double width); diff --git a/src/ExcelReader.Core/Writer/Internal/CellFormatter.cs b/src/ExcelReader.Core/Writer/Internal/CellFormatter.cs index d3616f74..88bd04ee 100644 --- a/src/ExcelReader.Core/Writer/Internal/CellFormatter.cs +++ b/src/ExcelReader.Core/Writer/Internal/CellFormatter.cs @@ -10,70 +10,93 @@ namespace ExcelReader.Core.Writer.Internal { internal static class CellFormatter { + // A raw carriage return is escaped too: an XML parser reads it as a line feed. private const string Special = "&<>\"'_" + - "\u0000\u0001\u0002\u0003\u0004\u0005\u0006\u0007\u0008\u000B\u000C" + + "\u0000\u0001\u0002\u0003\u0004\u0005\u0006\u0007\u0008\u000B\u000C\u000D" + "\u000E\u000F\u0010\u0011\u0012\u0013\u0014\u0015\u0016\u0017\u0018\u0019\u001A\u001B\u001C\u001D\u001E\u001F"; - private static readonly SearchValues SpecialChars = SearchValues.Create(Special); + // U+FFFE and U+FFFF are not XML characters, so they are written as _xHHHH_ as well. + private static readonly SearchValues SpecialChars = SearchValues.Create(Special + "\uFFFE\uFFFF"); - private static readonly SearchValues SpecialBytes = SearchValues.Create(Encoding.ASCII.GetBytes(Special)); + // 0xEF leads the UTF-8 form of U+FFFE and U+FFFF, and of every other U+F000..U+FFFF character. + private static readonly SearchValues SpecialBytes = SearchValues.Create([.. Encoding.ASCII.GetBytes(Special), 0xEF]); - [SkipLocalsInit] - private static void WriteRef(BiffBuffer xml, int columnIndex, int rowNumber) - { - Span buf = stackalloc byte[10]; - int len = ColumnName.Write(buf, columnIndex); - Utf8Formatter.TryFormat(rowNumber, buf[len..], out int rowLen); - xml.Write(buf[..(len + rowLen)]); - } + // + private const int MaxCellOpenLength = 48; + // -2.2250738585072014E-308 + private const int MaxValueLength = 40; - private static void WriteCellOpen(BiffBuffer xml, int columnIndex, int rowNumber, bool includeReference, - int styleId, ReadOnlySpan typeAttr, bool selfClose) + /// + /// Writes the cell's start tag, through its closing >, at the start of + /// and returns its length. Cells are written whole into one span: separate small writes to the + /// buffer, each checking capacity, cost more than formatting the cell. + /// + private static int WriteCellOpen(Span dest, int columnIndex, int rowNumber, bool includeReference, + int styleId, ReadOnlySpan typeAttr) { - xml.Write(""u8 : ">"u8); + typeAttr.CopyTo(dest[pos..]); + pos += typeAttr.Length; + dest[pos++] = (byte)'>'; + return pos; } internal static void WriteEmpty(BiffBuffer xml, int columnIndex, int rowNumber, bool includeReference, int styleId = 0) { - WriteCellOpen(xml, columnIndex, rowNumber, includeReference, styleId, default, selfClose: true); + Span dest = xml.GetSpan(MaxCellOpenLength + 1); + int pos = WriteCellOpen(dest, columnIndex, rowNumber, includeReference, styleId, default); + "/>"u8.CopyTo(dest[(pos - 1)..]); + xml.Advance(pos + 1); } internal static void WriteString(BiffBuffer xml, ReadOnlySpan value, int columnIndex, int rowNumber, bool includeReference, int styleId = 0) { - WriteCellOpen(xml, columnIndex, rowNumber, includeReference, styleId, " t=\"inlineStr\""u8, selfClose: false); - xml.Write(HasEdgeWhitespace(value) ? ""u8 : ""u8); + ReadOnlySpan textOpen = HasEdgeWhitespace(value) ? ""u8 : ""u8; + Span dest = xml.GetSpan(MaxCellOpenLength + textOpen.Length); + int pos = WriteCellOpen(dest, columnIndex, rowNumber, includeReference, styleId, " t=\"inlineStr\""u8); + textOpen.CopyTo(dest[pos..]); + xml.Advance(pos + textOpen.Length); WriteEscaped(xml, value); xml.Write(""u8); } internal static void WritePlainUtf8String(BiffBuffer xml, ReadOnlySpan utf8, int columnIndex, int rowNumber, bool includeReference, int styleId = 0) { - WriteCellOpen(xml, columnIndex, rowNumber, includeReference, styleId, " t=\"inlineStr\""u8, selfClose: false); - xml.Write(""u8); - xml.Write(utf8); - xml.Write(""u8); + Span dest = xml.GetSpan(MaxCellOpenLength + 20 + utf8.Length); + int pos = WriteCellOpen(dest, columnIndex, rowNumber, includeReference, styleId, " t=\"inlineStr\""u8); + ""u8.CopyTo(dest[pos..]); + pos += 7; + utf8.CopyTo(dest[pos..]); + pos += utf8.Length; + ""u8.CopyTo(dest[pos..]); + xml.Advance(pos + 13); } /// /// True when would emit 's own bytes unchanged: /// non-empty, within the cell limit, valid UTF-8, nothing to escape, and no whitespace at either - /// end. Every escaped character is ASCII, and UTF-8 multi-byte sequences never contain an ASCII - /// byte, so a byte search over the same set is exact. + /// end. Every escaped ASCII character is searched as its own byte, which no UTF-8 multi-byte + /// sequence contains; text with a character from U+F000 up is never plain, so that U+FFFE and + /// U+FFFF are escaped. /// internal static bool IsPlainUtf8Text(ReadOnlySpan utf8) { @@ -97,101 +120,77 @@ private static bool HasEdgeWhitespace(ReadOnlySpan value) internal static void WriteSharedString(BiffBuffer xml, int sharedStringIndex, int columnIndex, int rowNumber, bool includeReference, int styleId = 0) { - WriteCellOpen(xml, columnIndex, rowNumber, includeReference, styleId, " t=\"s\""u8, selfClose: false); - xml.Write(""u8); - WriteValue(xml, sharedStringIndex, sizeHint: 16); - xml.Write(""u8); + Span dest = xml.GetSpan(MaxCellOpenLength + MaxValueLength); + int pos = WriteValueOpen(dest, columnIndex, rowNumber, includeReference, styleId, " t=\"s\""u8); + Utf8Formatter.TryFormat(sharedStringIndex, dest[pos..], out int written); + WriteValueClose(xml, dest, pos + written); } internal static void WriteBool(BiffBuffer xml, bool value, int columnIndex, int rowNumber, bool includeReference, int styleId = 0) { - WriteCellOpen(xml, columnIndex, rowNumber, includeReference, styleId, " t=\"b\""u8, selfClose: false); - xml.Write(""u8); - xml.WriteByte(value ? (byte)'1' : (byte)'0'); - xml.Write(""u8); + Span dest = xml.GetSpan(MaxCellOpenLength + MaxValueLength); + int pos = WriteValueOpen(dest, columnIndex, rowNumber, includeReference, styleId, " t=\"b\""u8); + dest[pos] = value ? (byte)'1' : (byte)'0'; + WriteValueClose(xml, dest, pos + 1); } internal static void WriteDateTime(BiffBuffer xml, DateTime value, bool date1904, int columnIndex, int rowNumber, bool includeReference, int styleId) { - WriteCellOpen(xml, columnIndex, rowNumber, includeReference, styleId, default, selfClose: false); - xml.Write(""u8); - WriteValue(xml, ExcelEpoch.OADateToSerial(value.ToOADate(), date1904), sizeHint: 32); - xml.Write(""u8); - } - - internal static void WriteNumber(BiffBuffer xml, T value, int columnIndex, int rowNumber, bool includeReference, int styleId = 0) - where T : IUtf8SpanFormattable - { - WriteCellOpen(xml, columnIndex, rowNumber, includeReference, styleId, default, selfClose: false); - xml.Write(""u8); - WriteValue(xml, value, sizeHint: 64); - xml.Write(""u8); - } - - internal static void WriteNumber(BiffBuffer xml, int value, int columnIndex, int rowNumber, bool includeReference, int styleId = 0) - { - WriteCellOpen(xml, columnIndex, rowNumber, includeReference, styleId, default, selfClose: false); - xml.Write(""u8); - WriteValue(xml, value, sizeHint: 16); - xml.Write(""u8); - } - - internal static void WriteNumber(BiffBuffer xml, long value, int columnIndex, int rowNumber, bool includeReference, int styleId = 0) - { - WriteCellOpen(xml, columnIndex, rowNumber, includeReference, styleId, default, selfClose: false); - xml.Write(""u8); - WriteValue(xml, value, sizeHint: 32); - xml.Write(""u8); + WriteNumber(xml, ExcelEpoch.OADateToSerial(value.ToOADate(), date1904), columnIndex, rowNumber, includeReference, styleId); } internal static void WriteNumber(BiffBuffer xml, double value, int columnIndex, int rowNumber, bool includeReference, int styleId = 0) { - WriteCellOpen(xml, columnIndex, rowNumber, includeReference, styleId, default, selfClose: false); - xml.Write(""u8); - WriteValue(xml, value, sizeHint: 32); - xml.Write(""u8); - } - - internal static void WriteNumber(BiffBuffer xml, decimal value, int columnIndex, int rowNumber, bool includeReference, int styleId = 0) - { - WriteCellOpen(xml, columnIndex, rowNumber, includeReference, styleId, default, selfClose: false); - xml.Write(""u8); - WriteValue(xml, value, sizeHint: 64); - xml.Write(""u8); + CellValueGuards.ThrowIfNonFinite(value, nameof(value)); + Span dest = xml.GetSpan(MaxCellOpenLength + MaxValueLength); + int pos = WriteValueOpen(dest, columnIndex, rowNumber, includeReference, styleId, default); + TryFormatDouble(value, dest[pos..], out int written); + WriteValueClose(xml, dest, pos + written); } - private static void WriteValue(BiffBuffer xml, int value, int sizeHint) + internal static void WriteNumber(BiffBuffer xml, T value, int columnIndex, int rowNumber, bool includeReference, int styleId = 0) + where T : IUtf8SpanFormattable { - int size = sizeHint; - int written; - while (!Utf8Formatter.TryFormat(value, xml.GetSpan(size), out written)) + if (typeof(T) == typeof(double)) + { + WriteNumber(xml, Unsafe.As(ref value), columnIndex, rowNumber, includeReference, styleId); + return; + } + if (typeof(T) == typeof(float)) + { + CellValueGuards.ThrowIfNonFinite(Unsafe.As(ref value), nameof(value)); + } + int size = 64; + while (true) { + Span dest = xml.GetSpan(MaxCellOpenLength + size); + int pos = WriteValueOpen(dest, columnIndex, rowNumber, includeReference, styleId, default); + if (value.TryFormat(dest[pos..^ValueCloseLength], out int written, default, CultureInfo.InvariantCulture)) + { + if (!CellValueGuards.IsAlwaysFinite() && typeof(T) != typeof(float)) + { + CellValueGuards.ThrowIfNotFiniteNumberText(dest.Slice(pos, written), typeof(T), nameof(value)); + } + WriteValueClose(xml, dest, pos + written); + return; + } size = checked(size * 2); } - xml.Advance(written); } - private static void WriteValue(BiffBuffer xml, long value, int sizeHint) + private const int ValueCloseLength = 8; + + private static int WriteValueOpen(Span dest, int columnIndex, int rowNumber, bool includeReference, int styleId, ReadOnlySpan typeAttr) { - int size = sizeHint; - int written; - while (!Utf8Formatter.TryFormat(value, xml.GetSpan(size), out written)) - { - size = checked(size * 2); - } - xml.Advance(written); + int pos = WriteCellOpen(dest, columnIndex, rowNumber, includeReference, styleId, typeAttr); + ""u8.CopyTo(dest[pos..]); + return pos + 3; } - private static void WriteValue(BiffBuffer xml, double value, int sizeHint) + private static void WriteValueClose(BiffBuffer xml, Span dest, int pos) { - CellValueGuards.ThrowIfNonFinite(value, nameof(value)); - int size = sizeHint; - int written; - while (!TryFormatDouble(value, xml.GetSpan(size), out written)) - { - size = checked(size * 2); - } - xml.Advance(written); + ""u8.CopyTo(dest[pos..]); + xml.Advance(pos + ValueCloseLength); } /// @@ -249,33 +248,6 @@ private static bool TryFormatScaled(long scaled, int scale, Span destinati return true; } - private static void WriteValue(BiffBuffer xml, T value, int sizeHint) - where T : IUtf8SpanFormattable - { - if (typeof(T) == typeof(double)) - { - WriteValue(xml, Unsafe.As(ref value), sizeHint); - return; - } - if (typeof(T) == typeof(float)) - { - CellValueGuards.ThrowIfNonFinite(Unsafe.As(ref value), nameof(value)); - } - int size = sizeHint; - int written; - Span destination = xml.GetSpan(size); - while (!value.TryFormat(destination, out written, default, CultureInfo.InvariantCulture)) - { - size = checked(size * 2); - destination = xml.GetSpan(size); - } - if (!CellValueGuards.IsAlwaysFinite() && typeof(T) != typeof(double) && typeof(T) != typeof(float)) - { - CellValueGuards.ThrowIfNotFiniteNumberText(destination[..written], typeof(T), nameof(value)); - } - xml.Advance(written); - } - internal static void WriteEscaped(BiffBuffer xml, ReadOnlySpan value) { int start = 0; diff --git a/src/ExcelReader.Core/Writer/Internal/CellValueGuards.cs b/src/ExcelReader.Core/Writer/Internal/CellValueGuards.cs index fe059de9..a8b28b71 100644 --- a/src/ExcelReader.Core/Writer/Internal/CellValueGuards.cs +++ b/src/ExcelReader.Core/Writer/Internal/CellValueGuards.cs @@ -50,7 +50,7 @@ internal static void ThrowIfNonFiniteConversion(double value, Type sourceType, s [DoesNotReturn] [MethodImpl(MethodImplOptions.NoInlining)] - private static void ThrowUnrepresentable(Type sourceType, string paramName) + internal static void ThrowUnrepresentable(Type sourceType, string paramName) { throw new ArgumentException( $"Cannot write a value of type '{sourceType}' to a spreadsheet cell: it does not convert to a finite number.", diff --git a/src/ExcelReader.Core/Writer/Internal/SharedStringTable.cs b/src/ExcelReader.Core/Writer/Internal/SharedStringTable.cs index 2ea79cb6..f3b9c53e 100644 --- a/src/ExcelReader.Core/Writer/Internal/SharedStringTable.cs +++ b/src/ExcelReader.Core/Writer/Internal/SharedStringTable.cs @@ -48,6 +48,12 @@ internal int GetOrAdd(ReadOnlySpan value) return index; } + /// Counts references that were resolved without calling . + internal void AddReferences(int count) + { + Count += count; + } + internal BiffBuffer ToXlsxBytes() { BiffBuffer xml = new(Math.Max(256, _values.Count * 32)); diff --git a/src/ExcelReader.Core/Writer/Internal/SheetColumnValidation.cs b/src/ExcelReader.Core/Writer/Internal/SheetColumnValidation.cs index 263d5690..b8015939 100644 --- a/src/ExcelReader.Core/Writer/Internal/SheetColumnValidation.cs +++ b/src/ExcelReader.Core/Writer/Internal/SheetColumnValidation.cs @@ -2,11 +2,15 @@ namespace ExcelReader.Core.Writer.Internal { internal static class SheetColumnValidation { + // Excel's widest column; BIFF8 stores the width in 16 bits as 256ths of a character. + private const double MaxColumnWidth = 255; + internal static void SetColumnStyle( - ref Dictionary? columnStyles, int columnIndex, int styleId, int styleCount, + ref Dictionary? columnStyles, int columnIndex, int styleId, int styleCount, int maxColumns, WriterState state, object writer) { ArgumentOutOfRangeException.ThrowIfNegative(columnIndex); + ArgumentOutOfRangeException.ThrowIfGreaterThanOrEqual(columnIndex, maxColumns); ArgumentOutOfRangeException.ThrowIfNegative(styleId); ArgumentOutOfRangeException.ThrowIfGreaterThanOrEqual(styleId, styleCount); RequireNotStarted(state, writer); @@ -15,11 +19,13 @@ internal static void SetColumnStyle( } internal static void SetColumnWidth( - ref Dictionary? columnWidths, int columnIndex, double width, + ref Dictionary? columnWidths, int columnIndex, double width, int maxColumns, WriterState state, object writer) { ArgumentOutOfRangeException.ThrowIfNegative(columnIndex); + ArgumentOutOfRangeException.ThrowIfGreaterThanOrEqual(columnIndex, maxColumns); ArgumentOutOfRangeException.ThrowIfNegative(width); + ArgumentOutOfRangeException.ThrowIfGreaterThan(width, MaxColumnWidth); RequireNotStarted(state, writer); columnWidths ??= []; columnWidths[columnIndex] = width; diff --git a/src/ExcelReader.Core/Writer/Internal/StyleTable.cs b/src/ExcelReader.Core/Writer/Internal/StyleTable.cs index c0d7dfd4..b8a11dc7 100644 --- a/src/ExcelReader.Core/Writer/Internal/StyleTable.cs +++ b/src/ExcelReader.Core/Writer/Internal/StyleTable.cs @@ -1,4 +1,5 @@ using System.Runtime.InteropServices; +using ExcelReader.Core.Reader; namespace ExcelReader.Core.Writer.Internal { @@ -6,8 +7,14 @@ internal sealed class StyleTable { internal static readonly CellStyle DateStyle = new() { NumberFormat = "mm-dd-yy" }; + // Excel refuses a workbook with more custom number formats than this (its documented limit + // is "between 200 and 250"), and a BIFF12 format string longer than 255 characters. + private const int MaxCustomNumberFormats = 200; + private const int MaxNumberFormatLength = 255; + private readonly Dictionary _indices = []; private readonly List _styles = []; + private readonly HashSet _customFormats = new(StringComparer.Ordinal); internal StyleTable() { @@ -27,12 +34,31 @@ internal int Add(CellStyle style) { return index; } + if (style.NumberFormat is { Length: > MaxNumberFormatLength }) + { + throw new ArgumentException( + $"A number format cannot be longer than {MaxNumberFormatLength} characters ({style.NumberFormat.Length} chars).", nameof(style)); + } + ClaimNumberFormat(style.NumberFormat); index = _styles.Count; _styles.Add(style); _indices[style] = index; return index; } + private void ClaimNumberFormat(string? format) + { + if (format is null || _customFormats.Contains(format)) + { + return; + } + if (_customFormats.Count == MaxCustomNumberFormats) + { + throw new ExcelLimitExceededException("NumberFormats", MaxCustomNumberFormats, MaxCustomNumberFormats + 1L); + } + _customFormats.Add(format); + } + internal Dictionary AssignCustomNumberFormatIds() { var ids = new Dictionary(StringComparer.Ordinal); diff --git a/src/ExcelReader.Core/Writer/Internal/WriteOffloadStream.cs b/src/ExcelReader.Core/Writer/Internal/WriteOffloadStream.cs index 8a30666e..690f507e 100644 --- a/src/ExcelReader.Core/Writer/Internal/WriteOffloadStream.cs +++ b/src/ExcelReader.Core/Writer/Internal/WriteOffloadStream.cs @@ -9,10 +9,14 @@ internal sealed class WriteOffloadStream : Stream { private const int ChannelCapacity = 4; + // A queue item with this length carries no bytes: it marks the point a DrainAsync call waits for. + private const int BarrierLength = -1; + private readonly Stream _inner; private readonly Channel<(byte[] Buffer, int Length, bool Owned)> _channel; private readonly Task _consumer; private ExceptionDispatchInfo? _consumerException; + private TaskCompletionSource? _barrier; private bool _writerCompleted; private bool _disposed; @@ -174,6 +178,28 @@ public override async Task FlushAsync(CancellationToken cancellationToken) await _inner.FlushAsync(cancellationToken).ConfigureAwait(false); } + /// + /// Waits until everything enqueued so far has reached the inner stream and the consumer is idle. + /// The stream stays writable, which does not allow. + /// + internal async ValueTask DrainAsync(CancellationToken cancellationToken = default) + { + ThrowIfFaulted(); + var barrier = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + _barrier = barrier; + try + { + await _channel.Writer.WriteAsync(([], BarrierLength, Owned: false), cancellationToken).ConfigureAwait(false); + } + catch (ChannelClosedException) + { + ThrowIfFaulted(); + throw; + } + await barrier.Task.ConfigureAwait(false); + ThrowIfFaulted(); + } + private void CompleteWriterOnce() { if (!_writerCompleted) @@ -196,6 +222,11 @@ private async Task ConsumeAsync() { await foreach ((byte[] buf, int len, bool owned) in _channel.Reader.ReadAllAsync().ConfigureAwait(false)) { + if (len == BarrierLength) + { + _barrier?.TrySetResult(); + continue; + } try { await _inner.WriteAsync(buf.AsMemory(0, len)).ConfigureAwait(false); @@ -212,8 +243,12 @@ private async Task ConsumeAsync() _channel.Writer.TryComplete(); while (_channel.Reader.TryRead(out (byte[] Buffer, int Length, bool Owned) queued)) { - ReturnBuffer(queued.Buffer, queued.Owned); + if (queued.Length != BarrierLength) + { + ReturnBuffer(queued.Buffer, queued.Owned); + } } + _barrier?.TrySetResult(); } } diff --git a/src/ExcelReader.Core/Writer/Internal/WriterStateGuard.cs b/src/ExcelReader.Core/Writer/Internal/WriterStateGuard.cs index 3e49c90c..700ae88e 100644 --- a/src/ExcelReader.Core/Writer/Internal/WriterStateGuard.cs +++ b/src/ExcelReader.Core/Writer/Internal/WriterStateGuard.cs @@ -72,6 +72,26 @@ private static void ValidateSheetName(string name) { throw new ArgumentException("Sheet names cannot contain : \\ / ? * [ or ].", nameof(name)); } + if (name[0] == '\'' || name[^1] == '\'') + { + throw new ArgumentException("Sheet names cannot begin or end with an apostrophe.", nameof(name)); + } + if (HasUnwritableCharacter(name)) + { + throw new ArgumentException("Sheet names cannot contain control characters, U+FFFE or U+FFFF.", nameof(name)); + } + } + + private static bool HasUnwritableCharacter(string name) + { + foreach (char c in name) + { + if (c < ' ' || c >= '\uFFFE') + { + return true; + } + } + return false; } } } diff --git a/src/ExcelReader.Core/Writer/Internal/ZipEntryWriter.cs b/src/ExcelReader.Core/Writer/Internal/ZipEntryWriter.cs index 0f05fb83..4a7aca7d 100644 --- a/src/ExcelReader.Core/Writer/Internal/ZipEntryWriter.cs +++ b/src/ExcelReader.Core/Writer/Internal/ZipEntryWriter.cs @@ -1,6 +1,7 @@ using System.Buffers; using System.IO.Compression; using System.Text; +using ExcelReader.Core.Writer.Zip; namespace ExcelReader.Core.Writer.Internal { @@ -8,7 +9,7 @@ internal static class ZipEntryWriter { private static readonly UTF8Encoding Utf8NoBom = new(encoderShouldEmitUTF8Identifier: false); - internal static async ValueTask WriteTextAsync(ZipArchive zip, string entryName, string content, CompressionLevel compression, CancellationToken ct) + internal static async ValueTask WriteTextAsync(ZipPackageWriter zip, string entryName, string content, CompressionLevel compression, CancellationToken ct) { byte[] rented = ArrayPool.Shared.Rent(Utf8NoBom.GetMaxByteCount(content.Length)); try @@ -22,10 +23,9 @@ internal static async ValueTask WriteTextAsync(ZipArchive zip, string entryName, } } - internal static async ValueTask WriteBytesAsync(ZipArchive zip, string entryName, ReadOnlyMemory content, CompressionLevel compression, CancellationToken ct) + internal static async ValueTask WriteBytesAsync(ZipPackageWriter zip, string entryName, ReadOnlyMemory content, CompressionLevel compression, CancellationToken ct) { - ZipArchiveEntry entry = zip.CreateEntry(entryName, compression); - Stream stream = await entry.OpenAsync(ct).ConfigureAwait(false); + Stream stream = zip.OpenEntry(entryName, compression); await using (stream.ConfigureAwait(false)) { await stream.WriteAsync(content, ct).ConfigureAwait(false); @@ -38,8 +38,7 @@ internal static ValueTask FlushAsync(Stream stream, CancellationToken ct) return new ValueTask(stream.FlushAsync(ct)); } - - internal static void WriteText(ZipArchive zip, string entryName, string content, CompressionLevel compression) + internal static void WriteText(ZipPackageWriter zip, string entryName, string content, CompressionLevel compression) { byte[] rented = ArrayPool.Shared.Rent(Utf8NoBom.GetMaxByteCount(content.Length)); try @@ -53,16 +52,10 @@ internal static void WriteText(ZipArchive zip, string entryName, string content, } } - internal static void WriteBytes(ZipArchive zip, string entryName, ReadOnlySpan content, CompressionLevel compression) + internal static void WriteBytes(ZipPackageWriter zip, string entryName, ReadOnlySpan content, CompressionLevel compression) { - ZipArchiveEntry entry = zip.CreateEntry(entryName, compression); - using Stream stream = entry.Open(); + using Stream stream = zip.OpenEntry(entryName, compression); stream.Write(content); } - - internal static void Flush(Stream stream) - { - stream.Flush(); - } } } diff --git a/src/ExcelReader.Core/Writer/Parallel/BatchRunner.cs b/src/ExcelReader.Core/Writer/Parallel/BatchRunner.cs new file mode 100644 index 00000000..f9f19b1b --- /dev/null +++ b/src/ExcelReader.Core/Writer/Parallel/BatchRunner.cs @@ -0,0 +1,160 @@ +using System.Buffers; +using System.Diagnostics.CodeAnalysis; +using System.Runtime.CompilerServices; +using ExcelReader.Core.Reader.Internal; + +namespace ExcelReader.Core.Writer.Parallel +{ + /// + /// Cuts a record source into batches, formats at most a fixed number of them at once and hands the + /// results to the caller in order. The source is enumerated, and results are emitted, only on the + /// calling flow. + /// + internal sealed class BatchRunner + { + private readonly Queue> _inFlight = new(); + private readonly Func _format; + private readonly Func _emit; + private readonly CancellationToken _ct; + private readonly int _degree; + private readonly int _batchRows; + private T[] _batch; + private int _count; + private int _rowsBefore; + + internal BatchRunner(ParallelWriteOptions options, int rowsBefore, Func format, + Func emit, CancellationToken ct) + { + _degree = options.DegreeOfParallelism == 0 ? Environment.ProcessorCount : options.DegreeOfParallelism; + _batchRows = Math.Min(options.BatchRows, ExcelLimits.MaxRows); + _rowsBefore = rowsBefore; + _format = format; + _emit = emit; + _ct = ct; + _batch = ArrayPool.Shared.Rent(_batchRows); + } + + internal async ValueTask RunAsync(RecordSource source) + { + try + { + if (source.Sync is not null) + { + foreach (T record in source.Sync) + { + await AddAsync(record).ConfigureAwait(false); + } + } + else + { + await foreach (T record in source.Async!.WithCancellation(_ct).ConfigureAwait(false)) + { + await AddAsync(record).ConfigureAwait(false); + } + } + await CompleteAsync().ConfigureAwait(false); + } + catch + { + await AbandonAsync().ConfigureAwait(false); + throw; + } + } + + private ValueTask AddAsync(T record) + { + _batch[_count++] = record; + if (_count == _batchRows) + { + return DispatchAsync(); + } + return ValueTask.CompletedTask; + } + + private async ValueTask CompleteAsync() + { + if (_count > 0) + { + await DispatchAsync().ConfigureAwait(false); + } + while (_inFlight.Count > 0) + { + await EmitOldestAsync().ConfigureAwait(false); + } + ReturnBatch(_batch); + _batch = []; + } + + private async ValueTask DispatchAsync() + { + _ct.ThrowIfCancellationRequested(); + if (_rowsBefore + (long)_count > ExcelLimits.MaxRows) + { + ExcelLimits.ThrowRowLimit(_rowsBefore + (long)_count); + } + if (_inFlight.Count == _degree) + { + await EmitOldestAsync().ConfigureAwait(false); + } + T[] batch = _batch; + int count = _count; + int rowsBefore = _rowsBefore; + _batch = ArrayPool.Shared.Rent(_batchRows); + _count = 0; + _rowsBefore += count; + if (_degree == 1) + { + _inFlight.Enqueue(Task.FromResult(Format(batch, count, rowsBefore))); + } + else + { + _inFlight.Enqueue(Task.Run(() => Format(batch, count, rowsBefore), CancellationToken.None)); + } + } + + private BatchSegment Format(T[] batch, int count, int rowsBefore) + { + try + { + return _format(batch, count, rowsBefore); + } + finally + { + ReturnBatch(batch); + } + } + + private async ValueTask EmitOldestAsync() + { + using BatchSegment segment = await _inFlight.Dequeue().ConfigureAwait(false); + await _emit(segment, _ct).ConfigureAwait(false); + } + + // After a failure: no worker may outlive the call, and every finished batch gives its buffer back. + [SuppressMessage("Design", "CA1031:Do not catch general exception types", + Justification = "The caller is already rethrowing the first failure; a later batch's failure must not replace it.")] + private async ValueTask AbandonAsync() + { + while (_inFlight.Count > 0) + { + try + { + (await _inFlight.Dequeue().ConfigureAwait(false)).Dispose(); + } + catch (Exception) + { + } + } + ReturnBatch(_batch); + _batch = []; + } + + private static void ReturnBatch(T[] batch) + { + if (batch.Length > 0) + { + ArrayPool.Shared.Return(batch, RuntimeHelpers.IsReferenceOrContainsReferences()); + } + } + } +} diff --git a/src/ExcelReader.Core/Writer/Parallel/BatchSegment.cs b/src/ExcelReader.Core/Writer/Parallel/BatchSegment.cs new file mode 100644 index 00000000..5b677783 --- /dev/null +++ b/src/ExcelReader.Core/Writer/Parallel/BatchSegment.cs @@ -0,0 +1,46 @@ +using System.IO.Compression; +using ExcelReader.Core.Writer.Internal; +using ExcelReader.Core.Writer.Zip; + +namespace ExcelReader.Core.Writer.Parallel +{ + /// + /// One batch of rows after formatting and compression: the deflate segment, the CRC and length of + /// the rows it was made from, and the range of rows and columns they used. + /// + internal sealed class BatchSegment : IDisposable + { + internal BatchSegment(int rows) + { + Rows = rows; + } + + internal BiffBuffer Data { get; } = new(1 << 16); + + internal int Rows { get; } + + internal uint Crc { get; private set; } + + internal long RawLength { get; private set; } + + internal int RowFirst { get; set; } = -1; + + internal int RowLast { get; set; } = -1; + + internal int ColumnFirst { get; set; } = int.MaxValue; + + internal int ColumnLast { get; set; } = -1; + + internal void Fill(ReadOnlySpan raw, CompressionLevel level) + { + Crc = Crc32.Append(0, raw); + RawLength = raw.Length; + SegmentCompressor.Compress(raw, level, Data); + } + + public void Dispose() + { + Data.Dispose(); + } + } +} diff --git a/src/ExcelReader.Core/Writer/Parallel/IBatchSheet.cs b/src/ExcelReader.Core/Writer/Parallel/IBatchSheet.cs new file mode 100644 index 00000000..e84a633e --- /dev/null +++ b/src/ExcelReader.Core/Writer/Parallel/IBatchSheet.cs @@ -0,0 +1,39 @@ +using System.IO.Compression; + +namespace ExcelReader.Core.Writer.Parallel +{ + /// + /// What a sheet writer offers the parallel record path. The first five members act on the real + /// sheet, from the calling flow. The rest act on a detached copy, which formats batches into its + /// own buffer on a pool thread and never touches the output. + /// + internal interface IBatchSheet + where TSelf : class, IBatchSheet + where TRow : IRowWriter + { + CompressionLevel BatchCompression { get; } + + /// Starts a parallel stretch and returns how many rows the sheet already holds. + ValueTask BeginSegmentsAsync(CancellationToken ct); + + /// Writes one batch to the sheet's entry; called in batch order. + ValueTask AppendSegmentAsync(BatchSegment segment, CancellationToken ct); + + void EndSegments(); + + /// Faults the sheet after a failed parallel stretch. Never throws. + ValueTask FailSegmentsAsync(); + + /// Makes a detached copy whose buffer is sized for batches of rows. + TSelf CreateDetached(int batchRows); + + void BeginBatch(int rowsBefore); + + TRow StartBatchRow(); + + /// Ends the batch, records its used range in and returns its bytes. + ReadOnlySpan EndBatch(BatchSegment segment); + + void ReleaseDetached(); + } +} diff --git a/src/ExcelReader.Core/Writer/Parallel/ParallelRecordWriter.cs b/src/ExcelReader.Core/Writer/Parallel/ParallelRecordWriter.cs new file mode 100644 index 00000000..b500abcb --- /dev/null +++ b/src/ExcelReader.Core/Writer/Parallel/ParallelRecordWriter.cs @@ -0,0 +1,118 @@ +using System.Collections.Concurrent; + +namespace ExcelReader.Core.Writer.Parallel +{ + /// Drives a parallel record write for any sheet writer that implements . + internal static class ParallelRecordWriter + { + internal static async ValueTask WriteRecordsAsync(TSheet sheet, RecordSource records, + ExcelRecordLayout layout, ParallelWriteOptions? options, CancellationToken ct) + where TSheet : class, ISheetWriter, IBatchSheet + where TRow : IRowWriter + { + ParallelWriteOptions effective = Resolve(options); + (string[] headers, Action writeRow) = layout.Columns(); + await SheetWriterExtensions.WriteHeaderAsync(sheet, headers, ct).ConfigureAwait(false); + await RunAsync(sheet, records, writeRow, effective, ct).ConfigureAwait(false); + } + + internal static ValueTask WriteRowsAsync(TSheet sheet, RecordSource records, + Action writeRow, ParallelWriteOptions? options, CancellationToken ct) + where TSheet : class, IBatchSheet + where TRow : IRowWriter + { + return RunAsync(sheet, records, writeRow, Resolve(options), ct); + } + + internal static async ValueTask WriteTableAsync(TSheet sheet, RecordSource records, string name, + ExcelRecordLayout layout, ExcelTableOptions? tableOptions, ParallelWriteOptions? options, CancellationToken ct) + where TSheet : class, ITableSheetWriter, IBatchSheet + where TRow : IRowWriter + { + ParallelWriteOptions effective = Resolve(options); + (string[] headers, Action writeRow) = layout.Columns(); + await sheet.BeginTableAsync(name, headers, tableOptions, ct).ConfigureAwait(false); + await RunAsync(sheet, records, SheetWriterExtensions.ShiftToTable(writeRow, tableOptions), effective, ct).ConfigureAwait(false); + sheet.EndTable(); + } + + private static ParallelWriteOptions Resolve(ParallelWriteOptions? options) + { + ParallelWriteOptions effective = options ?? ParallelWriteOptions.Default; + effective.Validate(); + return effective; + } + + private static async ValueTask RunAsync(TSheet sheet, RecordSource records, + Action writeRow, ParallelWriteOptions options, CancellationToken ct) + where TSheet : class, IBatchSheet + where TRow : IRowWriter + { + int rowsBefore = await sheet.BeginSegmentsAsync(ct).ConfigureAwait(false); + var workers = new ConcurrentBag(); + try + { + var runner = new BatchRunner(options, rowsBefore, + (batch, count, before) => FormatBatch(sheet, workers, writeRow, batch, count, before), + sheet.AppendSegmentAsync, ct); + await runner.RunAsync(records).ConfigureAwait(false); + } + catch + { + await sheet.FailSegmentsAsync().ConfigureAwait(false); + throw; + } + finally + { + foreach (TSheet worker in workers) + { + worker.ReleaseDetached(); + } + } + sheet.EndSegments(); + } + + private static BatchSegment FormatBatch(TSheet sheet, ConcurrentBag workers, + Action writeRow, T[] batch, int count, int rowsBefore) + where TSheet : class, IBatchSheet + where TRow : IRowWriter + { + if (!workers.TryTake(out TSheet? worker)) + { + worker = sheet.CreateDetached(count); + } + try + { + worker.BeginBatch(rowsBefore); + for (int i = 0; i < count; i++) + { + TRow row = worker.StartBatchRow(); + writeRow(row, batch[i]); + row.Dispose(); + } + return Compress(sheet, worker, count); + } + finally + { + workers.Add(worker); + } + } + + private static BatchSegment Compress(TSheet sheet, TSheet worker, int rows) + where TSheet : class, IBatchSheet + where TRow : IRowWriter + { + var segment = new BatchSegment(rows); + try + { + segment.Fill(worker.EndBatch(segment), sheet.BatchCompression); + return segment; + } + catch + { + segment.Dispose(); + throw; + } + } + } +} diff --git a/src/ExcelReader.Core/Writer/Parallel/RecordSource.cs b/src/ExcelReader.Core/Writer/Parallel/RecordSource.cs new file mode 100644 index 00000000..8e84150f --- /dev/null +++ b/src/ExcelReader.Core/Writer/Parallel/RecordSource.cs @@ -0,0 +1,22 @@ +namespace ExcelReader.Core.Writer.Parallel +{ + /// A record source that is either synchronous or asynchronous. + internal readonly struct RecordSource + { + internal RecordSource(IEnumerable records) + { + Sync = records; + Async = null; + } + + internal RecordSource(IAsyncEnumerable records) + { + Sync = null; + Async = records; + } + + internal IEnumerable? Sync { get; } + + internal IAsyncEnumerable? Async { get; } + } +} diff --git a/src/ExcelReader.Core/Writer/Parallel/WorkerStringCache.cs b/src/ExcelReader.Core/Writer/Parallel/WorkerStringCache.cs new file mode 100644 index 00000000..2218aedb --- /dev/null +++ b/src/ExcelReader.Core/Writer/Parallel/WorkerStringCache.cs @@ -0,0 +1,74 @@ +using ExcelReader.Core.Writer.Internal; + +namespace ExcelReader.Core.Writer.Parallel +{ + /// + /// One worker's view of the workbook's shared-string table: texts it has already resolved are + /// answered locally, and only a text new to this worker takes the table's lock. + /// + internal sealed class WorkerStringCache + { + private const int MaxEntries = 65_536; + + private readonly SharedStringTable _table; + private readonly Lock _gate; + private readonly Dictionary _indexes = new(StringComparer.Ordinal); + private readonly Dictionary.AlternateLookup> _spanIndexes; + private int _localHits; + + internal WorkerStringCache(SharedStringTable table, Lock gate) + { + _table = table; + _gate = gate; + _spanIndexes = _indexes.GetAlternateLookup>(); + } + + internal int GetOrAdd(string value) + { + if (_indexes.TryGetValue(value, out int index)) + { + _localHits++; + return index; + } + return Resolve(value); + } + + internal int GetOrAdd(ReadOnlySpan value) + { + if (_spanIndexes.TryGetValue(value, out int index)) + { + _localHits++; + return index; + } + return Resolve(value.ToString()); + } + + // The table counts every reference; ones answered locally are reported in bulk. + internal void FlushReferences() + { + if (_localHits == 0) + { + return; + } + lock (_gate) + { + _table.AddReferences(_localHits); + } + _localHits = 0; + } + + private int Resolve(string value) + { + int index; + lock (_gate) + { + index = _table.GetOrAdd(value); + } + if (_indexes.Count < MaxEntries) + { + _indexes.Add(value, index); + } + return index; + } + } +} diff --git a/src/ExcelReader.Core/Writer/ParallelSheetWriterExtensions.cs b/src/ExcelReader.Core/Writer/ParallelSheetWriterExtensions.cs new file mode 100644 index 00000000..1a61249e --- /dev/null +++ b/src/ExcelReader.Core/Writer/ParallelSheetWriterExtensions.cs @@ -0,0 +1,180 @@ +using ExcelReader.Core.Writer.Parallel; +using ExcelReader.Core.Writer.Xlsb; +using ExcelReader.Core.Writer.Xlsx; + +namespace ExcelReader.Core.Writer +{ + /// + /// Record writes that format and compress batches of rows on several threads and write them in + /// order. Memory is bounded by batches of + /// rows. + /// + /// + /// The source is enumerated by one flow at a time, never concurrently. A writeRow delegate runs on pool + /// threads, several at once, so whatever it captures must be safe to use concurrently; the layouts + /// from are. Rows may be written through the normal API before + /// and after a call, but not while one is running. A call that throws, a cancelled one included, + /// ends the sheet and its workbook. With shared strings on, string indexes depend on thread + /// timing, so two runs give the same content in files that differ byte for byte. + /// + public static class ParallelSheetWriterExtensions + { + /// Writes a header row from , then one row per record, in parallel batches. + /// The record type being written. + /// The sheet to write rows to. + /// The records to write, one row each, in enumeration order. + /// The columns to write, from . + /// Degree of parallelism and batch size. Defaults to . + /// A token to cancel the operation between batches. + /// , or is . + /// holds a negative degree or a batch size below 1. + /// The sheet has already been ended. + /// A row is still active on the sheet. + /// The records do not fit in the sheet's 1,048,576 rows. + /// was cancelled. Unlike a serial write cancelled between rows, this ends the sheet and its workbook. + public static ValueTask WriteRecordsParallelAsync(this XlsxSheetWriter sheet, IEnumerable records, + ExcelRecordLayout layout, ParallelWriteOptions? options = null, CancellationToken ct = default) + { + ArgumentNullException.ThrowIfNull(sheet); + ArgumentNullException.ThrowIfNull(records); + ArgumentNullException.ThrowIfNull(layout); + return ParallelRecordWriter.WriteRecordsAsync(sheet, new RecordSource(records), layout, options, ct); + } + + /// + public static ValueTask WriteRecordsParallelAsync(this XlsxSheetWriter sheet, IAsyncEnumerable records, + ExcelRecordLayout layout, ParallelWriteOptions? options = null, CancellationToken ct = default) + { + ArgumentNullException.ThrowIfNull(sheet); + ArgumentNullException.ThrowIfNull(records); + ArgumentNullException.ThrowIfNull(layout); + return ParallelRecordWriter.WriteRecordsAsync(sheet, new RecordSource(records), layout, options, ct); + } + + /// Writes one row per record in parallel batches, calling to populate each row. No header is written. + /// The record type being written. + /// The sheet to write rows to. + /// The records to write, one row each, in enumeration order. + /// Populates a row's cells from a single record. It runs on pool threads, several at once. + /// Degree of parallelism and batch size. Defaults to . + /// A token to cancel the operation between batches. + /// , or is . + /// holds a negative degree or a batch size below 1. + /// The sheet has already been ended. + /// A row is still active on the sheet. + /// The records do not fit in the sheet's 1,048,576 rows. + /// was cancelled. Unlike a serial write cancelled between rows, this ends the sheet and its workbook. + public static ValueTask WriteRecordsParallelAsync(this XlsxSheetWriter sheet, IEnumerable records, + Action writeRow, ParallelWriteOptions? options = null, CancellationToken ct = default) + { + ArgumentNullException.ThrowIfNull(sheet); + ArgumentNullException.ThrowIfNull(records); + ArgumentNullException.ThrowIfNull(writeRow); + return ParallelRecordWriter.WriteRowsAsync(sheet, new RecordSource(records), writeRow, options, ct); + } + + /// + public static ValueTask WriteRecordsParallelAsync(this XlsxSheetWriter sheet, IAsyncEnumerable records, + Action writeRow, ParallelWriteOptions? options = null, CancellationToken ct = default) + { + ArgumentNullException.ThrowIfNull(sheet); + ArgumentNullException.ThrowIfNull(records); + ArgumentNullException.ThrowIfNull(writeRow); + return ParallelRecordWriter.WriteRowsAsync(sheet, new RecordSource(records), writeRow, options, ct); + } + + /// + /// Writes as an Excel table named : a header row from + /// , one row per record in parallel batches, then closes the table. + /// + /// The record type being written. + /// The sheet to write to. + /// The records to write, one row each, in enumeration order. + /// The table name; see . + /// The columns to write, from . + /// Position and style. Defaults to . + /// Degree of parallelism and batch size. Defaults to . + /// A token to cancel the operation between batches. + /// , , or is . + /// holds a negative degree or a batch size below 1. + /// was cancelled. Unlike a serial write cancelled between rows, this ends the sheet and its workbook. + public static ValueTask WriteTableParallelAsync(this XlsxSheetWriter sheet, IEnumerable records, string name, + ExcelRecordLayout layout, ExcelTableOptions? tableOptions = null, ParallelWriteOptions? options = null, CancellationToken ct = default) + { + ArgumentNullException.ThrowIfNull(sheet); + ArgumentNullException.ThrowIfNull(records); + ArgumentNullException.ThrowIfNull(layout); + return ParallelRecordWriter.WriteTableAsync(sheet, new RecordSource(records), name, layout, tableOptions, options, ct); + } + + /// + public static ValueTask WriteTableParallelAsync(this XlsxSheetWriter sheet, IAsyncEnumerable records, string name, + ExcelRecordLayout layout, ExcelTableOptions? tableOptions = null, ParallelWriteOptions? options = null, CancellationToken ct = default) + { + ArgumentNullException.ThrowIfNull(sheet); + ArgumentNullException.ThrowIfNull(records); + ArgumentNullException.ThrowIfNull(layout); + return ParallelRecordWriter.WriteTableAsync(sheet, new RecordSource(records), name, layout, tableOptions, options, ct); + } + + /// + public static ValueTask WriteRecordsParallelAsync(this XlsbSheetWriter sheet, IEnumerable records, + ExcelRecordLayout layout, ParallelWriteOptions? options = null, CancellationToken ct = default) + { + ArgumentNullException.ThrowIfNull(sheet); + ArgumentNullException.ThrowIfNull(records); + ArgumentNullException.ThrowIfNull(layout); + return ParallelRecordWriter.WriteRecordsAsync(sheet, new RecordSource(records), layout, options, ct); + } + + /// + public static ValueTask WriteRecordsParallelAsync(this XlsbSheetWriter sheet, IAsyncEnumerable records, + ExcelRecordLayout layout, ParallelWriteOptions? options = null, CancellationToken ct = default) + { + ArgumentNullException.ThrowIfNull(sheet); + ArgumentNullException.ThrowIfNull(records); + ArgumentNullException.ThrowIfNull(layout); + return ParallelRecordWriter.WriteRecordsAsync(sheet, new RecordSource(records), layout, options, ct); + } + + /// + public static ValueTask WriteRecordsParallelAsync(this XlsbSheetWriter sheet, IEnumerable records, + Action writeRow, ParallelWriteOptions? options = null, CancellationToken ct = default) + { + ArgumentNullException.ThrowIfNull(sheet); + ArgumentNullException.ThrowIfNull(records); + ArgumentNullException.ThrowIfNull(writeRow); + return ParallelRecordWriter.WriteRowsAsync(sheet, new RecordSource(records), writeRow, options, ct); + } + + /// + public static ValueTask WriteRecordsParallelAsync(this XlsbSheetWriter sheet, IAsyncEnumerable records, + Action writeRow, ParallelWriteOptions? options = null, CancellationToken ct = default) + { + ArgumentNullException.ThrowIfNull(sheet); + ArgumentNullException.ThrowIfNull(records); + ArgumentNullException.ThrowIfNull(writeRow); + return ParallelRecordWriter.WriteRowsAsync(sheet, new RecordSource(records), writeRow, options, ct); + } + + /// + public static ValueTask WriteTableParallelAsync(this XlsbSheetWriter sheet, IEnumerable records, string name, + ExcelRecordLayout layout, ExcelTableOptions? tableOptions = null, ParallelWriteOptions? options = null, CancellationToken ct = default) + { + ArgumentNullException.ThrowIfNull(sheet); + ArgumentNullException.ThrowIfNull(records); + ArgumentNullException.ThrowIfNull(layout); + return ParallelRecordWriter.WriteTableAsync(sheet, new RecordSource(records), name, layout, tableOptions, options, ct); + } + + /// + public static ValueTask WriteTableParallelAsync(this XlsbSheetWriter sheet, IAsyncEnumerable records, string name, + ExcelRecordLayout layout, ExcelTableOptions? tableOptions = null, ParallelWriteOptions? options = null, CancellationToken ct = default) + { + ArgumentNullException.ThrowIfNull(sheet); + ArgumentNullException.ThrowIfNull(records); + ArgumentNullException.ThrowIfNull(layout); + return ParallelRecordWriter.WriteTableAsync(sheet, new RecordSource(records), name, layout, tableOptions, options, ct); + } + } +} diff --git a/src/ExcelReader.Core/Writer/ParallelWriteOptions.cs b/src/ExcelReader.Core/Writer/ParallelWriteOptions.cs new file mode 100644 index 00000000..5d0d0760 --- /dev/null +++ b/src/ExcelReader.Core/Writer/ParallelWriteOptions.cs @@ -0,0 +1,31 @@ +using System.Diagnostics.CodeAnalysis; + +namespace ExcelReader.Core.Writer +{ + /// Configures the parallel record writes of . + public sealed class ParallelWriteOptions + { + /// Gets the default options: one batch in flight per processor, 8,192 rows per batch. + public static ParallelWriteOptions Default { get; } = new(); + + /// + /// Gets the most batches formatted and compressed at once. 0, the default, means + /// . With 1 the batches run on the calling flow, + /// without the thread pool. + /// + public int DegreeOfParallelism { get; init; } + + /// + /// Gets the number of rows per batch. Each batch in flight holds its formatted rows and their + /// compressed form, so this and bound the memory used. + /// + public int BatchRows { get; init; } = 8192; + + [SuppressMessage("Design", "MA0015:Specify the parameter name in ArgumentException", Justification = "The invalid values are properties of this options object, so the property name is the right parameter name.")] + internal void Validate() + { + ArgumentOutOfRangeException.ThrowIfNegative(DegreeOfParallelism, nameof(DegreeOfParallelism)); + ArgumentOutOfRangeException.ThrowIfLessThan(BatchRows, 1, nameof(BatchRows)); + } + } +} diff --git a/src/ExcelReader.Core/Writer/SheetWriterExtensions.cs b/src/ExcelReader.Core/Writer/SheetWriterExtensions.cs index f312fed4..939a1bc2 100644 --- a/src/ExcelReader.Core/Writer/SheetWriterExtensions.cs +++ b/src/ExcelReader.Core/Writer/SheetWriterExtensions.cs @@ -112,7 +112,7 @@ public static async ValueTask WriteTableAsync(this ITableSheetWriter ShiftToTable(Action writeRow, ExcelTableOptions? options) + internal static Action ShiftToTable(Action writeRow, ExcelTableOptions? options) where TRow : IRowWriter { int firstColumn = (options ?? ExcelTableOptions.Default).FirstColumn; @@ -127,7 +127,7 @@ private static Action ShiftToTable(Action writeRow, E }; } - private static async ValueTask WriteHeaderAsync(ISheetWriter sheet, string[] headers, CancellationToken ct) + internal static async ValueTask WriteHeaderAsync(ISheetWriter sheet, string[] headers, CancellationToken ct) where TRow : IRowWriter { TRow row = await sheet.StartRowAsync(ct).ConfigureAwait(false); diff --git a/src/ExcelReader.Core/Writer/Xls/BiffRecord.cs b/src/ExcelReader.Core/Writer/Xls/BiffRecord.cs index 53f15481..50eeb169 100644 --- a/src/ExcelReader.Core/Writer/Xls/BiffRecord.cs +++ b/src/ExcelReader.Core/Writer/Xls/BiffRecord.cs @@ -30,7 +30,31 @@ internal static class BiffRecord internal const int Blank = 0x0201; internal const int Number = 0x0203; internal const int Label = 0x0204; + internal const int LabelSst = 0x00FD; + internal const int Rk = 0x027E; + internal const int MulRk = 0x00BD; internal const int BoolErr = 0x0205; + internal const int Index = 0x020B; + internal const int Row = 0x0208; + internal const int DbCell = 0x00D7; + internal const int DefColWidth = 0x0055; + internal const int DefaultRowHeight = 0x0225; + internal const int CalcMode = 0x000D; + internal const int CalcCount = 0x000C; + internal const int CalcRefMode = 0x000F; + internal const int CalcIter = 0x0011; + internal const int CalcDelta = 0x0010; + internal const int CalcSaveRecalc = 0x005F; + internal const int PrintRowCol = 0x002A; + internal const int PrintGrid = 0x002B; + internal const int GridSet = 0x0082; + internal const int Guts = 0x0080; + internal const int WsBool = 0x0081; + internal const int WinProtect = 0x0019; + internal const int Protect = 0x0012; + internal const int Password = 0x0013; + internal const int Prot4Rev = 0x01AF; + internal const int Prot4RevPass = 0x01BC; internal const int Biff8Version = 0x0600; internal const int SubstreamGlobals = 0x0005; diff --git a/src/ExcelReader.Core/Writer/Xls/BiffRecordWriter.cs b/src/ExcelReader.Core/Writer/Xls/BiffRecordWriter.cs index 1b205a00..73adeb96 100644 --- a/src/ExcelReader.Core/Writer/Xls/BiffRecordWriter.cs +++ b/src/ExcelReader.Core/Writer/Xls/BiffRecordWriter.cs @@ -1,4 +1,7 @@ -using ExcelReader.Core.Reader.Internal; +using System.Buffers; +using System.Buffers.Binary; +using System.Runtime.InteropServices; +using System.Text; using ExcelReader.Core.Writer.Internal; namespace ExcelReader.Core.Writer.Xls @@ -10,10 +13,11 @@ internal static void WriteBof(BiffBuffer buffer, int substreamType) int len = buffer.BeginRecord(BiffRecord.Bof); buffer.WriteU16(BiffRecord.Biff8Version); buffer.WriteU16(substreamType); - for (int i = 0; i < 12; i++) - { - buffer.WriteByte(0); - } + // [MS-XLS] requires rupYear to be 1996 or 1997, and verLowestBiff to be 6. + buffer.WriteU16(0x0DBB); + buffer.WriteU16(0x07CC); + buffer.WriteI32(0); + buffer.WriteI32(6); buffer.EndRecord(len); } @@ -54,8 +58,10 @@ internal static void WriteWriteAccess(BiffBuffer buffer) { int len = buffer.BeginRecord(BiffRecord.WriteAccess); ReadOnlySpan name = "ExcelReader"u8; + buffer.WriteU16(name.Length); + buffer.WriteByte(0); buffer.Write(name); - for (int i = name.Length; i < 112; i++) + for (int i = 3 + name.Length; i < 112; i++) { buffer.WriteByte((byte)' '); } @@ -81,18 +87,29 @@ internal static void WriteFnGroupCount(BiffBuffer buffer) { WriteU16Record(buffer, BiffRecord.FnGroupCount, 17); } - private static ReadOnlySpan WriteWindow1Payload => [ - 0xFF, 0x7F, 0xFF, 0x7F, 0x80, 0x70, 0x30, 0x39, - 0x38, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, 0xC0, 0x20, - ]; - internal static void WriteWindow1(BiffBuffer buffer) + internal static void WriteWindow1(BiffBuffer buffer, int activeSheet) { int len = buffer.BeginRecord(BiffRecord.Window1); - buffer.Write(WriteWindow1Payload); + buffer.WriteU16(0x7FFF); + buffer.WriteU16(0x7FFF); + buffer.WriteU16(0x7080); + buffer.WriteU16(0x3930); + buffer.WriteU16(0x0038); + buffer.WriteU16(activeSheet); + buffer.WriteU16(0); + buffer.WriteU16(1); + buffer.WriteU16(600); buffer.EndRecord(len); } + internal static void WriteU16Records(BiffBuffer buffer, ReadOnlySpan ids, int value) + { + foreach (int id in ids) + { + WriteU16Record(buffer, id, value); + } + } + internal static void WriteBackup(BiffBuffer buffer) { WriteU16Record(buffer, BiffRecord.Backup, 0); @@ -210,81 +227,175 @@ internal static void WriteDimension(BiffBuffer buffer, int rowCount, int colCoun buffer.WriteU16(0); buffer.EndRecord(len); } - private static ReadOnlySpan ThirdPayload => [0xB6, 0x06, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00]; - internal static void WriteWindow2(BiffBuffer buffer) + internal static void WriteWindow2(BiffBuffer buffer, bool selected) { int len = buffer.BeginRecord(BiffRecord.Window2); - buffer.Write(ThirdPayload); + buffer.WriteU16(selected ? 0x06B6 : 0x00B6); + buffer.WriteU16(0); + buffer.WriteU16(0); + buffer.WriteU16(0x0040); + buffer.WriteU16(0); + buffer.WriteU16(0); + buffer.WriteU16(0); + buffer.WriteI32(0); buffer.EndRecord(len); } - internal static void WriteBlank(BiffBuffer buffer, int row, int col, int xf) + internal static void WriteSheetSettings(BiffBuffer buffer) + { + WriteU16Record(buffer, BiffRecord.CalcMode, 1); + WriteU16Record(buffer, BiffRecord.CalcCount, 100); + WriteU16Record(buffer, BiffRecord.CalcRefMode, 1); + WriteU16Record(buffer, BiffRecord.CalcIter, 0); + int delta = buffer.BeginRecord(BiffRecord.CalcDelta); + buffer.WriteDouble(0.001); + buffer.EndRecord(delta); + WriteU16Record(buffer, BiffRecord.CalcSaveRecalc, 1); + WriteU16Record(buffer, BiffRecord.PrintRowCol, 0); + WriteU16Record(buffer, BiffRecord.PrintGrid, 0); + WriteU16Record(buffer, BiffRecord.GridSet, 1); + int guts = buffer.BeginRecord(BiffRecord.Guts); + buffer.WriteI32(0); + buffer.WriteI32(0); + buffer.EndRecord(guts); + int rowHeight = buffer.BeginRecord(BiffRecord.DefaultRowHeight); + buffer.WriteU16(0); + buffer.WriteU16(DefaultRowHeightTwips); + buffer.EndRecord(rowHeight); + WriteU16Record(buffer, BiffRecord.WsBool, 0x04C1); + } + + internal const int DefaultRowHeightTwips = 255; + + internal static void WriteDefColWidth(BiffBuffer buffer) { - int len = buffer.BeginRecord(BiffRecord.Blank); - WriteCellHeader(buffer, row, col, xf); + WriteU16Record(buffer, BiffRecord.DefColWidth, 8); + } + + internal static void WriteRow(BiffBuffer buffer, int row, int colMic, int colMac) + { + int len = buffer.BeginRecord(BiffRecord.Row); + buffer.WriteU16(row); + buffer.WriteU16(colMic); + buffer.WriteU16(colMac); + buffer.WriteU16(DefaultRowHeightTwips); + buffer.WriteI32(0); + buffer.WriteU16(0x0100); + buffer.WriteU16(GeneralStyleXf); buffer.EndRecord(len); } + private const int GeneralStyleXf = 15; + + // Cell records are written whole into one span: they are most of a sheet's bytes, and going + // through BeginRecord/WriteU16/EndRecord field by field measured as most of their cost. + internal static void WriteBlank(BiffBuffer buffer, int row, int col, int xf) + { + Span record = CellRecord(buffer, BiffRecord.Blank, 6, row, col, xf); + buffer.Advance(record.Length); + } + internal static void WriteNumber(BiffBuffer buffer, int row, int col, int xf, double value) { CellValueGuards.ThrowIfNonFinite(value, nameof(value)); - int len = buffer.BeginRecord(BiffRecord.Number); - WriteCellHeader(buffer, row, col, xf); - buffer.WriteDouble(value); - buffer.EndRecord(len); + Span record = CellRecord(buffer, BiffRecord.Number, 14, row, col, xf); + BinaryPrimitives.WriteDoubleLittleEndian(record[10..], value); + buffer.Advance(record.Length); } - internal static void WriteLabel(BiffBuffer buffer, int row, int col, int xf, ReadOnlySpan value) + /// + /// Opens a run of RK cells: header, row and first column now, then 6 bytes (ixfe, RK) per cell. + /// A run of one is an RK record, whose layout is exactly that; a longer run is a MULRK record, + /// which only adds the last column. fixes the id and length either way. + /// + internal static int BeginRkRun(BiffBuffer buffer, int row, int firstCol) + { + int start = buffer.Length; + Span header = buffer.GetSpan(8); + BinaryPrimitives.WriteUInt16LittleEndian(header[4..], (ushort)row); + BinaryPrimitives.WriteUInt16LittleEndian(header[6..], (ushort)firstCol); + buffer.Advance(8); + return start; + } + + internal static void AppendRk(BiffBuffer buffer, int xf, int rk) { - ExcelLimits.ThrowIfCellTextTooLong(value.Length, nameof(value)); - bool compressed = BiffStringEncoder.CanCompress(value); - int charBytes = compressed ? value.Length : value.Length * 2; - if (9 + charBytes <= BiffRecord.MaxPayload) + Span cell = buffer.GetSpan(6); + BinaryPrimitives.WriteUInt16LittleEndian(cell, (ushort)xf); + BinaryPrimitives.WriteInt32LittleEndian(cell[2..], rk); + buffer.Advance(6); + } + + internal static void EndRkRun(BiffBuffer buffer, int start, int firstCol, int count) + { + Span header = buffer.Slice(start, 4); + if (count == 1) { - int len = buffer.BeginRecord(BiffRecord.Label); - WriteCellHeader(buffer, row, col, xf); - buffer.WriteU16(value.Length); - buffer.WriteByte((byte)(compressed ? 0 : 1)); - BiffStringEncoder.WriteChars(buffer, value, compressed); - buffer.EndRecord(len); + BinaryPrimitives.WriteUInt16LittleEndian(header, BiffRecord.Rk); + BinaryPrimitives.WriteUInt16LittleEndian(header[2..], 10); return; } - int firstChars = compressed ? 8215 : 4107; - int lenSplit = buffer.BeginRecord(BiffRecord.Label); - WriteCellHeader(buffer, row, col, xf); - buffer.WriteU16(value.Length); - buffer.WriteByte((byte)(compressed ? 0 : 1)); - BiffStringEncoder.WriteChars(buffer, value[..firstChars], compressed); - buffer.EndRecord(lenSplit); - int offset = firstChars; - int maxContChars = compressed ? 8223 : 4111; - while (offset < value.Length) + BinaryPrimitives.WriteUInt16LittleEndian(header, BiffRecord.MulRk); + BinaryPrimitives.WriteUInt16LittleEndian(header[2..], (ushort)(4 + (6 * count) + 2)); + Span last = buffer.GetSpan(2); + BinaryPrimitives.WriteUInt16LittleEndian(last, (ushort)(firstCol + count - 1)); + buffer.Advance(2); + } + + internal static void WriteLabel(BiffBuffer buffer, int row, int col, int xf, ReadOnlySpan value) + { + Span dest = buffer.GetSpan(13 + (value.Length * 2)); + Span chars = dest[13..]; + int charBytes; + byte flags = 0; + if (Ascii.FromUtf16(value, chars, out _) == OperationStatus.Done) + { + charBytes = value.Length; + } + else if (BiffStringEncoder.CanCompress(value)) { - int take = Math.Min(maxContChars, value.Length - offset); - int contLen = buffer.BeginRecord(0x003C); - buffer.WriteByte((byte)(compressed ? 0 : 1)); - BiffStringEncoder.WriteChars(buffer, value.Slice(offset, take), compressed); - buffer.EndRecord(contLen); - offset += take; + charBytes = Encoding.Latin1.GetBytes(value, chars); } + else + { + MemoryMarshal.AsBytes(value).CopyTo(chars); + charBytes = value.Length * 2; + flags = 1; + } + Span record = CellRecord(buffer, BiffRecord.Label, 9 + charBytes, row, col, xf); + BinaryPrimitives.WriteUInt16LittleEndian(record[10..], (ushort)value.Length); + record[12] = flags; + buffer.Advance(record.Length); + } + + internal static void WriteLabelSst(BiffBuffer buffer, int row, int col, int xf, int sharedStringIndex) + { + Span record = CellRecord(buffer, BiffRecord.LabelSst, 10, row, col, xf); + BinaryPrimitives.WriteInt32LittleEndian(record[10..], sharedStringIndex); + buffer.Advance(record.Length); } internal static void WriteBool(BiffBuffer buffer, int row, int col, int xf, bool value) { - int len = buffer.BeginRecord(BiffRecord.BoolErr); - WriteCellHeader(buffer, row, col, xf); - buffer.WriteByte((byte)(value ? 1 : 0)); - buffer.WriteByte(0); - buffer.EndRecord(len); + Span record = CellRecord(buffer, BiffRecord.BoolErr, 8, row, col, xf); + record[10] = (byte)(value ? 1 : 0); + record[11] = 0; + buffer.Advance(record.Length); } - private static void WriteCellHeader(BiffBuffer buffer, int row, int col, int xf) + /// Reserves a cell record and fills its header and row/col/xf; the caller writes the rest of the payload, then advances by the returned length. + private static Span CellRecord(BiffBuffer buffer, int id, int payload, int row, int col, int xf) { - buffer.WriteU16(row); - buffer.WriteU16(col); - buffer.WriteU16(xf); + Span record = buffer.GetSpan(4 + payload)[..(4 + payload)]; + BinaryPrimitives.WriteUInt16LittleEndian(record, (ushort)id); + BinaryPrimitives.WriteUInt16LittleEndian(record[2..], (ushort)payload); + BinaryPrimitives.WriteUInt16LittleEndian(record[4..], (ushort)row); + BinaryPrimitives.WriteUInt16LittleEndian(record[6..], (ushort)col); + BinaryPrimitives.WriteUInt16LittleEndian(record[8..], (ushort)xf); + return record; } + private static void WriteU16Record(BiffBuffer buffer, int id, int value) { int len = buffer.BeginRecord(id); diff --git a/src/ExcelReader.Core/Writer/Xls/XlsGlobals.cs b/src/ExcelReader.Core/Writer/Xls/XlsGlobals.cs index 287be875..46dfe743 100644 --- a/src/ExcelReader.Core/Writer/Xls/XlsGlobals.cs +++ b/src/ExcelReader.Core/Writer/Xls/XlsGlobals.cs @@ -14,7 +14,8 @@ internal static int CustomXf(int abstractStyleIndex) } internal static int[] Write(BiffBuffer buffer, ReadOnlySpan sheetNames, - ReadOnlySpan sheetVisibilities, bool date1904, StyleTable styles) + ReadOnlySpan sheetVisibilities, int activeSheet, bool date1904, StyleTable styles, + XlsSharedStrings sharedStrings) { BiffRecordWriter.WriteBof(buffer, BiffRecord.SubstreamGlobals); BiffRecordWriter.WriteInterfaceHdr(buffer, 1200); @@ -25,17 +26,20 @@ internal static int[] Write(BiffBuffer buffer, ReadOnlySpan sheetNames, BiffRecordWriter.WriteDsf(buffer); BiffRecordWriter.WriteTabId(buffer, sheetNames.Length); BiffRecordWriter.WriteFnGroupCount(buffer); - BiffRecordWriter.WriteWindow1(buffer); + BiffRecordWriter.WriteU16Records(buffer, + [BiffRecord.WinProtect, BiffRecord.Protect, BiffRecord.Password, BiffRecord.Prot4Rev, BiffRecord.Prot4RevPass], 0); + BiffRecordWriter.WriteWindow1(buffer, activeSheet); BiffRecordWriter.WriteBackup(buffer); BiffRecordWriter.WriteHideObj(buffer); - if (date1904) - { - BiffRecordWriter.WriteDate1904(buffer, date1904: true); - } + BiffRecordWriter.WriteDate1904(buffer, date1904); BiffRecordWriter.WritePrecision(buffer); BiffRecordWriter.WriteRefreshAll(buffer); BiffRecordWriter.WriteBookBool(buffer); - BiffRecordWriter.WriteFont(buffer); + // Excel writes at least four FONT records; every XF here refers to the first. + for (int i = 0; i < 4; i++) + { + BiffRecordWriter.WriteFont(buffer); + } Dictionary numFmtIds = styles.AssignCustomNumberFormatIds(); foreach (var (format, id) in numFmtIds) { @@ -62,6 +66,7 @@ internal static int[] Write(BiffBuffer buffer, ReadOnlySpan sheetNames, BiffRecordWriter.WriteBoundSheet(buffer, sheetOffset: 0, sheetNames[i], sheetVisibilities[i]); } + sharedStrings.Write(buffer); BiffRecordWriter.WriteEof(buffer); return offsetPositions; } diff --git a/src/ExcelReader.Core/Writer/Xls/XlsRk.cs b/src/ExcelReader.Core/Writer/Xls/XlsRk.cs new file mode 100644 index 00000000..2b3dbe5e --- /dev/null +++ b/src/ExcelReader.Core/Writer/Xls/XlsRk.cs @@ -0,0 +1,66 @@ +namespace ExcelReader.Core.Writer.Xls +{ + /// + /// Encodes a number as an RkNumber, the same in [MS-XLS] and [MS-XLSB]: a 30-bit integer, or the top + /// 30 bits of a double, either one optionally divided by 100. A value is encoded only when decoding + /// gives back the same bits, so RK never changes a number; anything else keeps an 8-byte double. + /// + internal static class XlsRk + { + private const double IntLimit = 1 << 29; + private const long LowBits = 0x3_FFFF_FFFF; + + internal static bool TryEncode(double value, out int rk) + { + // Excel picks the truncated double whenever it is exact, so that form goes first. + long bits = BitConverter.DoubleToInt64Bits(value); + if ((bits & LowBits) == 0) + { + rk = (int)(bits >> 32); + return true; + } + if (Math.Abs(value) < IntLimit) + { + int whole = (int)value; + if (whole == value) + { + rk = (whole << 2) | 2; + return true; + } + } + + // Only these two forms divide by 100 on decoding, so only they need decoding to prove exact. + // value * 100 can miss an integer by an ulp (0.07 * 100) while n / 100 still gives value back. + double scaled = value * 100; + if (Math.Abs(scaled) < IntLimit) + { + double hundredths = Math.Round(scaled); + if (Math.Abs(scaled - hundredths) < 1e-6 && Matches((((int)hundredths) << 2) | 3, bits, out rk)) + { + return true; + } + } + long scaledBits = BitConverter.DoubleToInt64Bits(scaled); + if ((scaledBits & LowBits) == 0 && Matches(((int)(scaledBits >> 32) & ~3) | 1, bits, out rk)) + { + return true; + } + rk = 0; + return false; + } + + internal static double Decode(int rk) + { + double value = (rk & 0x02) != 0 + ? rk >> 2 + : BitConverter.Int64BitsToDouble((long)((ulong)(uint)(rk & ~3) << 32)); + return (rk & 0x01) != 0 ? value / 100.0 : value; + } + + private static bool Matches(int candidate, long bits, out int rk) + { + rk = candidate; + return BitConverter.DoubleToInt64Bits(Decode(candidate)) == bits; + } + } +} diff --git a/src/ExcelReader.Core/Writer/Xls/XlsSharedStrings.cs b/src/ExcelReader.Core/Writer/Xls/XlsSharedStrings.cs new file mode 100644 index 00000000..0191d8eb --- /dev/null +++ b/src/ExcelReader.Core/Writer/Xls/XlsSharedStrings.cs @@ -0,0 +1,231 @@ +using System.Buffers.Binary; +using System.Runtime.InteropServices; +using ExcelReader.Core.Writer.Internal; + +namespace ExcelReader.Core.Writer.Xls +{ + /// + /// The SST and EXTSST records for text longer than : Excel truncates a + /// LABEL cell to that length on load and refuses a LABEL continued past one record, so longer text + /// goes through LABELSST instead. + /// + /// + /// Each string is encoded once, straight into its SST form (character count, flags, characters), + /// and deduplicated by comparing those bytes; the record boundaries are only drawn in + /// . + /// + internal sealed class XlsSharedStrings : IDisposable + { + internal const int MaxLabelLength = 255; + + private const int Continue = 0x003C; + private const int Sst = 0x00FC; + private const int ExtSst = 0x00FF; + private const int EntryHeaderLength = 3; + + // Strings are found by a hash of a few sampled bytes. Text that differs only between the samples + // piles into one chain; once a chain holds this many, its strings move to a hash of every byte, + // so no chain is ever searched linearly past this length. + private const int MaxSampleChain = 8; + private const int Crowded = -1; + + private BiffBuffer? _entries; + private readonly List _starts = []; + private readonly List _next = []; + private readonly Dictionary _bySample = []; + private readonly Dictionary _byFullHash = []; + private int _references; + + internal int Add(ReadOnlySpan value) + { + _references++; + BiffBuffer entries = _entries ??= new BiffBuffer(64 * 1024); + int start = entries.Length; + bool compressed = BiffStringEncoder.CanCompress(value); + entries.WriteU16(value.Length); + entries.WriteByte((byte)(compressed ? 0 : 1)); + BiffStringEncoder.WriteChars(entries, value, compressed); + ReadOnlySpan entry = entries.Span[start..]; + + ref int head = ref CollectionsMarshal.GetValueRefOrAddDefault(_bySample, SampleHash(entry), out bool exists); + if (!exists) + { + head = Append(start, next: -1); + return head; + } + if (head != Crowded) + { + int length = 0; + for (int candidate = head; candidate >= 0; candidate = _next[candidate], length++) + { + if (Entry(candidate, start).SequenceEqual(entry)) + { + entries.Truncate(start); + return candidate; + } + } + if (length < MaxSampleChain) + { + head = Append(start, next: head); + return head; + } + MoveToFullHash(head, start); + head = Crowded; + } + + ref int fullHead = ref CollectionsMarshal.GetValueRefOrAddDefault(_byFullHash, FullHash(entry), out bool hashed); + if (hashed) + { + for (int candidate = fullHead; candidate >= 0; candidate = _next[candidate]) + { + if (Entry(candidate, start).SequenceEqual(entry)) + { + entries.Truncate(start); + return candidate; + } + } + } + fullHead = Append(start, next: hashed ? fullHead : -1); + return fullHead; + } + + private int Append(int start, int next) + { + _starts.Add(start); + _next.Add(next); + return _starts.Count - 1; + } + + private void MoveToFullHash(int chain, int end) + { + for (int index = chain; index >= 0;) + { + int following = _next[index]; + ref int fullHead = ref CollectionsMarshal.GetValueRefOrAddDefault(_byFullHash, FullHash(Entry(index, end)), out bool hashed); + _next[index] = hashed ? fullHead : -1; + fullHead = index; + index = following; + } + } + + /// An upper bound on what adds, so the globals buffer can be sized once. + internal int EstimatedLength + { + get + { + if (_entries is null) + { + return 0; + } + int records = (_entries.Length / (BiffRecord.MaxPayload - EntryHeaderLength)) + 2; + return _entries.Length + (records * 5) + 12 + ((_starts.Count / 8) + 1) * 8; + } + } + + /// Writes SST and EXTSST at the buffer's end, which must be their position in the Workbook stream. + internal void Write(BiffBuffer buffer) + { + int count = _starts.Count; + if (_entries is null || count == 0) + { + return; + } + int perBucket = Math.Max(8, (count + 127) / 128); + var buckets = new List<(int Position, int RecordOffset)>((count + perBucket - 1) / perBucket); + + int lengthPos = buffer.BeginRecord(Sst); + buffer.WriteI32(_references); + buffer.WriteI32(count); + for (int i = 0; i < count; i++) + { + ReadOnlySpan entry = Entry(i, _entries.Length); + byte flags = entry[2]; + int charSize = flags == 0 ? 1 : 2; + if (Room(buffer, lengthPos) < EntryHeaderLength + charSize) + { + lengthPos = NextRecord(buffer, lengthPos); + } + if (i % perBucket == 0) + { + buckets.Add((buffer.Length, buffer.Length - (lengthPos - 2))); + } + buffer.Write(entry[..EntryHeaderLength]); + ReadOnlySpan chars = entry[EntryHeaderLength..]; + while (!chars.IsEmpty) + { + int take = Math.Min(Room(buffer, lengthPos) / charSize * charSize, chars.Length); + if (take == 0) + { + lengthPos = NextRecord(buffer, lengthPos); + buffer.WriteByte(flags); + continue; + } + buffer.Write(chars[..take]); + chars = chars[take..]; + } + } + buffer.EndRecord(lengthPos); + + int extSst = buffer.BeginRecord(ExtSst); + buffer.WriteU16(perBucket); + foreach ((int position, int recordOffset) in buckets) + { + buffer.WriteI32(position); + buffer.WriteU16(recordOffset); + buffer.WriteU16(0); + } + buffer.EndRecord(extSst); + } + + public void Dispose() + { + _entries?.Dispose(); + } + + private ReadOnlySpan Entry(int index, int end) + { + int start = _starts[index]; + int next = index + 1 < _starts.Count ? _starts[index + 1] : end; + return _entries!.Span[start..next]; + } + + /// + /// Hashes the length, the first and last 32 bytes and three 8-byte samples in between: hashing + /// all of every long string cost more than the rest of writing it. Every entry is longer than + /// 255 bytes, so each read is in range. + /// + private static int SampleHash(ReadOnlySpan entry) + { + ReadOnlySpan head = MemoryMarshal.Cast(entry[..32]); + ReadOnlySpan tail = MemoryMarshal.Cast(entry[^32..]); + var hasher = default(HashCode); + hasher.Add(entry.Length); + hasher.Add(head[0] ^ tail[0]); + hasher.Add(head[1] ^ tail[1]); + hasher.Add(head[2] ^ tail[2]); + hasher.Add(head[3] ^ tail[3]); + hasher.Add(BinaryPrimitives.ReadUInt64LittleEndian(entry[(entry.Length / 4)..])); + hasher.Add(BinaryPrimitives.ReadUInt64LittleEndian(entry[(entry.Length / 2)..])); + hasher.Add(BinaryPrimitives.ReadUInt64LittleEndian(entry[(entry.Length * 3 / 4)..])); + return hasher.ToHashCode(); + } + + private static int FullHash(ReadOnlySpan entry) + { + var hasher = default(HashCode); + hasher.AddBytes(entry); + return hasher.ToHashCode(); + } + + private static int Room(BiffBuffer buffer, int lengthPos) + { + return BiffRecord.MaxPayload - (buffer.Length - lengthPos - 2); + } + + private static int NextRecord(BiffBuffer buffer, int lengthPos) + { + buffer.EndRecord(lengthPos); + return buffer.BeginRecord(Continue); + } + } +} diff --git a/src/ExcelReader.Core/Writer/Xls/XlsSheetLayout.cs b/src/ExcelReader.Core/Writer/Xls/XlsSheetLayout.cs new file mode 100644 index 00000000..e598e252 --- /dev/null +++ b/src/ExcelReader.Core/Writer/Xls/XlsSheetLayout.cs @@ -0,0 +1,135 @@ +using System.Runtime.InteropServices; +using ExcelReader.Core.Writer.Internal; + +namespace ExcelReader.Core.Writer.Xls +{ + /// + /// Lays out one worksheet substream in the order [MS-XLS] requires: INDEX, the sheet settings, + /// DEFCOLWIDTH and COLINFO, DIMENSIONS, the cell table in blocks of up to 32 rows (their ROW records, + /// their cells, then a DBCELL pointing back at both), WINDOW2 and EOF. The cells stay in the sheet's + /// own buffer and are written between the framing records rather than copied. + /// + internal sealed class XlsSheetLayout : IDisposable + { + private const int RowsPerBlock = 32; + private const int RowRecordLength = 4 + 16; + + private readonly BiffBuffer _frame = new(1024); + private readonly ReadOnlyMemory _cells; + private readonly List _segments = []; + private readonly List<(int Position, int Relative)> _filePointers = []; + + internal XlsSheetLayout(XlsSheetWriter sheet, bool selected) + { + _cells = sheet.CellsMemory; + ReadOnlySpan rows = sheet.Rows; + int blockCount = (rows.Length + RowsPerBlock - 1) / RowsPerBlock; + + BiffRecordWriter.WriteBof(_frame, BiffRecord.SubstreamWorksheet); + int index = _frame.BeginRecord(BiffRecord.Index); + _frame.WriteI32(0); + _frame.WriteI32(0); + _frame.WriteI32(rows.Length); + int defColWidthPointer = _frame.Length; + _frame.WriteI32(0); + int dbCellPointers = _frame.Length; + for (int i = 0; i < blockCount; i++) + { + _frame.WriteI32(0); + } + _frame.EndRecord(index); + + BiffRecordWriter.WriteSheetSettings(_frame); + _filePointers.Add((defColWidthPointer, _frame.Length)); + BiffRecordWriter.WriteDefColWidth(_frame); + _frame.Write(sheet.ColInfoMemory.Span); + BiffRecordWriter.WriteDimension(_frame, sheet.RowCount, sheet.ColCount); + AddFrame(0); + + int position = _frame.Length; + for (int block = 0; block < blockCount; block++) + { + int first = block * RowsPerBlock; + int count = Math.Min(RowsPerBlock, rows.Length - first); + int cellsStart = rows[first].CellStart; + int cellsEnd = first + count < rows.Length ? rows[first + count].CellStart : _cells.Length; + + int rowsFrame = _frame.Length; + for (int i = first; i < first + count; i++) + { + BiffRecordWriter.WriteRow(_frame, i, rows[i].ColMic, rows[i].ColMac); + } + AddFrame(rowsFrame); + _segments.Add(new Segment(Cells: true, cellsStart, cellsEnd - cellsStart)); + int blockLength = (count * RowRecordLength) + (cellsEnd - cellsStart); + position += blockLength; + + _filePointers.Add((dbCellPointers + (4 * block), position)); + int dbCellFrame = _frame.Length; + int dbCell = _frame.BeginRecord(BiffRecord.DbCell); + _frame.WriteI32(blockLength); + // The first offset runs from the block's second ROW record to the first row's cells; + // each later one is the length of the previous row's cells. + _frame.WriteU16((count - 1) * RowRecordLength); + for (int i = first + 1; i < first + count; i++) + { + _frame.WriteU16(Math.Min(rows[i].CellStart - rows[i - 1].CellStart, ushort.MaxValue)); + } + _frame.EndRecord(dbCell); + AddFrame(dbCellFrame); + position += _frame.Length - dbCellFrame; + } + + int tailFrame = _frame.Length; + BiffRecordWriter.WriteWindow2(_frame, selected); + BiffRecordWriter.WriteEof(_frame); + AddFrame(tailFrame); + Length = position + (_frame.Length - tailFrame); + } + + internal int Length { get; } + + /// INDEX holds absolute stream positions, known only once the globals have been sized. + internal void Place(int substreamStart) + { + foreach ((int pointer, int relative) in _filePointers) + { + _frame.PatchI32(pointer, substreamStart + relative); + } + } + + internal void WriteTo(Stream destination) + { + foreach (Segment segment in _segments) + { + destination.Write(Slice(segment).Span); + } + } + + internal async ValueTask WriteToAsync(Stream destination, CancellationToken ct) + { + foreach (Segment segment in _segments) + { + await destination.WriteAsync(Slice(segment), ct).ConfigureAwait(false); + } + } + + public void Dispose() + { + _frame.Dispose(); + } + + private void AddFrame(int start) + { + _segments.Add(new Segment(Cells: false, start, _frame.Length - start)); + } + + private ReadOnlyMemory Slice(Segment segment) + { + return (segment.Cells ? _cells : _frame.Memory).Slice(segment.Start, segment.Length); + } + + [StructLayout(LayoutKind.Auto)] + private readonly record struct Segment(bool Cells, int Start, int Length); + } +} diff --git a/src/ExcelReader.Core/Writer/Xls/XlsSheetWriter.cs b/src/ExcelReader.Core/Writer/Xls/XlsSheetWriter.cs index 86517f61..23af5443 100644 --- a/src/ExcelReader.Core/Writer/Xls/XlsSheetWriter.cs +++ b/src/ExcelReader.Core/Writer/Xls/XlsSheetWriter.cs @@ -1,4 +1,7 @@ using System.Diagnostics.CodeAnalysis; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using ExcelReader.Core.Reader.Internal; using ExcelReader.Core.Writer.Internal; namespace ExcelReader.Core.Writer.Xls @@ -14,8 +17,6 @@ public sealed class XlsSheetWriter : ISheetWriter internal const int MaxColumn = 255; private const int MaxSheetNameLength = 31; - private const int FramingBytes = 20 + 18 + 22 + 4; - private readonly XlsWorkbookWriter _owner; private readonly bool _date1904; private readonly bool _isContinuation; @@ -29,6 +30,18 @@ public sealed class XlsSheetWriter : ISheetWriter private int _maxRow = -1; private int _maxCol = -1; private int _rowNumber = -1; + [SuppressMessage("Usage", "CA2213:Disposable fields should be disposed", + Justification = "Read while the workbook is written; released with _cells via ReleaseBuffer.")] + private readonly BiffBuffer _rows = new(4096); + private int _rowCellStart; + private int _rowColMic; + private int _rowColMax; + + // Numbers in consecutive columns of the current row are appended to one open MULRK record, ended + // by a gap or another kind of cell; see BiffRecordWriter.BeginRkRun. + private int _rkRecord; + private int _rkCount; + private int _rkFirstCol; private WriterState _state = WriterState.Created; private bool _rowActive; private bool _rowHasCells; @@ -52,13 +65,12 @@ internal XlsSheetWriter(XlsWorkbookWriter owner, string name, bool date1904, internal ExcelSheetVisibility Visibility { get; } - internal int SubstreamLength => FramingBytes + _colInfos.Length + _cells.Length; - internal int RowCount => _maxRow + 1; internal int ColCount => _maxCol + 1; internal ReadOnlyMemory CellsMemory => _cells.Memory; internal ReadOnlyMemory ColInfoMemory => _colInfos.Memory; - internal bool ResourcesReleased => _cells.IsReleased && _colInfos.IsReleased; + internal ReadOnlySpan Rows => MemoryMarshal.Cast(_rows.Span); + internal bool ResourcesReleased => _cells.IsReleased && _colInfos.IsReleased && _rows.IsReleased; private bool HasActiveRow => _rowActive || (_continuation?.HasActiveRow ?? false); [SuppressMessage("Usage", "CA2213:Disposable fields should be disposed", @@ -70,7 +82,7 @@ internal XlsSheetWriter(XlsWorkbookWriter owner, string name, bool date1904, /// The sheet has already been started. public void SetColumnStyle(int columnIndex, int styleId) { - SheetColumnValidation.SetColumnStyle(ref _columnStyles, columnIndex, styleId, _owner.StyleCount, _state, this); + SheetColumnValidation.SetColumnStyle(ref _columnStyles, columnIndex, styleId, _owner.StyleCount, MaxColumn + 1, _state, this); } /// @@ -78,7 +90,7 @@ public void SetColumnStyle(int columnIndex, int styleId) /// The sheet has already been started. public void SetColumnWidth(int columnIndex, double width) { - SheetColumnValidation.SetColumnWidth(ref _columnWidths, columnIndex, width, _state, this); + SheetColumnValidation.SetColumnWidth(ref _columnWidths, columnIndex, width, MaxColumn + 1, _state, this); } private int EffectiveStyle(int columnIndex) @@ -160,6 +172,7 @@ public XlsRowWriter StartRow(int styleId) _activeRowStyle = styleId; _rowActive = true; _rowHasCells = false; + _rowCellStart = _cells.Length; _rowWriter ??= new XlsRowWriter(this, _rowNumber); _rowWriter.Reset(_rowNumber); return _rowWriter; @@ -168,11 +181,18 @@ public XlsRowWriter StartRow(int styleId) /// A row left without cells gets one blank cell, since BIFF8 has no other trace of it for a reader to find. internal void NotifyRowEnded() { + FlushRkRun(); if (_rowActive && !_rowHasCells) { BiffRecordWriter.WriteBlank(_cells, _rowNumber, 0, XlsGlobals.GeneralXf); Track(_rowNumber, 0); } + if (_rowActive) + { + var extent = new XlsRowExtent(_rowCellStart, _rowColMic, _rowColMax + 1); + MemoryMarshal.Write(_rows.GetSpan(Unsafe.SizeOf()), in extent); + _rows.Advance(Unsafe.SizeOf()); + } _rowActive = false; } @@ -180,7 +200,7 @@ internal void EmitNumber(int row, int col, double value) { ValidateColumn(col); int abstractStyle = EffectiveStyle(col); - BiffRecordWriter.WriteNumber(_cells, row, col, abstractStyle == 0 ? XlsGlobals.GeneralXf : XlsGlobals.CustomXf(abstractStyle), value); + WriteNumberCell(row, col, abstractStyle == 0 ? XlsGlobals.GeneralXf : XlsGlobals.CustomXf(abstractStyle), value); Track(row, col); } @@ -189,15 +209,25 @@ internal void EmitDate(int row, int col, DateTime value) ValidateColumn(col); int abstractStyle = EffectiveStyle(col); double serial = ExcelEpoch.OADateToSerial(value.ToOADate(), _date1904); - BiffRecordWriter.WriteNumber(_cells, row, col, abstractStyle == 0 ? XlsGlobals.DateXf : XlsGlobals.CustomXf(abstractStyle), serial); + WriteNumberCell(row, col, abstractStyle == 0 ? XlsGlobals.DateXf : XlsGlobals.CustomXf(abstractStyle), serial); Track(row, col); } internal void EmitLabel(int row, int col, ReadOnlySpan value) { ValidateColumn(col); + ExcelLimits.ThrowIfCellTextTooLong(value.Length, nameof(value)); int abstractStyle = EffectiveStyle(col); - BiffRecordWriter.WriteLabel(_cells, row, col, abstractStyle == 0 ? XlsGlobals.GeneralXf : XlsGlobals.CustomXf(abstractStyle), value); + int xf = abstractStyle == 0 ? XlsGlobals.GeneralXf : XlsGlobals.CustomXf(abstractStyle); + FlushRkRun(); + if (value.Length > XlsSharedStrings.MaxLabelLength) + { + BiffRecordWriter.WriteLabelSst(_cells, row, col, xf, _owner.AddSharedString(value)); + } + else + { + BiffRecordWriter.WriteLabel(_cells, row, col, xf, value); + } Track(row, col); } @@ -205,18 +235,56 @@ internal void EmitBool(int row, int col, bool value) { ValidateColumn(col); int abstractStyle = EffectiveStyle(col); + FlushRkRun(); BiffRecordWriter.WriteBool(_cells, row, col, abstractStyle == 0 ? XlsGlobals.GeneralXf : XlsGlobals.CustomXf(abstractStyle), value); Track(row, col); } + private void WriteNumberCell(int row, int col, int xf, double value) + { + CellValueGuards.ThrowIfNonFinite(value, nameof(value)); + if (!XlsRk.TryEncode(value, out int rk)) + { + FlushRkRun(); + BiffRecordWriter.WriteNumber(_cells, row, col, xf, value); + return; + } + if (_rkCount > 0 && col != _rkFirstCol + _rkCount) + { + FlushRkRun(); + } + if (_rkCount == 0) + { + _rkFirstCol = col; + _rkRecord = BiffRecordWriter.BeginRkRun(_cells, row, col); + } + BiffRecordWriter.AppendRk(_cells, xf, rk); + _rkCount++; + } + + private void FlushRkRun() + { + if (_rkCount > 0) + { + BiffRecordWriter.EndRkRun(_cells, _rkRecord, _rkFirstCol, _rkCount); + _rkCount = 0; + } + } + internal void ReleaseBuffer() { _cells.Dispose(); _colInfos.Dispose(); + _rows.Dispose(); } private void Track(int row, int col) { + if (!_rowHasCells) + { + _rowColMic = col; + } + _rowColMax = col; _rowHasCells = true; if (row > _maxRow) { _maxRow = row; } if (col > _maxCol) { _maxCol = col; } @@ -299,4 +367,7 @@ public ValueTask DisposeAsync() return ValueTask.CompletedTask; } } + + [StructLayout(LayoutKind.Auto)] + internal readonly record struct XlsRowExtent(int CellStart, int ColMic, int ColMac); } diff --git a/src/ExcelReader.Core/Writer/Xls/XlsWorkbookWriter.cs b/src/ExcelReader.Core/Writer/Xls/XlsWorkbookWriter.cs index 2e88bd17..d9f429ae 100644 --- a/src/ExcelReader.Core/Writer/Xls/XlsWorkbookWriter.cs +++ b/src/ExcelReader.Core/Writer/Xls/XlsWorkbookWriter.cs @@ -1,3 +1,4 @@ +using System.Diagnostics.CodeAnalysis; using System.Runtime.InteropServices; using ExcelReader.Core.Writer.Internal; using ExcelReader.Core.Writer.Xlsx; @@ -20,6 +21,9 @@ public sealed class XlsWorkbookWriter : IWorkbookWriter private readonly bool _leaveOpen; private readonly bool _date1904; private readonly StyleTable _styles = new(); + [SuppressMessage("Usage", "CA2213:Disposable fields should be disposed", + Justification = "Released with the sheet buffers once End or EndAsync has written the workbook.")] + private readonly XlsSharedStrings _sharedStrings = new(); private readonly List _sheets = []; private bool _ended; private XlsSheetWriter? _activeSheet; @@ -76,6 +80,11 @@ public int AddStyle(CellStyle style) internal int StyleCount => _styles.Count; + internal int AddSharedString(ReadOnlySpan value) + { + return _sharedStrings.Add(value); + } + internal void RegisterSheet(XlsSheetWriter sheet) { _sheets.Add(sheet); @@ -87,31 +96,39 @@ internal void NotifySheetEnded() _activeSheet = null; } - private BiffBuffer BuildGlobals(out int workbookSize) + private BiffBuffer BuildGlobals(List layouts, out int workbookSize) { string[] names = new string[_sheets.Count]; ExcelSheetVisibility[] visibilities = new ExcelSheetVisibility[_sheets.Count]; - bool anyVisible = false; + int activeSheet = -1; for (int i = 0; i < _sheets.Count; i++) { names[i] = _sheets[i].Name; visibilities[i] = _sheets[i].Visibility; - anyVisible |= visibilities[i] is ExcelSheetVisibility.Visible; + if (activeSheet < 0 && visibilities[i] is ExcelSheetVisibility.Visible) + { + activeSheet = i; + } + } + WriterStateGuard.RequireVisibleSheet(activeSheet >= 0, nameof(XlsWorkbookWriter)); + + for (int i = 0; i < _sheets.Count; i++) + { + layouts.Add(new XlsSheetLayout(_sheets[i], selected: i == activeSheet)); } - WriterStateGuard.RequireVisibleSheet(anyVisible, nameof(XlsWorkbookWriter)); - BiffBuffer globals = new(1024); + BiffBuffer globals = new(4096 + _sharedStrings.EstimatedLength); try { - int[] offsetPositions = XlsGlobals.Write(globals, names, visibilities, _date1904, _styles); + int[] offsetPositions = XlsGlobals.Write(globals, names, visibilities, activeSheet, _date1904, _styles, _sharedStrings); int offset = globals.Length; - workbookSize = offset; - for (int i = 0; i < _sheets.Count; i++) + for (int i = 0; i < layouts.Count; i++) { globals.PatchI32(offsetPositions[i], offset); - offset += _sheets[i].SubstreamLength; - workbookSize += _sheets[i].SubstreamLength; + layouts[i].Place(offset); + offset += layouts[i].Length; } + workbookSize = offset; return globals; } catch @@ -144,8 +161,17 @@ private ValueTask ReleaseStreamAsync() return _faulted ? FailureCleanup.DisposeAsync(_stream) : _stream.DisposeAsync(); } + private static void DisposeLayouts(List layouts) + { + foreach (XlsSheetLayout layout in layouts) + { + layout.Dispose(); + } + } + private void ReleaseSheetBuffers() { + _sharedStrings.Dispose(); foreach (ref readonly var sheet in CollectionsMarshal.AsSpan(_sheets)) { sheet.ReleaseBuffer(); @@ -169,25 +195,23 @@ public void End() throw new InvalidOperationException("A workbook must contain at least one sheet."); } - using BiffBuffer globals = BuildGlobals(out int workbookSize); - using BiffBuffer frame = new(64); - OleCompoundWriter.Write(_stream, workbookSize, dest => + List layouts = []; + try { - dest.Write(globals.Span); - foreach (ref readonly var sheet in CollectionsMarshal.AsSpan(_sheets)) + using BiffBuffer globals = BuildGlobals(layouts, out int workbookSize); + OleCompoundWriter.Write(_stream, workbookSize, dest => { - frame.Reset(); - BiffRecordWriter.WriteBof(frame, BiffRecord.SubstreamWorksheet); - BiffRecordWriter.WriteDimension(frame, sheet.RowCount, sheet.ColCount); - BiffRecordWriter.WriteWindow2(frame); - dest.Write(frame.Span); - dest.Write(sheet.ColInfoMemory.Span); - dest.Write(sheet.CellsMemory.Span); - frame.Reset(); - BiffRecordWriter.WriteEof(frame); - dest.Write(frame.Span); - } - }); + dest.Write(globals.Span); + foreach (XlsSheetLayout layout in layouts) + { + layout.WriteTo(dest); + } + }); + } + finally + { + DisposeLayouts(layouts); + } } catch { @@ -218,25 +242,23 @@ public async ValueTask EndAsync(CancellationToken ct = default) throw new InvalidOperationException("A workbook must contain at least one sheet."); } - using BiffBuffer globals = BuildGlobals(out int workbookSize); - using BiffBuffer frame = new(64); - await OleCompoundWriter.WriteAsync(_stream, workbookSize, async (dest, canc) => + List layouts = []; + try { - await dest.WriteAsync(globals.Memory, canc).ConfigureAwait(false); - foreach (XlsSheetWriter sheet in _sheets) + using BiffBuffer globals = BuildGlobals(layouts, out int workbookSize); + await OleCompoundWriter.WriteAsync(_stream, workbookSize, async (dest, canc) => { - frame.Reset(); - BiffRecordWriter.WriteBof(frame, BiffRecord.SubstreamWorksheet); - BiffRecordWriter.WriteDimension(frame, sheet.RowCount, sheet.ColCount); - BiffRecordWriter.WriteWindow2(frame); - await dest.WriteAsync(frame.Memory, canc).ConfigureAwait(false); - await dest.WriteAsync(sheet.ColInfoMemory, canc).ConfigureAwait(false); - await dest.WriteAsync(sheet.CellsMemory, canc).ConfigureAwait(false); - frame.Reset(); - BiffRecordWriter.WriteEof(frame); - await dest.WriteAsync(frame.Memory, canc).ConfigureAwait(false); - } - }, ct).ConfigureAwait(false); + await dest.WriteAsync(globals.Memory, canc).ConfigureAwait(false); + foreach (XlsSheetLayout layout in layouts) + { + await layout.WriteToAsync(dest, canc).ConfigureAwait(false); + } + }, ct).ConfigureAwait(false); + } + finally + { + DisposeLayouts(layouts); + } } catch { diff --git a/src/ExcelReader.Core/Writer/Xlsb/XlsbRowWriter.cs b/src/ExcelReader.Core/Writer/Xlsb/XlsbRowWriter.cs index 2d89071d..48ee3a69 100644 --- a/src/ExcelReader.Core/Writer/Xlsb/XlsbRowWriter.cs +++ b/src/ExcelReader.Core/Writer/Xlsb/XlsbRowWriter.cs @@ -215,9 +215,7 @@ internal static double ToDouble(T value) where T : IUtf8SpanFormattable } if (value is IConvertible convertible) { - double converted = convertible.ToDouble(CultureInfo.InvariantCulture); - CellValueGuards.ThrowIfNonFiniteConversion(converted, typeof(T), nameof(value)); - return converted; + return ConvertToDouble(convertible, typeof(T)); } Span bytes = stackalloc byte[64]; if (value.TryFormat(bytes, out int written, default, CultureInfo.InvariantCulture) && @@ -226,8 +224,25 @@ internal static double ToDouble(T value) where T : IUtf8SpanFormattable CellValueGuards.ThrowIfNonFiniteConversion(parsed, typeof(T), nameof(value)); return parsed; } - throw new ArgumentException( - $"Cannot convert value of type '{typeof(T)}' to a numeric cell value.", nameof(value)); + CellValueGuards.ThrowUnrepresentable(typeof(T), nameof(value)); + return default; + } + + // IConvertible types with no numeric form (char, DateTime) throw InvalidCastException from ToDouble. + private static double ConvertToDouble(IConvertible value, Type sourceType) + { + double converted; + try + { + converted = value.ToDouble(CultureInfo.InvariantCulture); + } + catch (InvalidCastException) + { + CellValueGuards.ThrowUnrepresentable(sourceType, nameof(value)); + return default; + } + CellValueGuards.ThrowIfNonFiniteConversion(converted, sourceType, nameof(value)); + return converted; } /// diff --git a/src/ExcelReader.Core/Writer/Xlsb/XlsbSheetWriter.cs b/src/ExcelReader.Core/Writer/Xlsb/XlsbSheetWriter.cs index 61baab2f..a4ad1730 100644 --- a/src/ExcelReader.Core/Writer/Xlsb/XlsbSheetWriter.cs +++ b/src/ExcelReader.Core/Writer/Xlsb/XlsbSheetWriter.cs @@ -1,24 +1,34 @@ using System.Buffers.Binary; +using System.Diagnostics.CodeAnalysis; using System.IO.Compression; using System.Runtime.InteropServices; using ExcelReader.Core.Reader.Internal; using ExcelReader.Core.Reader.Xlsb; using ExcelReader.Core.Writer.Internal; +using ExcelReader.Core.Writer.Parallel; +using ExcelReader.Core.Writer.Xls; +using ExcelReader.Core.Writer.Zip; namespace ExcelReader.Core.Writer.Xlsb { /// Writes a single worksheet's rows into an .xlsb workbook produced by . - public sealed class XlsbSheetWriter : ISheetWriter, ITableSheetWriter + public sealed class XlsbSheetWriter : ISheetWriter, ITableSheetWriter, IBatchSheet { private const int SpillThreshold = 64 * 1024; private readonly XlsbWorkbookWriter _owner; - private readonly ZipArchive _zip; + private readonly ZipPackageWriter _zip; private readonly bool _date1904; private readonly CompressionLevel _compression; private readonly bool _offloadWrite; - private readonly BiffBuffer _records = new(4096); + private readonly BiffBuffer _records; private readonly TableTracker _tables = new(); + private readonly int _spillThreshold = SpillThreshold; + private readonly WorkerStringCache? _workerStrings; + [SuppressMessage("Usage", "CA2213:Disposable fields should be disposed", Justification = "An alias of the entry that _stream is or wraps; disposing _stream disposes it.")] + private ZipEntryStream? _entry; + private bool _segmentsOpen; + [SuppressMessage("Usage", "CA2213:Disposable fields should be disposed", Justification = "End disposes it and Fault cleans it up; the analyzer loses it once _entry aliases the same entry.")] private Stream? _stream; private WriterState _state = WriterState.Created; private bool _rowActive; @@ -40,7 +50,7 @@ public sealed class XlsbSheetWriter : ISheetWriter, ITableSheetWr internal XlsbSheetWriter( XlsbWorkbookWriter owner, - ZipArchive zip, + ZipPackageWriter zip, string name, int sheetId, ExcelSheetVisibility visibility, @@ -56,6 +66,33 @@ internal XlsbSheetWriter( _date1904 = date1904; _compression = compression; _offloadWrite = offloadWrite; + _records = new BiffBuffer(4096); + } + + // A detached copy formats batches for the parallel record path: it has its own record buffer, + // never spills it and never touches the ZIP. + private XlsbSheetWriter(XlsbSheetWriter source, int batchRows) + { + _records = new BiffBuffer(DetachedCapacity(batchRows)); + _owner = source._owner; + _zip = source._zip; + Name = source.Name; + SheetId = source.SheetId; + Visibility = source.Visibility; + _date1904 = source._date1904; + _compression = source._compression; + _columnStyles = source._columnStyles; + _spillThreshold = int.MaxValue; + _state = WriterState.Started; + if (_owner.UseSharedStrings) + { + _workerStrings = new WorkerStringCache(_owner.SharedStrings!, _owner.SharedStringsGate); + } + } + + private static int DetachedCapacity(int batchRows) + { + return (int)Math.Min(1 << 20, (long)batchRows * 128); } internal string Name { get; } @@ -78,7 +115,7 @@ internal IReadOnlyList Tables /// The first row has already been started. public void SetColumnStyle(int columnIndex, int styleId) { - SheetColumnValidation.SetColumnStyle(ref _columnStyles, columnIndex, styleId, _owner.StyleCount, _state, this); + SheetColumnValidation.SetColumnStyle(ref _columnStyles, columnIndex, styleId, _owner.StyleCount, ExcelLimits.MaxColumns, _state, this); } /// @@ -86,7 +123,7 @@ public void SetColumnStyle(int columnIndex, int styleId) /// The first row has already been started. public void SetColumnWidth(int columnIndex, double width) { - SheetColumnValidation.SetColumnWidth(ref _columnWidths, columnIndex, width, _state, this); + SheetColumnValidation.SetColumnWidth(ref _columnWidths, columnIndex, width, ExcelLimits.MaxColumns, _state, this); } private int EffectiveStyle(int columnIndex) @@ -350,10 +387,21 @@ private async ValueTask FaultAsync() Release(faulted: true); } + internal void Abandon() + { + Fault(); + } + + internal ValueTask AbandonAsync() + { + return FaultAsync(); + } + private void Release(bool faulted) { _state = WriterState.Ended; _stream = null; + _entry = null; ReleaseBuffers(); if (!faulted && !_registered) { @@ -392,7 +440,7 @@ internal void WriteRecord(int id, ReadOnlySpan payload = default) private void MaybeFlush() { - if (!_rowActive && _records.Length >= SpillThreshold) + if (!_rowActive && _records.Length >= _spillThreshold) { FlushRecords(); } @@ -404,9 +452,9 @@ private void EnsureStream() { return; } - ZipArchiveEntry entry = _zip.CreateEntry($"xl/worksheets/sheet{SheetId}.bin", _compression); - Stream stream = entry.Open(); - _stream = _offloadWrite ? new WriteOffloadStream(stream) : stream; + ZipEntryStream entry = _zip.OpenEntry($"xl/worksheets/sheet{SheetId}.bin", _compression); + _entry = entry; + _stream = _offloadWrite ? new WriteOffloadStream(entry) : entry; } private void FlushRecords() @@ -437,15 +485,13 @@ private void FlushRecords() private void WriteBufferedSheet() { - ZipArchiveEntry entry = _zip.CreateEntry($"xl/worksheets/sheet{SheetId}.bin", _compression); - using Stream stream = entry.Open(); + using Stream stream = _zip.OpenEntry($"xl/worksheets/sheet{SheetId}.bin", _compression); stream.Write(_records.Span); } private async ValueTask WriteBufferedSheetAsync(CancellationToken ct) { - ZipArchiveEntry entry = _zip.CreateEntry($"xl/worksheets/sheet{SheetId}.bin", _compression); - Stream stream = await entry.OpenAsync(ct).ConfigureAwait(false); + Stream stream = _zip.OpenEntry($"xl/worksheets/sheet{SheetId}.bin", _compression); await using (stream.ConfigureAwait(false)) { await stream.WriteAsync(_records.Memory, ct).ConfigureAwait(false); @@ -468,9 +514,9 @@ private void BeginRow(int styleId) WriterStateGuard.ThrowIfEnded(_state, this); WriterStateGuard.RequireNoActiveRowForStart(_rowActive, nameof(XlsbRowWriter)); EnsureStarted(); - if (_rowNumber >= ExcelLimits.MaxRows) + if (_rowNumber >= ExcelLimits.MaxRows - 1) { - ExcelLimits.ThrowRowLimit(_rowNumber + 1L); + ExcelLimits.ThrowRowLimit(_rowNumber + 2L); } _rowNumber++; _activeRowStyle = styleId; @@ -614,8 +660,6 @@ private void WriteCell(int columnIndex, XlsbCell cell) } private const int CellHeaderLength = 8; - private const int RkIntMin = -(1 << 29); - private const int RkIntMax = (1 << 29) - 1; internal void WriteStringCell(int columnIndex, string? value) { @@ -639,7 +683,7 @@ private void WriteTextCell(int columnIndex, ReadOnlySpan value, string? ow int style = EffectiveStyle(columnIndex); if (_owner.UseSharedStrings) { - int index = owned is null ? _owner.GetSharedStringIndex(value) : _owner.GetSharedStringIndex(owned); + int index = SharedStringIndex(value, owned); const int Length = CellHeaderLength + 4; Biff12RecordWriter.WriteFixedRecord(_records, Brt.CellIsst, Length, out Span shared); Biff12RecordWriter.WriteCellHeader(shared, columnIndex, style); @@ -655,6 +699,23 @@ private void WriteTextCell(int columnIndex, ReadOnlySpan value, string? ow MaybeFlush(); } + private int SharedStringIndex(ReadOnlySpan value, string? owned) + { + if (_workerStrings is null) + { + if (owned is null) + { + return _owner.GetSharedStringIndex(value); + } + return _owner.GetSharedStringIndex(owned); + } + if (owned is null) + { + return _workerStrings.GetOrAdd(value); + } + return _workerStrings.GetOrAdd(owned); + } + internal void WriteBoolCell(int columnIndex, bool value) { ValidateColumn(columnIndex); @@ -682,12 +743,12 @@ private void WriteDoubleCellCore(int columnIndex, double value, int style) ValidateColumn(columnIndex); CellValueGuards.ThrowIfNonFinite(value, nameof(value)); TrackColumn(columnIndex); - if (TryEncodeRk(value, out uint rk)) + if (XlsRk.TryEncode(value, out int rk)) { const int RkLength = CellHeaderLength + 4; Biff12RecordWriter.WriteFixedRecord(_records, Brt.CellRk, RkLength, out Span rkPayload); Biff12RecordWriter.WriteCellHeader(rkPayload, columnIndex, style); - BinaryPrimitives.WriteUInt32LittleEndian(rkPayload.Slice(8, 4), rk); + BinaryPrimitives.WriteInt32LittleEndian(rkPayload.Slice(8, 4), rk); MaybeFlush(); return; } @@ -698,22 +759,125 @@ private void WriteDoubleCellCore(int columnIndex, double value, int style) MaybeFlush(); } - private static bool TryEncodeRk(double value, out uint rk) + private static void ValidateColumn(int columnIndex) { - rk = 0; - if (value != Math.Truncate(value) || value < RkIntMin || value > RkIntMax - || (value == 0 && double.IsNegative(value))) + ExcelLimits.ThrowIfColumnOutOfRange(columnIndex); + } + + CompressionLevel IBatchSheet.BatchCompression + { + get { - return false; + return _compression; } - ulong bits = BitConverter.DoubleToUInt64Bits(value); - rk = (bits & 0x3_FFFF_FFFFUL) == 0 ? (uint)(bits >> 32) : ((uint)(int)value << 2) | 0x02; - return true; } - private static void ValidateColumn(int columnIndex) + ValueTask IBatchSheet.BeginSegmentsAsync(CancellationToken ct) { - ExcelLimits.ThrowIfColumnOutOfRange(columnIndex); + WriterStateGuard.ThrowIfEnded(_state, this); + WriterStateGuard.RequireNoActiveRowForStart(_rowActive, nameof(XlsbRowWriter)); + ct.ThrowIfCancellationRequested(); + EnsureStarted(); + // Held for the whole stretch, so StartRow and End are refused until it is over. + _rowActive = true; + _segmentsOpen = false; + return ValueTask.FromResult(_rowNumber + 1); + } + + async ValueTask IBatchSheet.AppendSegmentAsync(BatchSegment segment, CancellationToken ct) + { + WriterStateGuard.ThrowIfEnded(_state, this); + MergeUsedRange(segment); + if (!_segmentsOpen) + { + await OpenSegmentsAsync(ct).ConfigureAwait(false); + } + await _entry!.AppendSegmentAsync(segment.Data.Memory, segment.Crc, segment.RawLength, ct).ConfigureAwait(false); + _rowNumber += segment.Rows; + } + + // The sheet header leaves the buffer here, not when the stretch begins, so the dimension record + // it carries sees the first batch's rows and columns. + private async ValueTask OpenSegmentsAsync(CancellationToken ct) + { + FlushRecords(); + try + { + EnsureStream(); + if (_stream is WriteOffloadStream offload) + { + await offload.DrainAsync(ct).ConfigureAwait(false); + } + } + catch + { + await FaultAsync().ConfigureAwait(false); + throw; + } + _segmentsOpen = true; + } + + private void MergeUsedRange(BatchSegment segment) + { + if (segment.RowLast < 0) + { + return; + } + if (_usedRowFirst < 0) + { + _usedRowFirst = segment.RowFirst; + } + _usedRowLast = segment.RowLast; + _usedColFirst = Math.Min(_usedColFirst, segment.ColumnFirst); + _usedColLast = Math.Max(_usedColLast, segment.ColumnLast); + } + + void IBatchSheet.EndSegments() + { + _rowActive = false; + _segmentsOpen = false; + } + + async ValueTask IBatchSheet.FailSegmentsAsync() + { + _rowActive = false; + await FaultAsync().ConfigureAwait(false); + } + + XlsbSheetWriter IBatchSheet.CreateDetached(int batchRows) + { + return new XlsbSheetWriter(this, batchRows); + } + + void IBatchSheet.BeginBatch(int rowsBefore) + { + _records.Reset(); + _rowNumber = rowsBefore - 1; + _rowActive = false; + _usedRowFirst = -1; + _usedRowLast = -1; + _usedColFirst = int.MaxValue; + _usedColLast = -1; + } + + XlsbRowWriter IBatchSheet.StartBatchRow() + { + return StartRow(); + } + + ReadOnlySpan IBatchSheet.EndBatch(BatchSegment segment) + { + _workerStrings?.FlushReferences(); + segment.RowFirst = _usedRowFirst; + segment.RowLast = _usedRowLast; + segment.ColumnFirst = _usedColFirst; + segment.ColumnLast = _usedColLast; + return _records.Span; + } + + void IBatchSheet.ReleaseDetached() + { + ReleaseBuffers(); } } } diff --git a/src/ExcelReader.Core/Writer/Xlsb/XlsbWorkbookWriter.cs b/src/ExcelReader.Core/Writer/Xlsb/XlsbWorkbookWriter.cs index ab7b373d..436c7c96 100644 --- a/src/ExcelReader.Core/Writer/Xlsb/XlsbWorkbookWriter.cs +++ b/src/ExcelReader.Core/Writer/Xlsb/XlsbWorkbookWriter.cs @@ -4,13 +4,14 @@ using System.Text; using ExcelReader.Core.Reader.Xlsb; using ExcelReader.Core.Writer.Internal; +using ExcelReader.Core.Writer.Zip; namespace ExcelReader.Core.Writer.Xlsb { /// Writes an .xlsb workbook (BIFF12 binary format) to a stream, one sheet at a time. public sealed class XlsbWorkbookWriter : IWorkbookWriter { - private readonly ZipArchive _zip; + private readonly ZipPackageWriter _zip; private readonly Stream _stream; private readonly bool _leaveOpen; private readonly bool _date1904; @@ -26,7 +27,7 @@ public sealed class XlsbWorkbookWriter : IWorkbookWriter private bool _zipClosed; private bool _disposed; - private XlsbWorkbookWriter(ZipArchive zip, Stream stream, bool leaveOpen, XlsbWriterOptions options) + private XlsbWorkbookWriter(ZipPackageWriter zip, Stream stream, bool leaveOpen, XlsbWriterOptions options) { _zip = zip; _stream = stream; @@ -46,7 +47,7 @@ private XlsbWorkbookWriter(ZipArchive zip, Stream stream, bool leaveOpen, XlsbWr public static XlsbWorkbookWriter Create(Stream stream, bool leaveOpen = false, XlsbWriterOptions? options = null) { ArgumentNullException.ThrowIfNull(stream); - ZipArchive zip = new(stream, ZipArchiveMode.Create, leaveOpen: true); + ZipPackageWriter zip = new(stream); return new XlsbWorkbookWriter(zip, stream, leaveOpen, options ?? XlsbWriterOptions.Default); } @@ -96,6 +97,16 @@ internal int GetSharedStringIndex(ReadOnlySpan value) return _sharedStrings!.GetOrAdd(value); } + internal SharedStringTable? SharedStrings + { + get + { + return _sharedStrings; + } + } + + internal Lock SharedStringsGate { get; } = new(); + /// public int AddStyle(CellStyle style) { @@ -162,7 +173,7 @@ public async ValueTask EndAsync(CancellationToken ct = default) await WriteAppPropertiesAsync(ct).ConfigureAwait(false); await WriteTablesAsync(ct).ConfigureAwait(false); await WriteContentTypesAsync(ct).ConfigureAwait(false); - await CloseZipAsync().ConfigureAwait(false); + await CloseZipAsync(ct).ConfigureAwait(false); } catch { @@ -174,34 +185,38 @@ public async ValueTask EndAsync(CancellationToken ct = default) private void CloseZip() { _zipClosed = true; - _zip.Dispose(); + _zip.Finish(); } - private ValueTask CloseZipAsync() + private ValueTask CloseZipAsync(CancellationToken ct = default) { _zipClosed = true; - return _zip.DisposeAsync(); + return _zip.FinishAsync(ct); } private void AbandonZip() { _faulted = true; + _activeSheet?.Abandon(); if (!_zipClosed) { _zipClosed = true; - FailureCleanup.Dispose(_zip); + _zip.Abandon(); } } - private ValueTask AbandonZipAsync() + private async ValueTask AbandonZipAsync() { _faulted = true; - if (_zipClosed) + if (_activeSheet is not null) { - return ValueTask.CompletedTask; + await _activeSheet.AbandonAsync().ConfigureAwait(false); + } + if (!_zipClosed) + { + _zipClosed = true; + await _zip.AbandonAsync().ConfigureAwait(false); } - _zipClosed = true; - return FailureCleanup.DisposeAsync(_zip); } private void ReleaseStream() diff --git a/src/ExcelReader.Core/Writer/Xlsx/XlsxSheetWriter.cs b/src/ExcelReader.Core/Writer/Xlsx/XlsxSheetWriter.cs index 9bb2c5f1..20340061 100644 --- a/src/ExcelReader.Core/Writer/Xlsx/XlsxSheetWriter.cs +++ b/src/ExcelReader.Core/Writer/Xlsx/XlsxSheetWriter.cs @@ -1,26 +1,33 @@ using System.Buffers.Text; +using System.Diagnostics.CodeAnalysis; using System.Globalization; using System.IO.Compression; using System.Text; using ExcelReader.Core.Reader; using ExcelReader.Core.Reader.Internal; using ExcelReader.Core.Writer.Internal; +using ExcelReader.Core.Writer.Parallel; +using ExcelReader.Core.Writer.Zip; namespace ExcelReader.Core.Writer.Xlsx { /// /// Writes a single worksheet's XML into an XLSX ZIP archive, buffering rows and flushing them to the entry stream once they cross a size threshold. /// - public sealed class XlsxSheetWriter : ISheetWriter, ITableSheetWriter + public sealed class XlsxSheetWriter : ISheetWriter, ITableSheetWriter, IBatchSheet { private readonly TableTracker _tables = new(); private readonly XlsxWorkbookWriter _owner; - private readonly ZipArchive _zip; + private readonly ZipPackageWriter _zip; private readonly CompressionLevel _compression; private readonly bool _offloadWrite; - private readonly BiffBuffer _rowBuffer = new(512); + private readonly BiffBuffer _rowBuffer; // ponytail: flush to the deflate stream once buffered rows pass 64 KB — bounds memory on huge private const int FlushThreshold = 64 * 1024; + private readonly int _flushThreshold = FlushThreshold; + private readonly WorkerStringCache? _workerStrings; + [SuppressMessage("Usage", "CA2213:Disposable fields should be disposed", Justification = "An alias of the entry that _stream is or wraps; disposing _stream disposes it.")] + private ZipEntryStream? _entry; private XlsxRowWriter? _rowWriter; private Stream? _stream; private int _rowNumber; @@ -29,7 +36,7 @@ public sealed class XlsxSheetWriter : ISheetWriter, ITableSheetWr private Dictionary? _columnStyles; private Dictionary? _columnWidths; - internal XlsxSheetWriter(XlsxWorkbookWriter owner, ZipArchive zip, string name, int sheetId, + internal XlsxSheetWriter(XlsxWorkbookWriter owner, ZipPackageWriter zip, string name, int sheetId, ExcelSheetVisibility visibility, CompressionLevel compression, bool offloadWrite) { _owner = owner; @@ -39,6 +46,32 @@ internal XlsxSheetWriter(XlsxWorkbookWriter owner, ZipArchive zip, string name, Visibility = visibility; _compression = compression; _offloadWrite = offloadWrite; + _rowBuffer = new BiffBuffer(512); + } + + // A detached copy formats batches for the parallel record path: it has its own row buffer, + // never flushes it and never touches the ZIP. + private XlsxSheetWriter(XlsxSheetWriter source, int batchRows) + { + _rowBuffer = new BiffBuffer(DetachedCapacity(batchRows)); + _owner = source._owner; + _zip = source._zip; + Name = source.Name; + SheetId = source.SheetId; + Visibility = source.Visibility; + _compression = source._compression; + _columnStyles = source._columnStyles; + _flushThreshold = int.MaxValue; + _state = WriterState.Started; + if (_owner.UseSharedStrings) + { + _workerStrings = new WorkerStringCache(_owner.SharedStrings!, _owner.SharedStringsGate); + } + } + + private static int DetachedCapacity(int batchRows) + { + return (int)Math.Min(1 << 20, (long)batchRows * 128); } internal string Name { get; } @@ -51,12 +84,20 @@ internal XlsxSheetWriter(XlsxWorkbookWriter owner, ZipArchive zip, string name, internal int GetSharedStringIndex(string value) { - return _owner.GetSharedStringIndex(value); + if (_workerStrings is null) + { + return _owner.GetSharedStringIndex(value); + } + return _workerStrings.GetOrAdd(value); } internal int GetSharedStringIndex(ReadOnlySpan value) { - return _owner.GetSharedStringIndex(value); + if (_workerStrings is null) + { + return _owner.GetSharedStringIndex(value); + } + return _workerStrings.GetOrAdd(value); } internal int GetColumnStyle(int columnIndex) @@ -69,7 +110,7 @@ internal int GetColumnStyle(int columnIndex) /// The first row has already been started. public void SetColumnStyle(int columnIndex, int styleId) { - SheetColumnValidation.SetColumnStyle(ref _columnStyles, columnIndex, styleId, _owner.StyleCount, _state, this); + SheetColumnValidation.SetColumnStyle(ref _columnStyles, columnIndex, styleId, _owner.StyleCount, ExcelLimits.MaxColumns, _state, this); } /// @@ -77,7 +118,7 @@ public void SetColumnStyle(int columnIndex, int styleId) /// The first row has already been started. public void SetColumnWidth(int columnIndex, double width) { - SheetColumnValidation.SetColumnWidth(ref _columnWidths, columnIndex, width, _state, this); + SheetColumnValidation.SetColumnWidth(ref _columnWidths, columnIndex, width, ExcelLimits.MaxColumns, _state, this); } private void EnsureStarted() @@ -88,9 +129,9 @@ private void EnsureStarted() } try { - ZipArchiveEntry entry = _zip.CreateEntry($"xl/worksheets/sheet{SheetId}.xml", _compression); - Stream stream = entry.Open(); - _stream = _offloadWrite ? new WriteOffloadStream(stream) : stream; + ZipEntryStream entry = _zip.OpenEntry($"xl/worksheets/sheet{SheetId}.xml", _compression); + _entry = entry; + _stream = _offloadWrite ? new WriteOffloadStream(entry) : entry; _rowBuffer.Reset(); _rowBuffer.WriteUtf8( "" + @@ -117,9 +158,9 @@ private async ValueTask StartCoreAsync(CancellationToken ct) { try { - ZipArchiveEntry entry = _zip.CreateEntry($"xl/worksheets/sheet{SheetId}.xml", _compression); - Stream stream = await entry.OpenAsync(ct).ConfigureAwait(false); - _stream = _offloadWrite ? new WriteOffloadStream(stream) : stream; + ZipEntryStream entry = _zip.OpenEntry($"xl/worksheets/sheet{SheetId}.xml", _compression); + _entry = entry; + _stream = _offloadWrite ? new WriteOffloadStream(entry) : entry; _rowBuffer.Reset(); _rowBuffer.WriteUtf8( "" + @@ -345,10 +386,21 @@ private async ValueTask FaultAsync() Release(faulted: true); } + internal void Abandon() + { + Fault(); + } + + internal ValueTask AbandonAsync() + { + return FaultAsync(); + } + private void Release(bool faulted) { _state = WriterState.Ended; _stream = null; + _entry = null; _rowBuffer.Dispose(); _owner.NotifySheetEnded(faulted); } @@ -436,7 +488,7 @@ private void CloseBufferedRow() internal void EndBufferedRow() { CloseBufferedRow(); - if (_rowBuffer.Length >= FlushThreshold) + if (_rowBuffer.Length >= _flushThreshold) { FlushRowBuffer(); } @@ -465,7 +517,7 @@ private void FlushRowBuffer() internal ValueTask EndBufferedRowAsync(CancellationToken ct = default) { CloseBufferedRow(); - return _rowBuffer.Length >= FlushThreshold ? FlushRowBufferAsync(ct) : ValueTask.CompletedTask; + return _rowBuffer.Length >= _flushThreshold ? FlushRowBufferAsync(ct) : ValueTask.CompletedTask; } private async ValueTask FlushRowBufferAsync(CancellationToken ct) @@ -489,5 +541,85 @@ private async ValueTask FlushRowBufferAsync(CancellationToken ct) throw; } } + + CompressionLevel IBatchSheet.BatchCompression + { + get + { + return _compression; + } + } + + async ValueTask IBatchSheet.BeginSegmentsAsync(CancellationToken ct) + { + WriterStateGuard.ThrowIfEnded(_state, this); + WriterStateGuard.RequireNoActiveRowForStart(_rowActive, nameof(XlsxRowWriter)); + await EnsureStartedAsync(ct).ConfigureAwait(false); + if (_rowBuffer.Length > 0) + { + await FlushRowBufferAsync(ct).ConfigureAwait(false); + } + if (_stream is WriteOffloadStream offload) + { + try + { + await offload.DrainAsync(ct).ConfigureAwait(false); + } + catch + { + await FaultAsync().ConfigureAwait(false); + throw; + } + } + // Held for the whole stretch, so StartRow and End are refused until it is over. + _rowActive = true; + return _rowNumber; + } + + async ValueTask IBatchSheet.AppendSegmentAsync(BatchSegment segment, CancellationToken ct) + { + WriterStateGuard.ThrowIfEnded(_state, this); + await _entry!.AppendSegmentAsync(segment.Data.Memory, segment.Crc, segment.RawLength, ct).ConfigureAwait(false); + _rowNumber += segment.Rows; + } + + void IBatchSheet.EndSegments() + { + _rowActive = false; + } + + async ValueTask IBatchSheet.FailSegmentsAsync() + { + _rowActive = false; + await FaultAsync().ConfigureAwait(false); + } + + XlsxSheetWriter IBatchSheet.CreateDetached(int batchRows) + { + return new XlsxSheetWriter(this, batchRows); + } + + void IBatchSheet.BeginBatch(int rowsBefore) + { + _rowBuffer.Reset(); + _rowNumber = rowsBefore; + _rowActive = false; + } + + XlsxRowWriter IBatchSheet.StartBatchRow() + { + return StartRow(); + } + + ReadOnlySpan IBatchSheet.EndBatch(BatchSegment segment) + { + _workerStrings?.FlushReferences(); + return _rowBuffer.Span; + } + + void IBatchSheet.ReleaseDetached() + { + _rowBuffer.Dispose(); + } } } diff --git a/src/ExcelReader.Core/Writer/Xlsx/XlsxWorkbookWriter.cs b/src/ExcelReader.Core/Writer/Xlsx/XlsxWorkbookWriter.cs index 34b40541..2969f907 100644 --- a/src/ExcelReader.Core/Writer/Xlsx/XlsxWorkbookWriter.cs +++ b/src/ExcelReader.Core/Writer/Xlsx/XlsxWorkbookWriter.cs @@ -3,6 +3,7 @@ using System.Security; using System.Text; using ExcelReader.Core.Writer.Internal; +using ExcelReader.Core.Writer.Zip; namespace ExcelReader.Core.Writer.Xlsx { @@ -11,7 +12,7 @@ namespace ExcelReader.Core.Writer.Xlsx /// public sealed class XlsxWorkbookWriter : IWorkbookWriter { - private readonly ZipArchive _zip; + private readonly ZipPackageWriter _zip; private readonly Stream _stream; private readonly bool _leaveOpen; private readonly CompressionLevel _compression; @@ -28,7 +29,7 @@ public sealed class XlsxWorkbookWriter : IWorkbookWriter private bool _zipClosed; private bool _disposed; - private XlsxWorkbookWriter(ZipArchive zip, Stream stream, bool leaveOpen, XlsxWriterOptions options) + private XlsxWorkbookWriter(ZipPackageWriter zip, Stream stream, bool leaveOpen, XlsxWriterOptions options) { _zip = zip; _stream = stream; @@ -50,7 +51,7 @@ private XlsxWorkbookWriter(ZipArchive zip, Stream stream, bool leaveOpen, XlsxWr public static XlsxWorkbookWriter Create(Stream stream, bool leaveOpen = false, XlsxWriterOptions? options = null) { ArgumentNullException.ThrowIfNull(stream); - ZipArchive zip = new(stream, ZipArchiveMode.Create, leaveOpen: true); + ZipPackageWriter zip = new(stream); return new XlsxWorkbookWriter(zip, stream, leaveOpen, options ?? XlsxWriterOptions.Default); } @@ -145,6 +146,16 @@ internal int GetSharedStringIndex(ReadOnlySpan value) return _sharedStrings!.GetOrAdd(value); } + internal SharedStringTable? SharedStrings + { + get + { + return _sharedStrings; + } + } + + internal Lock SharedStringsGate { get; } = new(); + /// public int AddStyle(CellStyle style) { @@ -218,7 +229,7 @@ public async ValueTask EndAsync(CancellationToken ct = default) await WriteWorkbookAsync(ct).ConfigureAwait(false); await WriteWorkbookRelsAsync(ct).ConfigureAwait(false); await WriteContentTypesAsync(ct).ConfigureAwait(false); - await CloseZipAsync().ConfigureAwait(false); + await CloseZipAsync(ct).ConfigureAwait(false); } catch { @@ -230,34 +241,38 @@ public async ValueTask EndAsync(CancellationToken ct = default) private void CloseZip() { _zipClosed = true; - _zip.Dispose(); + _zip.Finish(); } - private ValueTask CloseZipAsync() + private ValueTask CloseZipAsync(CancellationToken ct = default) { _zipClosed = true; - return _zip.DisposeAsync(); + return _zip.FinishAsync(ct); } private void AbandonZip() { _faulted = true; + _activeSheet?.Abandon(); if (!_zipClosed) { _zipClosed = true; - FailureCleanup.Dispose(_zip); + _zip.Abandon(); } } - private ValueTask AbandonZipAsync() + private async ValueTask AbandonZipAsync() { _faulted = true; - if (_zipClosed) + if (_activeSheet is not null) { - return ValueTask.CompletedTask; + await _activeSheet.AbandonAsync().ConfigureAwait(false); + } + if (!_zipClosed) + { + _zipClosed = true; + await _zip.AbandonAsync().ConfigureAwait(false); } - _zipClosed = true; - return FailureCleanup.DisposeAsync(_zip); } private void ReleaseStream() diff --git a/src/ExcelReader.Core/Writer/Zip/BufferSink.cs b/src/ExcelReader.Core/Writer/Zip/BufferSink.cs new file mode 100644 index 00000000..2a910804 --- /dev/null +++ b/src/ExcelReader.Core/Writer/Zip/BufferSink.cs @@ -0,0 +1,97 @@ +using ExcelReader.Core.Writer.Internal; + +namespace ExcelReader.Core.Writer.Zip +{ + /// + /// A write-only stream that appends to a it does not own. While + /// is set it drops what it is given, which is how the final block a + /// DeflateStream writes on dispose is kept out of a stream of segments. + /// + internal sealed class BufferSink : Stream + { + private readonly BiffBuffer _target; + + internal BufferSink(BiffBuffer target) + { + _target = target; + } + + internal bool Discard { get; set; } + + public override bool CanRead + { + get + { + return false; + } + } + + public override bool CanSeek + { + get + { + return false; + } + } + + public override bool CanWrite + { + get + { + return true; + } + } + + public override long Length + { + get + { + throw new NotSupportedException(); + } + } + + public override long Position + { + get + { + throw new NotSupportedException(); + } + set + { + throw new NotSupportedException(); + } + } + + public override void Flush() + { + } + + public override int Read(byte[] buffer, int offset, int count) + { + throw new NotSupportedException(); + } + + public override long Seek(long offset, SeekOrigin origin) + { + throw new NotSupportedException(); + } + + public override void SetLength(long value) + { + throw new NotSupportedException(); + } + + public override void Write(byte[] buffer, int offset, int count) + { + Write(buffer.AsSpan(offset, count)); + } + + public override void Write(ReadOnlySpan buffer) + { + if (!Discard) + { + _target.Write(buffer); + } + } + } +} diff --git a/src/ExcelReader.Core/Writer/Zip/Crc32.cs b/src/ExcelReader.Core/Writer/Zip/Crc32.cs new file mode 100644 index 00000000..baccde2a --- /dev/null +++ b/src/ExcelReader.Core/Writer/Zip/Crc32.cs @@ -0,0 +1,277 @@ +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using System.Runtime.Intrinsics; +using System.Runtime.Intrinsics.X86; +using ArmCrc32 = System.Runtime.Intrinsics.Arm.Crc32; + +namespace ExcelReader.Core.Writer.Zip +{ + /// + /// The ZIP CRC-32: reflected polynomial 0xEDB88320, register inverted on entry and exit. + /// chains, so Append(Append(0, a), b) equals Append(0, a + b). + /// + /// + /// Long inputs are folded with carry-less multiplication, after Intel's "Fast CRC Computation for + /// Generic Polynomials Using PCLMULQDQ Instruction". The folded 128-bit remainder is finished by + /// running its 16 bytes through the byte table from a zero register. + /// + internal static class Crc32 + { + private const uint ReflectedPolynomial = 0xEDB8_8320; + private const uint Polynomial = 0x04C1_1DB7; + + private static readonly uint[] Table = BuildTable(); + private static readonly uint[] SquaredPowersOfX = BuildSquaredPowersOfX(); + private static readonly Vector128 Fold128By128 = FoldConstants(128); + private static readonly Vector128 Fold128By512 = FoldConstants(512); + private static readonly Vector256 Fold256By256 = Vector256.Create(FoldConstants(256), FoldConstants(256)); + private static readonly Vector256 Fold256By1024 = Vector256.Create(FoldConstants(1024), FoldConstants(1024)); + + internal static uint Append(uint crc, ReadOnlySpan data) + { + if (Pclmulqdq.V256.IsSupported && data.Length >= 256) + { + return AppendFold256(crc, data); + } + if (Pclmulqdq.IsSupported && data.Length >= 128) + { + return AppendFold128(crc, data); + } + if (ArmCrc32.Arm64.IsSupported) + { + return AppendArm64(crc, data); + } + return AppendTable(crc, data); + } + + internal static uint AppendTable(uint crc, ReadOnlySpan data) + { + return ~TableUpdate(~crc, data); + } + + internal static uint AppendArm64(uint crc, ReadOnlySpan data) + { + uint register = ~crc; + ref byte source = ref MemoryMarshal.GetReference(data); + int index = 0; + for (; index + 8 <= data.Length; index += 8) + { + register = ArmCrc32.Arm64.ComputeCrc32(register, Unsafe.ReadUnaligned(ref Unsafe.Add(ref source, index))); + } + for (; index < data.Length; index++) + { + register = ArmCrc32.ComputeCrc32(register, Unsafe.Add(ref source, index)); + } + return ~register; + } + + internal static uint AppendFold128(uint crc, ReadOnlySpan data) + { + if (data.Length < 64) + { + return AppendTable(crc, data); + } + uint register = Fold128(~crc, ref data); + return ~TableUpdate(register, data); + } + + internal static uint AppendFold256(uint crc, ReadOnlySpan data) + { + if (data.Length < 128) + { + return AppendTable(crc, data); + } + uint register = Fold256(~crc, ref data); + return ~TableUpdate(register, data); + } + + private static uint TableUpdate(uint register, ReadOnlySpan data) + { + ref uint table = ref MemoryMarshal.GetArrayDataReference(Table); + foreach (byte value in data) + { + register = Unsafe.Add(ref table, (byte)(register ^ value)) ^ (register >> 8); + } + return register; + } + + private static uint Fold256(uint register, ref ReadOnlySpan data) + { + ref byte source = ref MemoryMarshal.GetReference(data); + Vector256 by1024 = Fold256By1024; + Vector256 by256 = Fold256By256; + Vector256 x0 = Load256(ref source, 0) ^ Vector256.CreateScalar((ulong)register); + Vector256 x1 = Load256(ref source, 32); + Vector256 x2 = Load256(ref source, 64); + Vector256 x3 = Load256(ref source, 96); + int position = 128; + while (data.Length - position >= 128) + { + x0 = Fold(x0, by1024) ^ Load256(ref source, position); + x1 = Fold(x1, by1024) ^ Load256(ref source, position + 32); + x2 = Fold(x2, by1024) ^ Load256(ref source, position + 64); + x3 = Fold(x3, by1024) ^ Load256(ref source, position + 96); + position += 128; + } + x1 ^= Fold(x0, by256); + x2 ^= Fold(x1, by256); + x3 ^= Fold(x2, by256); + Vector128 folded = Fold(x3.GetLower(), Fold128By128) ^ x3.GetUpper(); + return Finish(folded, ref source, position, ref data); + } + + private static uint Fold128(uint register, ref ReadOnlySpan data) + { + ref byte source = ref MemoryMarshal.GetReference(data); + Vector128 by512 = Fold128By512; + Vector128 by128 = Fold128By128; + Vector128 x0 = Load128(ref source, 0) ^ Vector128.CreateScalar((ulong)register); + Vector128 x1 = Load128(ref source, 16); + Vector128 x2 = Load128(ref source, 32); + Vector128 x3 = Load128(ref source, 48); + int position = 64; + while (data.Length - position >= 64) + { + x0 = Fold(x0, by512) ^ Load128(ref source, position); + x1 = Fold(x1, by512) ^ Load128(ref source, position + 16); + x2 = Fold(x2, by512) ^ Load128(ref source, position + 32); + x3 = Fold(x3, by512) ^ Load128(ref source, position + 48); + position += 64; + } + x1 ^= Fold(x0, by128); + x2 ^= Fold(x1, by128); + Vector128 folded = x3 ^ Fold(x2, by128); + return Finish(folded, ref source, position, ref data); + } + + private static uint Finish(Vector128 folded, ref byte source, int position, ref ReadOnlySpan data) + { + while (data.Length - position >= 16) + { + folded = Fold(folded, Fold128By128) ^ Load128(ref source, position); + position += 16; + } + data = data[position..]; + Span remainder = stackalloc byte[16]; + folded.AsByte().CopyTo(remainder); + return TableUpdate(0, remainder); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector256 Fold(Vector256 lanes, Vector256 constants) + { + return Pclmulqdq.V256.CarrylessMultiply(lanes, constants, 0x00) ^ Pclmulqdq.V256.CarrylessMultiply(lanes, constants, 0x11); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector128 Fold(Vector128 lanes, Vector128 constants) + { + return Pclmulqdq.CarrylessMultiply(lanes, constants, 0x00) ^ Pclmulqdq.CarrylessMultiply(lanes, constants, 0x11); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector256 Load256(ref byte source, int offset) + { + return Vector256.LoadUnsafe(ref Unsafe.Add(ref source, offset)).AsUInt64(); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector128 Load128(ref byte source, int offset) + { + return Vector128.LoadUnsafe(ref Unsafe.Add(ref source, offset)).AsUInt64(); + } + + // Moving a 128-bit lane forward by D bits multiplies its low qword by k(D + 32) and its high + // qword by k(D - 32), where k(e) = reflect32(x^e mod P) << 1. + private static Vector128 FoldConstants(int distanceBits) + { + return Vector128.Create(FoldConstant(distanceBits + 32), FoldConstant(distanceBits - 32)); + } + + private static ulong FoldConstant(int exponent) + { + uint remainder = 1; + for (int i = 0; i < exponent; i++) + { + uint carry = 0u - (remainder >> 31); + remainder = (remainder << 1) ^ (Polynomial & carry); + } + uint reflected = 0; + for (int bit = 0; bit < 32; bit++) + { + reflected = (reflected << 1) | ((remainder >> bit) & 1); + } + return (ulong)reflected << 1; + } + + private static uint[] BuildTable() + { + var table = new uint[256]; + for (uint index = 0; index < 256; index++) + { + uint entry = index; + for (int bit = 0; bit < 8; bit++) + { + entry = (entry >> 1) ^ (ReflectedPolynomial & (0u - (entry & 1))); + } + table[index] = entry; + } + return table; + } + + /// + /// The CRC of two byte runs laid end to end, from the CRC of each and the length of the second. + /// + internal static uint Combine(uint first, uint second, long secondLength) + { + ArgumentOutOfRangeException.ThrowIfNegative(secondLength); + return MultiplyModP(PowerOfX(secondLength), first) ^ second; + } + + // x^(8 * byteCount) mod P, by square and multiply over the bits of byteCount. Entry k of the table + // is x^(2^k); the sequence repeats every 32 entries, and bytes start three squarings in. + private static uint PowerOfX(long byteCount) + { + uint power = 1u << 31; + int exponent = 3; + while (byteCount != 0) + { + if ((byteCount & 1) != 0) + { + power = MultiplyModP(SquaredPowersOfX[exponent & 31], power); + } + byteCount >>= 1; + exponent++; + } + return power; + } + + // Polynomial product mod P in the reflected domain, where bit 31 is x^0. + private static uint MultiplyModP(uint a, uint b) + { + uint product = 0; + for (uint mask = 1u << 31; mask != 0; mask >>= 1) + { + if ((a & mask) != 0) + { + product ^= b; + } + b = (b >> 1) ^ (ReflectedPolynomial & (0u - (b & 1))); + } + return product; + } + + private static uint[] BuildSquaredPowersOfX() + { + var table = new uint[32]; + uint power = 1u << 30; + table[0] = power; + for (int index = 1; index < table.Length; index++) + { + power = MultiplyModP(power, power); + table[index] = power; + } + return table; + } + } +} diff --git a/src/ExcelReader.Core/Writer/Zip/DeflateEncoder.cs b/src/ExcelReader.Core/Writer/Zip/DeflateEncoder.cs new file mode 100644 index 00000000..80b41fe5 --- /dev/null +++ b/src/ExcelReader.Core/Writer/Zip/DeflateEncoder.cs @@ -0,0 +1,272 @@ +using System.Buffers; +using System.Buffers.Binary; +using System.Numerics; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using ExcelReader.Core.Writer.Internal; + +namespace ExcelReader.Core.Writer.Zip +{ + /// + /// Raw deflate (RFC 1951) for CompressionLevel.Fastest: one hash candidate per position, a + /// 4-byte minimum match, no lazy matching and fixed Huffman codes. Input is encoded in blocks of at + /// most bytes, and a block the fixed codes would enlarge is written + /// stored instead. + /// + internal sealed class DeflateEncoder : IDisposable + { + internal const int MaxBlockLength = 65_535; + + private const int HashBits = 16; + private const int HistoryLength = 32_768; + private const int MinMatch = 4; + private const int MaxMatch = 258; + private const int WindowCapacity = HistoryLength + (1 << 20); + // Every byte a 9-bit literal, plus the block header, end-of-block, pending bits and the 8 bytes + // each symbol writes ahead of the position it advances to. + private const int MaxEncodedBlock = (MaxBlockLength * 9 / 8) + 24; + + private byte[] _window; + private uint[] _table; + private int _encoded; + private int _end; + private uint _windowStart; + private ulong _bits; + private int _bitCount; + + internal DeflateEncoder() + { + _window = ArrayPool.Shared.Rent(WindowCapacity); + _table = ArrayPool.Shared.Rent(1 << HashBits); + Array.Clear(_table, 0, 1 << HashBits); + } + + internal void Write(ReadOnlySpan input, BiffBuffer output) + { + ObjectDisposedException.ThrowIf(_window.Length == 0, this); + while (!input.IsEmpty) + { + if (_end == WindowCapacity) + { + Slide(); + } + int take = Math.Min(input.Length, WindowCapacity - _end); + input[..take].CopyTo(_window.AsSpan(_end)); + _end += take; + input = input[take..]; + while (_end - _encoded >= MaxBlockLength) + { + EncodeBlock(_encoded + MaxBlockLength, final: false, output); + } + } + } + + internal void Finish(BiffBuffer output) + { + EncodeBlock(_end, final: true, output); + Span tail = output.GetSpan(sizeof(ulong)); + BinaryPrimitives.WriteUInt64LittleEndian(tail, _bits); + output.Advance((_bitCount + 7) / 8); + _bits = 0; + _bitCount = 0; + } + + /// + /// Ends the pending input as a non-final block and appends an empty stored block, so the output + /// ends on a byte boundary and another deflate stream may be laid after it. + /// + internal void FlushSync(BiffBuffer output) + { + EncodeBlock(_end, final: false, output); + Span tail = output.GetSpan(sizeof(ulong) + 4); + int headerBytes = (_bitCount + 3 + 7) / 8; + BinaryPrimitives.WriteUInt64LittleEndian(tail, _bits); + BinaryPrimitives.WriteUInt32LittleEndian(tail[headerBytes..], 0xFFFF_0000); + output.Advance(headerBytes + 4); + _bits = 0; + _bitCount = 0; + } + + public void Dispose() + { + if (_window.Length == 0) + { + return; + } + ArrayPool.Shared.Return(_window); + ArrayPool.Shared.Return(_table); + _window = []; + _table = []; + } + + // Keeps the last 32 KB already encoded as history. The hash table holds absolute positions, so + // moving the window only moves _windowStart; it never rewrites the table. + private void Slide() + { + int keepFrom = Math.Max(0, _encoded - HistoryLength); + _window.AsSpan(keepFrom, _end - keepFrom).CopyTo(_window); + _windowStart += (uint)keepFrom; + _encoded -= keepFrom; + _end -= keepFrom; + } + + private void EncodeBlock(int blockEnd, bool final, BiffBuffer output) + { + Span destination = output.GetSpan(MaxEncodedBlock); + int length = blockEnd - _encoded; + var state = new BitState + { + Bits = _bits | ((final ? 3UL : 2UL) << _bitCount), + Count = _bitCount + 3, + }; + // Up to 31 pending bits plus the 3 header bits can exceed 32; flush so a 31-bit symbol still fits. + FlushWord(destination, ref state); + int next = EncodeMatches(blockEnd, destination, ref state); + EncodeLiterals(next, blockEnd, destination, ref state); + state.Count += 7; + FlushWord(destination, ref state); + long fixedBits = (8L * state.Written) + state.Count - _bitCount; + if (fixedBits <= StoredBits(length)) + { + output.Advance(state.Written); + _bits = state.Bits; + _bitCount = state.Count; + } + else + { + output.Advance(WriteStored(destination, length, final)); + } + _encoded = blockEnd; + } + + private long StoredBits(int length) + { + int padding = (8 - ((_bitCount + 3) & 7)) & 7; + return 3 + padding + 32 + (8L * length); + } + + private int WriteStored(Span destination, int length, bool final) + { + ulong header = _bits | ((final ? 1UL : 0UL) << _bitCount); + int headerBytes = (_bitCount + 3 + 7) / 8; + BinaryPrimitives.WriteUInt64LittleEndian(destination, header); + BinaryPrimitives.WriteUInt16LittleEndian(destination[headerBytes..], (ushort)length); + BinaryPrimitives.WriteUInt16LittleEndian(destination[(headerBytes + 2)..], (ushort)~length); + _window.AsSpan(_encoded, length).CopyTo(destination[(headerBytes + 4)..]); + _bits = 0; + _bitCount = 0; + return headerBytes + 4 + length; + } + + // Called once per block, so tiered compilation would run the loop as on-stack-replacement code + // for the first blocks of a process. + [MethodImpl(MethodImplOptions.AggressiveOptimization)] + private int EncodeMatches(int blockEnd, Span destination, ref BitState state) + { + byte[] windowArray = _window; + uint[] tableArray = _table; + ObjectDisposedException.ThrowIf(windowArray.Length < WindowCapacity || tableArray.Length < 1 << HashBits, this); + ref byte window = ref MemoryMarshal.GetArrayDataReference(windowArray); + ref uint table = ref MemoryMarshal.GetArrayDataReference(tableArray); + ref byte output = ref MemoryMarshal.GetReference(destination); + ref uint literals = ref MemoryMarshal.GetArrayDataReference(FixedHuffmanCodes.Literals); + ref uint lengths = ref MemoryMarshal.GetArrayDataReference(FixedHuffmanCodes.Lengths); + ref uint distances = ref MemoryMarshal.GetArrayDataReference(FixedHuffmanCodes.Distances); + ulong bits = state.Bits; + int count = state.Count; + int written = state.Written; + int index = _encoded; + int lastMatchStart = blockEnd - MinMatch; + while (index <= lastMatchStart) + { + uint current = Unsafe.ReadUnaligned(ref Unsafe.Add(ref window, index)); + ref uint slot = ref Unsafe.Add(ref table, (int)((current * 2654435761u) >> (32 - HashBits))); + uint position = _windowStart + (uint)index; + uint distance = position - slot; + slot = position; + if (distance - 1 < HistoryLength && distance <= (uint)index + && Unsafe.ReadUnaligned(ref Unsafe.Add(ref window, index - (int)distance)) == current) + { + int length = MatchLength(ref window, index - (int)distance, index, Math.Min(MaxMatch, blockEnd - index)); + uint lengthEntry = Unsafe.Add(ref lengths, length); + uint distanceEntry = Unsafe.Add(ref distances, (int)distance); + int lengthBits = (int)(lengthEntry & FixedHuffmanCodes.LengthMask); + ulong code = (lengthEntry >> FixedHuffmanCodes.LengthShift) + | ((ulong)(distanceEntry >> FixedHuffmanCodes.LengthShift) << lengthBits); + bits |= code << count; + count += lengthBits + (int)(distanceEntry & FixedHuffmanCodes.LengthMask); + index += length; + } + else + { + uint literal = Unsafe.Add(ref literals, Unsafe.Add(ref window, index)); + bits |= (ulong)(literal >> FixedHuffmanCodes.LengthShift) << count; + count += (int)(literal & FixedHuffmanCodes.LengthMask); + index++; + } + // Whole bytes leave the buffer after every symbol, so fewer than 8 bits stay pending + // and the next symbol, at most 31 bits, always fits. + Unsafe.WriteUnaligned(ref Unsafe.Add(ref output, written), bits); + written += count >> 3; + bits >>= count & ~7; + count &= 7; + } + state.Bits = bits; + state.Count = count; + state.Written = written; + return index; + } + + private void EncodeLiterals(int index, int blockEnd, Span destination, ref BitState state) + { + for (; index < blockEnd; index++) + { + uint literal = FixedHuffmanCodes.Literals[_window[index]]; + state.Bits |= (ulong)(literal >> FixedHuffmanCodes.LengthShift) << state.Count; + state.Count += (int)(literal & FixedHuffmanCodes.LengthMask); + FlushWord(destination, ref state); + } + } + + private static void FlushWord(Span destination, ref BitState state) + { + if (state.Count < 32) + { + return; + } + BinaryPrimitives.WriteUInt32LittleEndian(destination[state.Written..], (uint)state.Bits); + state.Written += 4; + state.Bits >>= 32; + state.Count -= 32; + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static int MatchLength(ref byte window, int candidate, int current, int maxLength) + { + int length = MinMatch; + while (length + 8 <= maxLength) + { + ulong difference = Unsafe.ReadUnaligned(ref Unsafe.Add(ref window, candidate + length)) + ^ Unsafe.ReadUnaligned(ref Unsafe.Add(ref window, current + length)); + if (difference != 0) + { + return length + (BitOperations.TrailingZeroCount(difference) >> 3); + } + length += 8; + } + while (length < maxLength && Unsafe.Add(ref window, candidate + length) == Unsafe.Add(ref window, current + length)) + { + length++; + } + return length; + } + + [StructLayout(LayoutKind.Auto)] + private struct BitState + { + internal ulong Bits; + internal int Count; + internal int Written; + } + } +} diff --git a/src/ExcelReader.Core/Writer/Zip/FixedHuffmanCodes.cs b/src/ExcelReader.Core/Writer/Zip/FixedHuffmanCodes.cs new file mode 100644 index 00000000..ebebf025 --- /dev/null +++ b/src/ExcelReader.Core/Writer/Zip/FixedHuffmanCodes.cs @@ -0,0 +1,104 @@ +namespace ExcelReader.Core.Writer.Zip +{ + /// + /// The fixed Huffman codes of RFC 1951 section 3.2.6, bit-reversed for LSB-first output. Each entry + /// packs the code, with any extra bits already appended above it, over a 5-bit total length. + /// + internal static class FixedHuffmanCodes + { + internal const int LengthShift = 5; + internal const uint LengthMask = 31; + + private static readonly int[] LengthBases = [3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 15, 17, 19, 23, 27, 31, 35, 43, 51, 59, 67, 83, 99, 115, 131, 163, 195, 227, 258]; + private static readonly int[] LengthExtraBits = [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 4, 5, 5, 5, 5, 0]; + private static readonly int[] DistanceBases = [1, 2, 3, 4, 5, 7, 9, 13, 17, 25, 33, 49, 65, 97, 129, 193, 257, 385, 513, 769, 1025, 1537, 2049, 3073, 4097, 6145, 8193, 12289, 16385, 24577]; + private static readonly int[] DistanceExtraBits = [0, 0, 0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, 8, 9, 9, 10, 10, 11, 11, 12, 12, 13, 13]; + + /// Indexed by byte value. + internal static readonly uint[] Literals = BuildLiterals(); + + /// Indexed by match length, 3 to 258. + internal static readonly uint[] Lengths = BuildLengths(); + + /// Indexed by match distance, 1 to 32768. + internal static readonly uint[] Distances = BuildDistances(); + + private static uint[] BuildLiterals() + { + var table = new uint[256]; + for (int literal = 0; literal < table.Length; literal++) + { + table[literal] = Symbol(literal); + } + return table; + } + + private static uint[] BuildLengths() + { + var table = new uint[259]; + for (int index = 0; index < LengthBases.Length; index++) + { + uint symbol = Symbol(257 + index); + int count = Math.Min(1 << LengthExtraBits[index], table.Length - LengthBases[index]); + for (int extra = 0; extra < count; extra++) + { + table[LengthBases[index] + extra] = AppendExtra(symbol, extra, LengthExtraBits[index]); + } + } + return table; + } + + private static uint[] BuildDistances() + { + var table = new uint[32_769]; + for (int index = 0; index < DistanceBases.Length; index++) + { + uint code = Pack(Reverse((uint)index, 5), 5); + for (int extra = 0; extra < 1 << DistanceExtraBits[index]; extra++) + { + table[DistanceBases[index] + extra] = AppendExtra(code, extra, DistanceExtraBits[index]); + } + } + return table; + } + + private static uint Symbol(int symbol) + { + if (symbol < 144) + { + return Pack(Reverse((uint)(0x30 + symbol), 8), 8); + } + if (symbol < 256) + { + return Pack(Reverse((uint)(0x190 + symbol - 144), 9), 9); + } + if (symbol < 280) + { + return Pack(Reverse((uint)(symbol - 256), 7), 7); + } + return Pack(Reverse((uint)(0xC0 + symbol - 280), 8), 8); + } + + private static uint AppendExtra(uint packed, int extra, int extraBits) + { + int length = (int)(packed & LengthMask); + uint code = (packed >> LengthShift) | ((uint)extra << length); + return Pack(code, length + extraBits); + } + + private static uint Pack(uint code, int length) + { + return (code << LengthShift) | (uint)length; + } + + private static uint Reverse(uint code, int length) + { + uint reversed = 0; + for (int bit = 0; bit < length; bit++) + { + reversed = (reversed << 1) | ((code >> bit) & 1); + } + return reversed; + } + } +} diff --git a/src/ExcelReader.Core/Writer/Zip/SegmentCompressor.cs b/src/ExcelReader.Core/Writer/Zip/SegmentCompressor.cs new file mode 100644 index 00000000..af4780af --- /dev/null +++ b/src/ExcelReader.Core/Writer/Zip/SegmentCompressor.cs @@ -0,0 +1,51 @@ +using System.IO.Compression; +using ExcelReader.Core.Writer.Internal; + +namespace ExcelReader.Core.Writer.Zip +{ + /// + /// Compresses one run of bytes into a deflate segment: data that ends on a byte boundary and has + /// no final block, so segments laid end to end, followed by , are one + /// valid deflate stream. A segment never refers to bytes outside itself. + /// + internal static class SegmentCompressor + { + /// An empty fixed-Huffman block marked final: it ends a stream of segments. + internal static ReadOnlySpan FinalBlock + { + get + { + return [0x03, 0x00]; + } + } + + internal static void Compress(ReadOnlySpan input, CompressionLevel level, BiffBuffer output) + { + if (level == CompressionLevel.NoCompression) + { + output.Write(input); + return; + } + // DeflateEncoder writes and compares native-order words, which is only deflate's bit order on little-endian. + if (level == CompressionLevel.Fastest && BitConverter.IsLittleEndian) + { + using var encoder = new DeflateEncoder(); + encoder.Write(input, output); + encoder.FlushSync(output); + return; + } + CompressWithZlib(input, level, output); + } + + // DeflateStream.Flush is zlib's sync flush, which ends on a byte boundary; the final block that + // Dispose writes afterwards is dropped. + private static void CompressWithZlib(ReadOnlySpan input, CompressionLevel level, BiffBuffer output) + { + var sink = new BufferSink(output); + using var zlib = new DeflateStream(sink, level); + zlib.Write(input); + zlib.Flush(); + sink.Discard = true; + } + } +} diff --git a/src/ExcelReader.Core/Writer/Zip/ZipEntryRecord.cs b/src/ExcelReader.Core/Writer/Zip/ZipEntryRecord.cs new file mode 100644 index 00000000..5a4ab602 --- /dev/null +++ b/src/ExcelReader.Core/Writer/Zip/ZipEntryRecord.cs @@ -0,0 +1,71 @@ +using System.IO.Compression; +using System.Text; + +namespace ExcelReader.Core.Writer.Zip +{ + internal sealed class ZipEntryRecord + { + internal ZipEntryRecord(string name, CompressionLevel level, long offset, DateTime modified) + { + Name = Encoding.UTF8.GetBytes(name); + Level = level; + Offset = offset; + (Time, Date) = ZipRecords.DosDateTime(modified); + Flags = ZipRecords.LevelFlags(level); + if (!Ascii.IsValid(name)) + { + Flags |= ZipRecords.FlagUtf8Name; + } + Method = ZipRecords.MethodDeflate; + if (level == CompressionLevel.NoCompression) + { + Method = ZipRecords.MethodStored; + } + } + + internal byte[] Name { get; } + + internal CompressionLevel Level { get; } + + /// Absolute position of the local header in the output stream. + internal long Offset { get; } + + internal ushort Time { get; } + + internal ushort Date { get; } + + internal ushort Flags { get; set; } + + internal ushort Method { get; set; } + + internal uint Crc { get; set; } + + internal long CompressedSize { get; set; } + + internal long UncompressedSize { get; set; } + + internal int LocalHeaderLength + { + get + { + return ZipRecords.LocalHeaderLength + Name.Length; + } + } + + internal bool NeedsZip64Sizes + { + get + { + return CompressedSize >= uint.MaxValue || UncompressedSize >= uint.MaxValue; + } + } + + internal bool NeedsZip64Offset + { + get + { + return Offset >= uint.MaxValue; + } + } + } +} diff --git a/src/ExcelReader.Core/Writer/Zip/ZipEntryStream.cs b/src/ExcelReader.Core/Writer/Zip/ZipEntryStream.cs new file mode 100644 index 00000000..41d01fbf --- /dev/null +++ b/src/ExcelReader.Core/Writer/Zip/ZipEntryStream.cs @@ -0,0 +1,496 @@ +using System.Diagnostics.CodeAnalysis; +using System.IO.Compression; +using ExcelReader.Core.Writer.Internal; + +namespace ExcelReader.Core.Writer.Zip +{ + /// + /// The write-only stream of one entry. It checksums and compresses + /// what it is given; closing it completes the local header in place, by seeking back, or with a + /// data descriptor. + /// + internal sealed class ZipEntryStream : Stream + { + private const int FlushThreshold = 64 * 1024; + private const int MaxWriteSlice = 1 << 20; + + private readonly ZipPackageWriter _owner; + private readonly ZipEntryRecord _record; + [SuppressMessage("Usage", "CA2213:Disposable fields should be disposed", + Justification = "Released by Release(), which every close path and Abandon call.")] + private readonly BiffBuffer _pending = new(FlushThreshold * 2); + [SuppressMessage("Usage", "CA2213:Disposable fields should be disposed", + Justification = "Released by Release(), which every close path and Abandon call.")] + private DeflateEncoder? _encoder; + [SuppressMessage("Usage", "CA2213:Disposable fields should be disposed", + Justification = "Finished by FinishCompression and released by Release().")] + private DeflateStream? _zlib; + [SuppressMessage("Usage", "CA2213:Disposable fields should be disposed", + Justification = "Owns no resources; the DeflateStream over it disposes it.")] + private BufferSink? _zlibSink; + private bool _segmented; + private uint _crc; + private long _uncompressed; + private long _compressed; + private bool _headerFlushed; + private bool _closed; + + internal ZipEntryStream(ZipPackageWriter owner, ZipEntryRecord record) + { + _owner = owner; + _record = record; + if (!owner.CanSeek) + { + record.Flags |= ZipRecords.FlagDataDescriptor; + } + ZipRecords.WriteLocalHeader(_pending, record); + // DeflateEncoder writes and compares native-order words, which is only deflate's bit order on little-endian. + if (record.Level == CompressionLevel.Fastest && BitConverter.IsLittleEndian) + { + _encoder = new DeflateEncoder(); + } + else if (record.Method == ZipRecords.MethodDeflate) + { + _zlibSink = new BufferSink(_pending); + _zlib = new DeflateStream(_zlibSink, record.Level); + } + } + + private enum Completion + { + Done, + PatchHeader, + PatchZip64, + } + + public override bool CanRead + { + get + { + return false; + } + } + + public override bool CanSeek + { + get + { + return false; + } + } + + public override bool CanWrite + { + get + { + return !_closed; + } + } + + public override long Length + { + get + { + throw new NotSupportedException(); + } + } + + public override long Position + { + get + { + throw new NotSupportedException(); + } + set + { + throw new NotSupportedException(); + } + } + + public override void Flush() + { + if (_closed) + { + return; + } + try + { + _owner.FlushOutput(); + } + catch + { + Fail(); + throw; + } + } + + public override async Task FlushAsync(CancellationToken cancellationToken) + { + if (_closed) + { + return; + } + try + { + await _owner.FlushOutputAsync(cancellationToken).ConfigureAwait(false); + } + catch + { + Fail(); + throw; + } + } + + public override int Read(byte[] buffer, int offset, int count) + { + throw new NotSupportedException(); + } + + public override long Seek(long offset, SeekOrigin origin) + { + throw new NotSupportedException(); + } + + public override void SetLength(long value) + { + throw new NotSupportedException(); + } + + public override void Write(byte[] buffer, int offset, int count) + { + Write(buffer.AsSpan(offset, count)); + } + + public override void Write(ReadOnlySpan buffer) + { + ObjectDisposedException.ThrowIf(_closed, this); + try + { + do + { + int take = Math.Min(buffer.Length, MaxWriteSlice); + Accept(buffer[..take]); + buffer = buffer[take..]; + if (_pending.Length >= FlushThreshold) + { + FlushPending(); + } + } + while (!buffer.IsEmpty); + } + catch + { + Fail(); + throw; + } + } + + public override Task WriteAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken) + { + return WriteAsync(buffer.AsMemory(offset, count), cancellationToken).AsTask(); + } + + public override async ValueTask WriteAsync(ReadOnlyMemory buffer, CancellationToken cancellationToken = default) + { + ObjectDisposedException.ThrowIf(_closed, this); + try + { + do + { + int take = Math.Min(buffer.Length, MaxWriteSlice); + Accept(buffer.Span[..take]); + buffer = buffer[take..]; + if (_pending.Length >= FlushThreshold) + { + await FlushPendingAsync(cancellationToken).ConfigureAwait(false); + } + } + while (!buffer.IsEmpty); + } + catch + { + Fail(); + throw; + } + } + + /// + /// Appends data that already compressed at this entry's level, + /// with the CRC and length of the bytes it was made from. + /// + internal void AppendSegment(ReadOnlySpan segment, uint crc, long uncompressedLength) + { + ObjectDisposedException.ThrowIf(_closed, this); + try + { + BeginSegments(); + MergeSegment(crc, uncompressedLength); + _pending.Write(segment); + if (_pending.Length >= FlushThreshold) + { + FlushPending(); + } + } + catch + { + Fail(); + throw; + } + } + + /// + internal async ValueTask AppendSegmentAsync(ReadOnlyMemory segment, uint crc, long uncompressedLength, CancellationToken cancellationToken) + { + ObjectDisposedException.ThrowIf(_closed, this); + try + { + BeginSegments(); + MergeSegment(crc, uncompressedLength); + _pending.Write(segment.Span); + if (_pending.Length >= FlushThreshold) + { + await FlushPendingAsync(cancellationToken).ConfigureAwait(false); + } + } + catch + { + Fail(); + throw; + } + } + + // From here on every piece of the entry is self-contained: a compressor kept across a segment + // could refer back to bytes the reader sees before that segment. + private void BeginSegments() + { + if (_segmented) + { + return; + } + _segmented = true; + if (_encoder is not null) + { + _encoder.FlushSync(_pending); + _encoder.Dispose(); + _encoder = null; + return; + } + if (_zlib is not null) + { + _zlib.Flush(); + _zlibSink!.Discard = true; + _zlib.Dispose(); + _zlib = null; + } + } + + private void MergeSegment(uint crc, long uncompressedLength) + { + _crc = Crc32.Combine(_crc, crc, uncompressedLength); + _uncompressed += uncompressedLength; + } + + internal void Abandon() + { + if (_closed) + { + return; + } + _closed = true; + Release(); + } + + // A failed write leaves the output position unknown, so the entry and the archive are dead. + private void Fail() + { + _closed = true; + _owner.EntryFailed(); + Release(); + } + + protected override void Dispose(bool disposing) + { + if (disposing && !_closed) + { + _closed = true; + try + { + Completion completion = PrepareCompletion(); + FlushPending(); + CompleteAfterFlush(completion); + _owner.EntryClosed(_record); + } + catch + { + _owner.EntryFailed(); + throw; + } + finally + { + Release(); + } + } + base.Dispose(disposing); + } + + public override async ValueTask DisposeAsync() + { + if (!_closed) + { + _closed = true; + try + { + Completion completion = PrepareCompletion(); + await FlushPendingAsync(CancellationToken.None).ConfigureAwait(false); + await CompleteAfterFlushAsync(completion).ConfigureAwait(false); + _owner.EntryClosed(_record); + } + catch + { + _owner.EntryFailed(); + throw; + } + finally + { + Release(); + } + } + await base.DisposeAsync().ConfigureAwait(false); + } + + private void Accept(ReadOnlySpan buffer) + { + _crc = Crc32.Append(_crc, buffer); + _uncompressed += buffer.Length; + if (_segmented) + { + SegmentCompressor.Compress(buffer, _record.Level, _pending); + return; + } + if (_encoder is not null) + { + _encoder.Write(buffer, _pending); + return; + } + if (_zlib is not null) + { + _zlib.Write(buffer); + return; + } + _pending.Write(buffer); + } + + private Completion PrepareCompletion() + { + if (_uncompressed == 0 && !_headerFlushed) + { + MarkEmpty(); + return Completion.Done; + } + FinishCompression(); + _record.Crc = _crc; + _record.UncompressedSize = _uncompressed; + _record.CompressedSize = _compressed + _pending.Length - PendingHeaderLength(); + if ((_record.Flags & ZipRecords.FlagDataDescriptor) != 0) + { + ZipRecords.WriteDataDescriptor(_pending, _record); + return Completion.Done; + } + if (!_headerFlushed) + { + ZipRecords.CrcAndSizes(_record).CopyTo(_pending.Slice(ZipRecords.LocalCrcOffset, 12)); + return Completion.Done; + } + if (_record.NeedsZip64Sizes) + { + _record.Flags |= ZipRecords.FlagDataDescriptor; + return Completion.PatchZip64; + } + return Completion.PatchHeader; + } + + private void CompleteAfterFlush(Completion completion) + { + if (completion == Completion.PatchHeader) + { + _owner.Patch(_record.Offset + ZipRecords.LocalCrcOffset, ZipRecords.CrcAndSizes(_record)); + return; + } + if (completion == Completion.PatchZip64) + { + _owner.Patch(_record.Offset + ZipRecords.LocalVersionOffset, ZipRecords.VersionAndFlags(_record)); + ZipRecords.WriteDataDescriptor(_pending, _record); + FlushPending(); + } + } + + private async ValueTask CompleteAfterFlushAsync(Completion completion) + { + if (completion == Completion.PatchHeader) + { + await _owner.PatchAsync(_record.Offset + ZipRecords.LocalCrcOffset, ZipRecords.CrcAndSizes(_record), CancellationToken.None).ConfigureAwait(false); + return; + } + if (completion == Completion.PatchZip64) + { + await _owner.PatchAsync(_record.Offset + ZipRecords.LocalVersionOffset, ZipRecords.VersionAndFlags(_record), CancellationToken.None).ConfigureAwait(false); + ZipRecords.WriteDataDescriptor(_pending, _record); + await FlushPendingAsync(CancellationToken.None).ConfigureAwait(false); + } + } + + // ZipArchive writes an entry that received no data as stored, with no data descriptor. + private void MarkEmpty() + { + _record.Method = ZipRecords.MethodStored; + _record.Flags = (ushort)(_record.Flags & ~ZipRecords.FlagDataDescriptor); + _pending.Reset(); + ZipRecords.WriteLocalHeader(_pending, _record); + } + + private void FinishCompression() + { + if (_segmented) + { + if (_record.Method == ZipRecords.MethodDeflate) + { + _pending.Write(SegmentCompressor.FinalBlock); + } + return; + } + _encoder?.Finish(_pending); + _zlib?.Dispose(); + } + + private int PendingHeaderLength() + { + return _headerFlushed ? 0 : _record.LocalHeaderLength; + } + + private void FlushPending() + { + _owner.WriteRaw(_pending.Span); + CountFlushed(); + } + + private async ValueTask FlushPendingAsync(CancellationToken ct) + { + await _owner.WriteRawAsync(_pending.Memory, ct).ConfigureAwait(false); + CountFlushed(); + } + + private void CountFlushed() + { + _compressed += _pending.Length - PendingHeaderLength(); + _headerFlushed = true; + _pending.Reset(); + } + + private void Release() + { + _encoder?.Dispose(); + FailureCleanup.Dispose(_zlib); + _pending.Dispose(); + } + } +} diff --git a/src/ExcelReader.Core/Writer/Zip/ZipPackageWriter.cs b/src/ExcelReader.Core/Writer/Zip/ZipPackageWriter.cs new file mode 100644 index 00000000..e3c51a59 --- /dev/null +++ b/src/ExcelReader.Core/Writer/Zip/ZipPackageWriter.cs @@ -0,0 +1,203 @@ +using System.Diagnostics.CodeAnalysis; +using System.IO.Compression; +using ExcelReader.Core.Writer.Internal; + +namespace ExcelReader.Core.Writer.Zip +{ + /// + /// Writes a ZIP archive one entry at a time to a stream it does not own, in the layout + /// ZipArchive uses in create mode. Offsets are absolute positions in that stream. + /// + [SuppressMessage("Design", "CA1001:Types that own disposable fields should be disposable", + Justification = "The open entry belongs to the caller that opened it and is completed by its Dispose; Abandon releases it without completing it.")] + internal sealed class ZipPackageWriter + { + private readonly Stream _output; + private readonly long _origin; + private readonly List _entries = []; + private ZipEntryStream? _open; + private long _written; + private bool _closed; + private bool _broken; + + internal ZipPackageWriter(Stream output) + { + _output = output; + CanSeek = output.CanSeek; + if (CanSeek) + { + _origin = output.Position; + } + } + + internal bool CanSeek { get; } + + internal long Position + { + get + { + return _origin + _written; + } + } + + internal ZipEntryStream OpenEntry(string name, CompressionLevel level) + { + ObjectDisposedException.ThrowIf(_closed, this); + if (!Enum.IsDefined(level)) + { + throw new ArgumentOutOfRangeException(nameof(level), level, "Unknown compression level."); + } + if (_open is not null) + { + throw new InvalidOperationException("The previous ZIP entry must be closed before another is opened."); + } + if (_broken) + { + throw new InvalidOperationException("A ZIP entry failed, so the archive cannot take more entries."); + } + _open = new ZipEntryStream(this, new ZipEntryRecord(name, level, Position, DateTime.Now)); + return _open; + } + + internal void Finish() + { + using BiffBuffer directory = BuildCentralDirectory(); + WriteRaw(directory.Span); + } + + internal async ValueTask FinishAsync(CancellationToken ct = default) + { + using BiffBuffer directory = BuildCentralDirectory(); + await WriteRawAsync(directory.Memory, ct).ConfigureAwait(false); + } + + [SuppressMessage("Design", "CA1031:Do not catch general exception types", + Justification = "Called while an earlier failure propagates; the directory is best effort and must not hide that failure.")] + internal void Abandon() + { + if (!TryClose()) + { + return; + } + try + { + using BiffBuffer directory = PrepareAbandonedDirectory(); + WriteRaw(directory.Span); + } + catch (Exception) + { + } + } + + [SuppressMessage("Design", "CA1031:Do not catch general exception types", + Justification = "Called while an earlier failure propagates; the directory is best effort and must not hide that failure.")] + internal async ValueTask AbandonAsync() + { + if (!TryClose()) + { + return; + } + try + { + using BiffBuffer directory = PrepareAbandonedDirectory(); + await WriteRawAsync(directory.Memory, CancellationToken.None).ConfigureAwait(false); + } + catch (Exception) + { + } + } + + internal void FlushOutput() + { + _output.Flush(); + } + + internal Task FlushOutputAsync(CancellationToken ct) + { + return _output.FlushAsync(ct); + } + + internal void WriteRaw(ReadOnlySpan bytes) + { + _output.Write(bytes); + _written += bytes.Length; + } + + internal async ValueTask WriteRawAsync(ReadOnlyMemory bytes, CancellationToken ct) + { + await _output.WriteAsync(bytes, ct).ConfigureAwait(false); + _written += bytes.Length; + } + + internal void Patch(long position, ReadOnlySpan bytes) + { + _output.Seek(position, SeekOrigin.Begin); + _output.Write(bytes); + _output.Seek(Position, SeekOrigin.Begin); + } + + internal async ValueTask PatchAsync(long position, ReadOnlyMemory bytes, CancellationToken ct) + { + _output.Seek(position, SeekOrigin.Begin); + await _output.WriteAsync(bytes, ct).ConfigureAwait(false); + _output.Seek(Position, SeekOrigin.Begin); + } + + internal void EntryClosed(ZipEntryRecord record) + { + _entries.Add(record); + _open = null; + } + + internal void EntryFailed() + { + _open = null; + _broken = true; + } + + private BiffBuffer BuildCentralDirectory() + { + ObjectDisposedException.ThrowIf(_closed, this); + if (_open is not null || _broken) + { + throw new InvalidOperationException("Every ZIP entry must be closed successfully before the archive is finished."); + } + _closed = true; + return BuildDirectory(); + } + + private bool TryClose() + { + if (_closed) + { + return false; + } + _closed = true; + return true; + } + + private BiffBuffer PrepareAbandonedDirectory() + { + ZipEntryStream? open = _open; + _open = null; + open?.Abandon(); + if (CanSeek) + { + _output.Seek(Position, SeekOrigin.Begin); + } + return BuildDirectory(); + } + + private BiffBuffer BuildDirectory() + { + long directoryOffset = Position; + var directory = new BiffBuffer(256 + (_entries.Count * 128)); + foreach (ZipEntryRecord entry in _entries) + { + ZipRecords.WriteCentralHeader(directory, entry); + } + ZipRecords.WriteEnd(directory, _entries.Count, directoryOffset, directory.Length); + return directory; + } + } +} diff --git a/src/ExcelReader.Core/Writer/Zip/ZipRecords.cs b/src/ExcelReader.Core/Writer/Zip/ZipRecords.cs new file mode 100644 index 00000000..2d244300 --- /dev/null +++ b/src/ExcelReader.Core/Writer/Zip/ZipRecords.cs @@ -0,0 +1,208 @@ +using System.Buffers.Binary; +using System.IO.Compression; +using ExcelReader.Core.Writer.Internal; + +namespace ExcelReader.Core.Writer.Zip +{ + /// + /// The ZIP records of PKWARE's APPNOTE.TXT section 4.3, with the field values ZipArchive + /// writes in create mode, and Zip64 only where a 32-bit field overflows. + /// + internal static class ZipRecords + { + internal const int LocalHeaderLength = 30; + internal const int LocalVersionOffset = 4; + internal const int LocalCrcOffset = 14; + internal const ushort FlagDataDescriptor = 0x0008; + internal const ushort FlagUtf8Name = 0x0800; + internal const ushort MethodStored = 0; + internal const ushort MethodDeflate = 8; + internal const ushort VersionDefault = 20; + internal const ushort VersionZip64 = 45; + + private const uint LocalHeaderSignature = 0x0403_4B50; + private const uint DataDescriptorSignature = 0x0807_4B50; + private const uint CentralHeaderSignature = 0x0201_4B50; + private const uint Zip64EndSignature = 0x0606_4B50; + private const uint Zip64LocatorSignature = 0x0706_4B50; + private const uint EndSignature = 0x0605_4B50; + private const ushort VersionMadeBy = 0x0014; + private const ushort Zip64ExtraId = 0x0001; + private const ulong Zip64EndRecordSize = 44; + private const int MaxClassicCount = 0xFFFF; + + internal static ushort LevelFlags(CompressionLevel level) + { + switch (level) + { + case CompressionLevel.Fastest: + return 0x0006; + case CompressionLevel.SmallestSize: + return 0x0002; + default: + return 0; + } + } + + internal static (ushort Time, ushort Date) DosDateTime(DateTime value) + { + DateTime clamped = value; + if (clamped.Year < 1980) + { + clamped = new DateTime(1980, 1, 1, 0, 0, 0, DateTimeKind.Local); + } + if (clamped.Year > 2107) + { + clamped = new DateTime(2107, 12, 31, 23, 59, 58, DateTimeKind.Local); + } + ushort time = (ushort)((clamped.Hour << 11) | (clamped.Minute << 5) | (clamped.Second / 2)); + ushort date = (ushort)(((clamped.Year - 1980) << 9) | (clamped.Month << 5) | clamped.Day); + return (time, date); + } + + internal static void WriteLocalHeader(BiffBuffer buffer, ZipEntryRecord entry) + { + buffer.WriteU32(LocalHeaderSignature); + buffer.WriteU16(VersionDefault); + buffer.WriteU16(entry.Flags); + buffer.WriteU16(entry.Method); + buffer.WriteU16(entry.Time); + buffer.WriteU16(entry.Date); + buffer.WriteU32(0); + buffer.WriteU32(0); + buffer.WriteU32(0); + buffer.WriteU16(entry.Name.Length); + buffer.WriteU16(0); + buffer.Write(entry.Name); + } + + /// The 12 bytes at : CRC-32 and 32-bit sizes. + internal static byte[] CrcAndSizes(ZipEntryRecord entry) + { + byte[] bytes = new byte[12]; + BinaryPrimitives.WriteUInt32LittleEndian(bytes, entry.Crc); + BinaryPrimitives.WriteUInt32LittleEndian(bytes.AsSpan(4), (uint)entry.CompressedSize); + BinaryPrimitives.WriteUInt32LittleEndian(bytes.AsSpan(8), (uint)entry.UncompressedSize); + return bytes; + } + + /// The 4 bytes at : version needed and flags. + internal static byte[] VersionAndFlags(ZipEntryRecord entry) + { + byte[] bytes = new byte[4]; + BinaryPrimitives.WriteUInt16LittleEndian(bytes, VersionZip64); + BinaryPrimitives.WriteUInt16LittleEndian(bytes.AsSpan(2), entry.Flags); + return bytes; + } + + internal static void WriteDataDescriptor(BiffBuffer buffer, ZipEntryRecord entry) + { + buffer.WriteU32(DataDescriptorSignature); + buffer.WriteU32(entry.Crc); + if (entry.NeedsZip64Sizes) + { + WriteU64(buffer, (ulong)entry.CompressedSize); + WriteU64(buffer, (ulong)entry.UncompressedSize); + return; + } + buffer.WriteU32((uint)entry.CompressedSize); + buffer.WriteU32((uint)entry.UncompressedSize); + } + + internal static void WriteCentralHeader(BiffBuffer buffer, ZipEntryRecord entry) + { + bool zip64Sizes = entry.NeedsZip64Sizes; + bool zip64Offset = entry.NeedsZip64Offset; + int extraFields = (zip64Sizes ? 2 : 0) + (zip64Offset ? 1 : 0); + int extraLength = 0; + if (extraFields != 0) + { + extraLength = 4 + (8 * extraFields); + } + buffer.WriteU32(CentralHeaderSignature); + buffer.WriteU16(VersionMadeBy); + buffer.WriteU16(extraFields == 0 ? VersionDefault : VersionZip64); + buffer.WriteU16(entry.Flags); + buffer.WriteU16(entry.Method); + buffer.WriteU16(entry.Time); + buffer.WriteU16(entry.Date); + buffer.WriteU32(entry.Crc); + buffer.WriteU32(Field(entry.CompressedSize, zip64Sizes)); + buffer.WriteU32(Field(entry.UncompressedSize, zip64Sizes)); + buffer.WriteU16(entry.Name.Length); + buffer.WriteU16(extraLength); + buffer.WriteU16(0); + buffer.WriteU16(0); + buffer.WriteU16(0); + buffer.WriteU32(0); + buffer.WriteU32(Field(entry.Offset, zip64Offset)); + buffer.Write(entry.Name); + if (extraFields == 0) + { + return; + } + buffer.WriteU16(Zip64ExtraId); + buffer.WriteU16(8 * extraFields); + if (zip64Sizes) + { + WriteU64(buffer, (ulong)entry.UncompressedSize); + WriteU64(buffer, (ulong)entry.CompressedSize); + } + if (zip64Offset) + { + WriteU64(buffer, (ulong)entry.Offset); + } + } + + internal static void WriteEnd(BiffBuffer buffer, int count, long directoryOffset, long directoryLength) + { + bool zip64 = count >= MaxClassicCount || directoryOffset >= uint.MaxValue || directoryLength >= uint.MaxValue; + if (zip64) + { + WriteZip64End(buffer, count, directoryOffset, directoryLength); + } + buffer.WriteU32(EndSignature); + buffer.WriteU16(0); + buffer.WriteU16(0); + buffer.WriteU16(Math.Min(count, MaxClassicCount)); + buffer.WriteU16(Math.Min(count, MaxClassicCount)); + buffer.WriteU32(Field(directoryLength, directoryLength >= uint.MaxValue)); + buffer.WriteU32(Field(directoryOffset, directoryOffset >= uint.MaxValue)); + buffer.WriteU16(0); + } + + private static void WriteZip64End(BiffBuffer buffer, int count, long directoryOffset, long directoryLength) + { + long zip64EndOffset = directoryOffset + directoryLength; + buffer.WriteU32(Zip64EndSignature); + WriteU64(buffer, Zip64EndRecordSize); + buffer.WriteU16(VersionZip64); + buffer.WriteU16(VersionZip64); + buffer.WriteU32(0); + buffer.WriteU32(0); + WriteU64(buffer, (ulong)count); + WriteU64(buffer, (ulong)count); + WriteU64(buffer, (ulong)directoryLength); + WriteU64(buffer, (ulong)directoryOffset); + buffer.WriteU32(Zip64LocatorSignature); + buffer.WriteU32(0); + WriteU64(buffer, (ulong)zip64EndOffset); + buffer.WriteU32(1); + } + + private static uint Field(long value, bool overflows) + { + if (overflows) + { + return uint.MaxValue; + } + return (uint)value; + } + + private static void WriteU64(BiffBuffer buffer, ulong value) + { + BinaryPrimitives.WriteUInt64LittleEndian(buffer.GetSpan(sizeof(ulong)), value); + buffer.Advance(sizeof(ulong)); + } + } +} diff --git a/src/ExcelReader.Native/NativeOpenOptions.cs b/src/ExcelReader.Native/NativeOpenOptions.cs index 5728fe5a..70e5a36e 100644 --- a/src/ExcelReader.Native/NativeOpenOptions.cs +++ b/src/ExcelReader.Native/NativeOpenOptions.cs @@ -74,6 +74,8 @@ internal struct NativeOpenOptionsRaw public long SourceCacheBytes; public long MaxBufferedBytes; public int SourceBlockSize; + // Occupies what was tail padding, so the struct's size did not change when it was added. + public int VerifyZipChecksums; internal const int V6Size = 88; @@ -114,6 +116,7 @@ internal readonly struct NativeOpenOptions internal int? MaxZipEntries { get; init; } internal bool? PrefetchDecompression { get; init; } internal bool? InternStrings { get; init; } + internal bool? VerifyZipChecksums { get; init; } internal string? Password { get; init; } @@ -174,6 +177,10 @@ internal ExcelReaderOptions ToExcelReaderOptions() { options = options with { InternStrings = internStrings }; } + if (VerifyZipChecksums is bool verifyChecksums) + { + options = options with { VerifyZipChecksums = verifyChecksums }; + } if (Password is not null) { options = options with { Password = Password }; @@ -209,7 +216,8 @@ internal static bool TryDecode(NativeOpenOptionsRaw raw, out NativeOpenOptions o || !NativeOptionDecode.TryState(raw.CsvDetectBom, OptionsName, "csv_detect_bom", out bool? detectBom, out error) || !NativeOptionDecode.TryState(raw.CsvInternStrings, OptionsName, "csv_intern_strings", out bool? csvInternStrings, out error) || !NativeOptionDecode.TryState(raw.PrefetchDecompression, OptionsName, "prefetch_decompression", out bool? prefetch, out error) - || !NativeOptionDecode.TryState(raw.InternStrings, OptionsName, "intern_strings", out bool? internStrings, out error)) + || !NativeOptionDecode.TryState(raw.InternStrings, OptionsName, "intern_strings", out bool? internStrings, out error) + || !NativeOptionDecode.TryState(raw.VerifyZipChecksums, OptionsName, "verify_zip_checksums", out bool? verifyChecksums, out error)) { return false; } @@ -247,6 +255,7 @@ internal static bool TryDecode(NativeOpenOptionsRaw raw, out NativeOpenOptions o MaxZipEntries = maxZipEntries, PrefetchDecompression = prefetch, InternStrings = internStrings, + VerifyZipChecksums = verifyChecksums, Password = password, SourceBlockSize = raw.SourceBlockSize == 0 ? null : raw.SourceBlockSize, SourceCacheBytes = sourceCacheBytes, diff --git a/src/ExcelReader.Native/Typed/TypedApi.cs b/src/ExcelReader.Native/Typed/TypedApi.cs index a7766d07..b2b0fb29 100644 --- a/src/ExcelReader.Native/Typed/TypedApi.cs +++ b/src/ExcelReader.Native/Typed/TypedApi.cs @@ -467,6 +467,11 @@ private bool AppendTimestamp(in Cell cell, bool isDate1904) { bool ok = ColumnParserFactory.ReadDateTime(in cell, new ExcelRowContext { IsDate1904 = isDate1904 }, out DateTime value) || ColumnParserFactory.ReadTextDateTime(in cell, new ExcelRowContext { IsDate1904 = isDate1904 }, out value); + // Text with a "Z" or an offset parses to the machine's local time; the wire value is UTC. + if (value.Kind == DateTimeKind.Local) + { + value = value.ToUniversalTime(); + } return Append(_longs, ok, (value - DateTime.UnixEpoch).Ticks / 10); } diff --git a/src/ExcelReader.Native/include/excelreader.h b/src/ExcelReader.Native/include/excelreader.h index 12cf422c..b17f5a93 100644 --- a/src/ExcelReader.Native/include/excelreader.h +++ b/src/ExcelReader.Native/include/excelreader.h @@ -83,6 +83,8 @@ typedef struct xl_open_options { int64_t source_cache_bytes; int64_t max_buffered_bytes; int32_t source_block_size; + /* XL_OPT_*: check each XLSX/XLSB ZIP entry against its CRC-32. On by default. */ + int32_t verify_zip_checksums; } xl_open_options; int32_t xl_open_file(const uint8_t* path, int32_t path_len, int32_t format, diff --git a/src/ExcelReader.Native/include/excelreader.hpp b/src/ExcelReader.Native/include/excelreader.hpp index 29b713c6..67fb8abe 100644 --- a/src/ExcelReader.Native/include/excelreader.hpp +++ b/src/ExcelReader.Native/include/excelreader.hpp @@ -160,6 +160,7 @@ namespace xl int64_t source_cache_bytes = 0; ///< Block cache of Workbook::open_source. Default 64 MiB. int64_t max_buffered_bytes = 0; ///< Most bytes Workbook::open_stream reads before an XLSX, XLSB or XLS opens. int32_t source_block_size = 0; ///< Block size of Workbook::open_source's cache. Default 4 MiB; negative turns the cache off. + int32_t verify_zip_checksums = XL_OPT_DEFAULT; ///< Check each XLSX/XLSB ZIP entry against its CRC-32. On by default; a mismatch fails the read. std::string password_{}; ///< Set through password(). @@ -196,6 +197,7 @@ namespace xl opts.source_cache_bytes = source_cache_bytes; opts.max_buffered_bytes = max_buffered_bytes; opts.source_block_size = source_block_size; + opts.verify_zip_checksums = verify_zip_checksums; opts.password = password_.empty() ? nullptr : reinterpret_cast(password_.data()); opts.password_len = static_cast(password_.size()); return opts; diff --git a/src/ExcelReader.Native/pgo/excelreader.mibc b/src/ExcelReader.Native/pgo/excelreader.mibc index 4e912f64..d9c38b97 100644 Binary files a/src/ExcelReader.Native/pgo/excelreader.mibc and b/src/ExcelReader.Native/pgo/excelreader.mibc differ diff --git a/tests/ExcelReader.Benchmarks/Read/XlsxSharedStringHotPathBenchmark.cs b/tests/ExcelReader.Benchmarks/Read/XlsxSharedStringHotPathBenchmark.cs index e7354893..4833186b 100644 --- a/tests/ExcelReader.Benchmarks/Read/XlsxSharedStringHotPathBenchmark.cs +++ b/tests/ExcelReader.Benchmarks/Read/XlsxSharedStringHotPathBenchmark.cs @@ -68,19 +68,34 @@ private static bool Prologue(byte[] workbook) } [Benchmark(Baseline = true)] - public long EndToEnd() => ReadAll(_xlsx, null); + public long EndToEnd() + { + return ReadAll(_xlsx, null); + } [Benchmark] - public long EndToEnd_Prefetch() => ReadAll(_xlsx, _prefetchOptions); + public long EndToEnd_Prefetch() + { + return ReadAll(_xlsx, _prefetchOptions); + } [Benchmark] - public bool SharedStringPrologue() => Prologue(_xlsx); + public bool SharedStringPrologue() + { + return Prologue(_xlsx); + } [Benchmark] - public long Stored_EndToEnd() => ReadAll(_stored, null); + public long Stored_EndToEnd() + { + return ReadAll(_stored, null); + } [Benchmark] - public bool Stored_SharedStringPrologue() => Prologue(_stored); + public bool Stored_SharedStringPrologue() + { + return Prologue(_stored); + } [Benchmark] public long IndexParse_Utf8Parser() diff --git a/tests/ExcelReader.Fuzz/Harnesses.cs b/tests/ExcelReader.Fuzz/Harnesses.cs index 5f97c322..619a55c2 100644 --- a/tests/ExcelReader.Fuzz/Harnesses.cs +++ b/tests/ExcelReader.Fuzz/Harnesses.cs @@ -10,6 +10,8 @@ using ExcelReader.Core.Reader.Xlsb; using ExcelReader.Core.Reader.Xlsx; using ExcelReader.Core.Reader.Zip.Inflate; +using ExcelReader.Core.Writer.Internal; +using ExcelReader.Core.Writer.Zip; namespace ExcelReader.Fuzz { @@ -19,10 +21,12 @@ namespace ExcelReader.Fuzz /// internal static class Harnesses { + // Checksums stay off: a mutated entry would otherwise fail its CRC-32 before reaching a parser. private static readonly ExcelReaderOptions Limits = new() { MaxCellBytes = 1 << 20, MaxSharedStringBytes = 1 << 22, + VerifyZipChecksums = false, }; private static readonly CsvReaderOptions CsvLimits = new() @@ -35,6 +39,7 @@ internal static class Harnesses MaxCellBytes = 1 << 20, MaxSharedStringBytes = 1 << 22, Password = "hunter2", + VerifyZipChecksums = false, }; internal static void Xlsx(ReadOnlySpan data) @@ -249,6 +254,77 @@ internal static void InflateDivergenceSelfCheck() throw new InvalidOperationException("The fuzz oracle swallows an inflate divergence."); } + internal static void Deflate(ReadOnlySpan data) + { + if (data.IsEmpty) + { + return; + } + int chunk = 1 + (data[0] * 4099 % 70_000); + ReadOnlySpan payload = data[1..]; + if (payload.Length > MaxInflateOutput) + { + payload = payload[..MaxInflateOutput]; + } + byte[] compressed = DeflateWithEncoder(payload, chunk); + byte[] viaZlib = InflateWith(static source => new DeflateStream(source, CompressionMode.Decompress), compressed); + byte[] viaLibrary = InflateWith(static source => new InflateStream(source), compressed); + if (!viaZlib.AsSpan().SequenceEqual(payload) || !viaLibrary.AsSpan().SequenceEqual(payload)) + { + throw new OracleDivergenceException( + $"Deflate round trip changed {payload.Length} bytes written in {chunk}-byte pieces: DeflateStream gave {viaZlib.Length}, InflateStream {viaLibrary.Length}."); + } + } + + private static byte[] DeflateWithEncoder(ReadOnlySpan payload, int chunk) + { + using var encoder = new DeflateEncoder(); + using var output = new BiffBuffer(4096); + while (!payload.IsEmpty) + { + int take = Math.Min(chunk, payload.Length); + encoder.Write(payload[..take], output); + payload = payload[take..]; + } + encoder.Finish(output); + return output.Span.ToArray(); + } + + internal static void DeflateSegments(ReadOnlySpan data) + { + if (data.IsEmpty) + { + return; + } + int chunk = 1 + (data[0] * 4099 % 70_000); + ReadOnlySpan payload = data[1..]; + if (payload.Length > MaxInflateOutput) + { + payload = payload[..MaxInflateOutput]; + } + byte[] compressed = DeflateInSegments(payload, chunk); + byte[] viaZlib = InflateWith(static source => new DeflateStream(source, CompressionMode.Decompress), compressed); + byte[] viaLibrary = InflateWith(static source => new InflateStream(source), compressed); + if (!viaZlib.AsSpan().SequenceEqual(payload) || !viaLibrary.AsSpan().SequenceEqual(payload)) + { + throw new OracleDivergenceException( + $"Deflate segments changed {payload.Length} bytes cut into {chunk}-byte segments: DeflateStream gave {viaZlib.Length}, InflateStream {viaLibrary.Length}."); + } + } + + private static byte[] DeflateInSegments(ReadOnlySpan payload, int chunk) + { + using var output = new BiffBuffer(4096); + while (!payload.IsEmpty) + { + int take = Math.Min(chunk, payload.Length); + SegmentCompressor.Compress(payload[..take], CompressionLevel.Fastest, output); + payload = payload[take..]; + } + output.Write(SegmentCompressor.FinalBlock); + return output.Span.ToArray(); + } + private static void RequireSameInflate(byte[] actual, byte[] reference, bool prefixOnly) { bool same = prefixOnly ? reference.AsSpan().StartsWith(actual) : actual.AsSpan().SequenceEqual(reference); diff --git a/tests/ExcelReader.Fuzz/Program.cs b/tests/ExcelReader.Fuzz/Program.cs index f60d1b89..023a6a64 100644 --- a/tests/ExcelReader.Fuzz/Program.cs +++ b/tests/ExcelReader.Fuzz/Program.cs @@ -31,6 +31,8 @@ internal static class Program ["csv-sniff"] = Harnesses.CsvSniff, ["csv-parallel"] = Harnesses.CsvParallel, ["inflate"] = Harnesses.Inflate, + ["deflate"] = Harnesses.Deflate, + ["deflate-segments"] = Harnesses.DeflateSegments, }; private static async Task Main(string[] args) diff --git a/tests/ExcelReader.Fuzz/SeedCorpus.cs b/tests/ExcelReader.Fuzz/SeedCorpus.cs index a63119a4..ef8f01ab 100644 --- a/tests/ExcelReader.Fuzz/SeedCorpus.cs +++ b/tests/ExcelReader.Fuzz/SeedCorpus.cs @@ -50,6 +50,8 @@ private static readonly (string Target, string[] Seeds)[] _layout = ("csv-parallel", ["csv", "csv-ragged", "csv-parallel-0", "csv-parallel-1", "csv-parallel-2", "csv-parallel-3", "csv-parallel-4", "csv-parallel-5", "csv-parallel-6"]), ("inflate", ["inflate-stored", "inflate-fixed", "inflate-dynamic"]), + ("deflate", ["deflate-text", "deflate-zeros", "deflate-noise"]), + ("deflate-segments", ["deflate-segments-text", "deflate-segments-zeros", "deflate-segments-noise"]), ]; internal static async Task GenerateAsync(string rootDirectory) @@ -101,6 +103,10 @@ private static async Task> BuildAllAsync() { seeds[name] = bytes; } + foreach ((string name, byte[] bytes) in DeflateSeeds()) + { + seeds[name] = bytes; + } return seeds; } @@ -113,6 +119,23 @@ private static async Task> BuildAllAsync() yield return ("inflate-dynamic", Deflate(sheet, CompressionLevel.Optimal)); } + private static IEnumerable<(string Name, byte[] Bytes)> DeflateSeeds() + { + byte[] sheet = Utf8(string.Concat(Enumerable.Range(1, 400).Select(static row => string.Create( + CultureInfo.InvariantCulture, $"{row * 1.5}")))); + byte[] noise = new byte[5000]; + for (int i = 0; i < noise.Length; i++) + { + noise[i] = (byte)((uint)(i * 2654435761u) >> 24); + } + yield return ("deflate-text", [16, .. sheet]); + yield return ("deflate-zeros", [255, .. new byte[70_000]]); + yield return ("deflate-noise", [1, .. noise]); + yield return ("deflate-segments-text", [1, .. sheet]); + yield return ("deflate-segments-zeros", [2, .. new byte[70_000]]); + yield return ("deflate-segments-noise", [0, .. noise]); + } + private static byte[] Deflate(byte[] raw, CompressionLevel level) { using var compressed = new MemoryStream(); diff --git a/tests/ExcelReader.Tests/Native/CallbackSourceTests.cs b/tests/ExcelReader.Tests/Native/CallbackSourceTests.cs index 29cc16c6..602084ca 100644 --- a/tests/ExcelReader.Tests/Native/CallbackSourceTests.cs +++ b/tests/ExcelReader.Tests/Native/CallbackSourceTests.cs @@ -24,14 +24,17 @@ private sealed class Pinned(TestSource source) : IDisposable { private GCHandle _handle = GCHandle.Alloc(source); - internal NativeSourceRaw Raw(long? length = null) => new() + internal NativeSourceRaw Raw(long? length = null) { - StructSize = sizeof(NativeSourceRaw), - UserData = (void*)GCHandle.ToIntPtr(_handle), - Length = length ?? source.Bytes.Length, - ReadAt = &ReadAt, - Release = &Release, - }; + return new() + { + StructSize = sizeof(NativeSourceRaw), + UserData = (void*)GCHandle.ToIntPtr(_handle), + Length = length ?? source.Bytes.Length, + ReadAt = &ReadAt, + Release = &Release, + }; + } public void Dispose() { @@ -72,7 +75,10 @@ private static int Open(Pinned pinned, int format, out NativeHandle? handle, Nat return ReadApi.OpenSource(source!, format, options, out handle); } - private static NativeOpenOptionsRaw Options() => new() { StructSize = sizeof(NativeOpenOptionsRaw) }; + private static NativeOpenOptionsRaw Options() + { + return new() { StructSize = sizeof(NativeOpenOptionsRaw) }; + } private static byte[] BuildPrefetchedXlsx() { diff --git a/tests/ExcelReader.Tests/Native/CallbackStreamTests.cs b/tests/ExcelReader.Tests/Native/CallbackStreamTests.cs index 4fac787a..97bc5e37 100644 --- a/tests/ExcelReader.Tests/Native/CallbackStreamTests.cs +++ b/tests/ExcelReader.Tests/Native/CallbackStreamTests.cs @@ -66,7 +66,10 @@ private static int Open(Pinned pinned, int format, out NativeHandle? handle, Nat return ReadApi.OpenStream(stream!, format, options, out handle); } - private static NativeOpenOptionsRaw Options() => new() { StructSize = sizeof(NativeOpenOptionsRaw) }; + private static NativeOpenOptionsRaw Options() + { + return new() { StructSize = sizeof(NativeOpenOptionsRaw) }; + } private static int CountRows(NativeHandle handle, int sheet) { diff --git a/tests/ExcelReader.Tests/Native/OpenOptionsLayoutTests.cs b/tests/ExcelReader.Tests/Native/OpenOptionsLayoutTests.cs index e3e690fd..c49fd572 100644 --- a/tests/ExcelReader.Tests/Native/OpenOptionsLayoutTests.cs +++ b/tests/ExcelReader.Tests/Native/OpenOptionsLayoutTests.cs @@ -12,6 +12,33 @@ public void The_6_0_Layout_Ends_Where_The_New_Fields_Begin() Assert.Equal(112, sizeof(NativeOpenOptionsRaw)); } + [Fact] + public void The_Checksum_Switch_Sits_In_The_Former_Tail_Padding() + { + Assert.Equal(108, (int)System.Runtime.InteropServices.Marshal.OffsetOf(nameof(NativeOpenOptionsRaw.VerifyZipChecksums))); + } + + [Theory] + [InlineData(0, true)] + [InlineData(1, false)] + [InlineData(2, true)] + public void The_Checksum_Switch_Reaches_The_Reader_Options(int state, bool expected) + { + NativeOpenOptionsRaw raw = new() { StructSize = sizeof(NativeOpenOptionsRaw), VerifyZipChecksums = state }; + + Assert.True(NativeOpenOptions.TryDecode(raw, out NativeOpenOptions options, out string? error), error); + Assert.Equal(expected, options.ToExcelReaderOptions().VerifyZipChecksums); + } + + [Fact] + public void An_Unknown_Checksum_State_Is_Rejected() + { + NativeOpenOptionsRaw raw = new() { StructSize = sizeof(NativeOpenOptionsRaw), VerifyZipChecksums = 3 }; + + Assert.False(NativeOpenOptions.TryDecode(raw, out _, out string? error)); + Assert.Contains("verify_zip_checksums", error, StringComparison.Ordinal); + } + [Fact] public void A_6_0_Sized_Struct_Is_Read_As_A_Prefix() { diff --git a/tests/ExcelReader.Tests/Parser/ParallelCsv/AggregateCsvModelTests.cs b/tests/ExcelReader.Tests/Parser/ParallelCsv/AggregateCsvModelTests.cs index 4c108968..96a14747 100644 --- a/tests/ExcelReader.Tests/Parser/ParallelCsv/AggregateCsvModelTests.cs +++ b/tests/ExcelReader.Tests/Parser/ParallelCsv/AggregateCsvModelTests.cs @@ -356,9 +356,21 @@ public struct WideRow : IEquatable { public int Last { get; set; } - public override bool Equals(object? obj) => obj is WideRow row && Equals(row); - public readonly bool Equals(WideRow other) => Last == other.Last; - public override readonly int GetHashCode() => Last.GetHashCode(); + public override bool Equals(object? obj) + { + return obj is WideRow row && Equals(row); + } + + public readonly bool Equals(WideRow other) + { + return Last == other.Last; + } + + public override readonly int GetHashCode() + { + return Last.GetHashCode(); + } + public static bool operator ==(WideRow left, WideRow right) => left.Equals(right); public static bool operator !=(WideRow left, WideRow right) => !left.Equals(right); } diff --git a/tests/ExcelReader.Tests/Parser/SheetPipeline/XlsxPipelineTests.cs b/tests/ExcelReader.Tests/Parser/SheetPipeline/XlsxPipelineTests.cs index f58ccccd..e2621211 100644 --- a/tests/ExcelReader.Tests/Parser/SheetPipeline/XlsxPipelineTests.cs +++ b/tests/ExcelReader.Tests/Parser/SheetPipeline/XlsxPipelineTests.cs @@ -99,7 +99,10 @@ private sealed class CancelAfterArmedStream(Stream inner) : Stream { private long _remaining = long.MaxValue; - public void Arm(long budget) => _remaining = budget; + public void Arm(long budget) + { + _remaining = budget; + } public override bool CanRead => inner.CanRead; public override bool CanSeek => inner.CanSeek; @@ -136,9 +139,20 @@ public override async ValueTask ReadAsync(Memory buffer, Cancellation return n; } - public override long Seek(long offset, SeekOrigin origin) => inner.Seek(offset, origin); - public override void SetLength(long value) => throw new NotSupportedException(); - public override void Write(byte[] buffer, int offset, int count) => throw new NotSupportedException(); + public override long Seek(long offset, SeekOrigin origin) + { + return inner.Seek(offset, origin); + } + + public override void SetLength(long value) + { + throw new NotSupportedException(); + } + + public override void Write(byte[] buffer, int offset, int count) + { + throw new NotSupportedException(); + } private void Spend(int n) { diff --git a/tests/ExcelReader.Tests/Parser/TypedCellFormatParityTests.cs b/tests/ExcelReader.Tests/Parser/TypedCellFormatParityTests.cs new file mode 100644 index 00000000..e4d771ab --- /dev/null +++ b/tests/ExcelReader.Tests/Parser/TypedCellFormatParityTests.cs @@ -0,0 +1,191 @@ +using ExcelReader.Core.Parser; +using ExcelReader.Core.Reader; +using ExcelReader.Core.Writer; +using ExcelReader.Core.Writer.Xls; +using ExcelReader.Core.Writer.Xlsb; +using ExcelReader.Core.Writer.Xlsx; + +namespace ExcelReader.Tests.Parser +{ + /// + /// A typed property binds the same way whichever format the cell came from. XLSX carries a number as + /// text; XLSB and XLS carry it in binary, with nothing in . + /// + public sealed class TypedCellFormatParityTests + { + public static TheoryData Formats => ["xlsx", "xlsb", "xls"]; + + private sealed class Flags + { + public bool On { get; set; } + public bool? Maybe { get; set; } + } + + private sealed class Spans + { + public TimeSpan Days { get; set; } + public TimeSpan? MaybeDays { get; set; } + } + + private sealed class Shifts + { + public TimeOnly Start { get; set; } + public TimeOnly? End { get; set; } + } + + private sealed class Days + { + public DayOfWeek Day { get; set; } + } + + [Theory] + [MemberData(nameof(Formats))] + public void ANumericZeroOrOneBindsToBool(string format) + { + byte[] bytes = Write(format, ["On", "Maybe"], row => + { + row.Write(1); + row.Write(0); + }, row => + { + row.Write(0.0); + row.Write(1.0); + }, row => + { + row.Write(2); + row.Write(0.5); + }); + + List rows = Parse(bytes); + + Assert.Equal([true, false, false], rows.Select(r => r.On)); + Assert.Equal([false, true, null], rows.Select(r => r.Maybe)); + } + + [Theory] + [MemberData(nameof(Formats))] + public void ANumberOtherThanZeroOrOneIsNotABool(string format) + { + byte[] bytes = Write(format, ["On", "Maybe"], row => + { + row.Write(2); + row.Write(1); + }); + var strict = new ExcelParserOptions { ThrowOnParseFailure = true }; + + using IExcelWorkbook workbook = Excel.Open(bytes); + Assert.Throws(() => + { + foreach (Flags _ in ExcelParser.FromAttributes(strict).Parse(workbook.FirstSheet)) + { + } + }); + } + + [Theory] + [MemberData(nameof(Formats))] + public void ANumberBindsToAParsableType(string format) + { + byte[] bytes = Write(format, ["Days", "MaybeDays"], row => + { + row.Write(3); + row.Write(300); + }); + + Spans row = Assert.Single(Parse(bytes)); + + Assert.Equal(TimeSpan.FromDays(3), row.Days); + Assert.Equal(TimeSpan.FromDays(300), row.MaybeDays); + } + + [Theory] + [MemberData(nameof(Formats))] + public void ATimeBindsFromItsFractionOrItsText(string format) + { + byte[] bytes = Write(format, ["Start", "End"], row => + { + row.Write(new TimeOnly(8, 30)); + row.Write("17:45"); + }, row => + { + row.Write("09:15"); + row.Write(0.75); + }); + + List rows = Parse(bytes); + + Assert.Equal([new TimeOnly(8, 30), new TimeOnly(9, 15)], rows.Select(r => r.Start)); + Assert.Equal([new TimeOnly(17, 45), new TimeOnly(18, 0)], rows.Select(r => r.End)); + } + + [Theory] + [MemberData(nameof(Formats))] + public void ADateStyledNumberBindsToAnEnumByValue(string format) + { + byte[] bytes = Write(format, ["Day"], row => row.Write(new DateTime(1900, 1, 1)), row => row.Write(5)); + + Assert.Equal([DayOfWeek.Monday, DayOfWeek.Friday], Parse(bytes).Select(r => r.Day)); + } + + private static List Parse(byte[] bytes) + where T : new() + { + using IExcelWorkbook workbook = Excel.Open(bytes); + var rows = new List(); + foreach (T row in ExcelParser.FromAttributes().Parse(workbook.FirstSheet)) + { + rows.Add(row); + } + return rows; + } + + private static byte[] Write(string format, string[] header, params Action[] rows) + { + using MemoryStream ms = new(); + switch (format) + { + case "xlsx": + using (XlsxWorkbookWriter wb = XlsxWorkbookWriter.Create(ms, leaveOpen: true)) + { + Fill(wb.AddSheet("S1"), header, rows); + wb.End(); + } + break; + case "xlsb": + using (XlsbWorkbookWriter wb = XlsbWorkbookWriter.Create(ms, leaveOpen: true)) + { + Fill(wb.AddSheet("S1"), header, rows); + wb.End(); + } + break; + default: + using (XlsWorkbookWriter wb = XlsWorkbookWriter.Create(ms, leaveOpen: true)) + { + Fill(wb.AddSheet("S1"), header, rows); + wb.End(); + } + break; + } + return ms.ToArray(); + } + + private static void Fill(TSheet sheet, string[] header, Action[] rows) + where TSheet : ISheetWriter + where TRow : IRowWriter, IDisposable + { + using (TRow row = sheet.StartRow()) + { + foreach (string name in header) + { + row.Write(name); + } + } + foreach (Action write in rows) + { + using TRow row = sheet.StartRow(); + write(row); + } + sheet.End(); + } + } +} diff --git a/tests/ExcelReader.Tests/Reader/ExcelWrittenTextTests.cs b/tests/ExcelReader.Tests/Reader/ExcelWrittenTextTests.cs new file mode 100644 index 00000000..6f4c7079 --- /dev/null +++ b/tests/ExcelReader.Tests/Reader/ExcelWrittenTextTests.cs @@ -0,0 +1,125 @@ +using System.IO.Compression; +using System.Text; +using ExcelReader.Core.Reader; + +namespace ExcelReader.Tests.Reader +{ + /// + /// Text as Excel itself stores it. excel-formula-strings was saved by Excel 16 in both formats: + /// A1 holds a line break, B1 and C1 formulas whose text has whitespace at an end, B2:B4 a formula + /// filled down, and C2:C4 an array formula. + /// + public sealed class ExcelWrittenTextTests + { + private static readonly string[][] Expected = + [ + ["a\nb", "\nnl", " pad "], + ["1", "1s", "1a"], + ["2", "2s", "2a"], + ["3", "3s", "3a"], + ]; + + [Theory] + [InlineData("xlsx")] + [InlineData("xls")] + public void AnExcelFileReadsAsExcelShowsIt(string extension) + { + string path = Path.Combine(AppContext.BaseDirectory, "data", "excel-formula-strings." + extension); + + using IExcelWorkbook workbook = Excel.Open(path); + + Assert.Equal(Expected, Rows(workbook, 3)); + } + + [Fact] + public void ALiteralCarriageReturnIsAnXmlLineEnd() + { + byte[] package = Xlsx( + "" + + "012" + + "in\r\nline" + + "Xfor\r\nmula" + + "34" + + "", + "a\r\nb" + + "lone\rcr" + + "kept \r\nref and _x000D_\r\nescape" + + "rich\r\nrun" + + "a_b\r\nc"); + + using IExcelWorkbook workbook = Excel.Open(package); + + string[] row = Assert.Single(Rows(workbook, 7)); + Assert.Equal( + ["a\nb", "lone\ncr", "kept\r\nref and \r\nescape", "in\nline", "for\nmula", "rich\nrun", "a_b\nc"], + row); + } + + [Fact] + public void AValueElementMayCarryAttributes() + { + byte[] package = Xlsx( + "" + + "X padded " + + "42" + + "Xplain" + + "X" + + "0" + + "", + "shared"); + + using IExcelWorkbook workbook = Excel.Open(package); + + string[] row = Assert.Single(Rows(workbook, 5)); + Assert.Equal([" padded ", "42", "plain", "", "shared"], row); + } + + private static List Rows(IExcelWorkbook workbook, int columns) + { + var rows = new List(); + using IExcelRowEnumerator enumerator = workbook.FirstSheet.GetEnumerator(); + while (enumerator.MoveNext()) + { + string[] cells = new string[columns]; + for (int c = 0; c < columns; c++) + { + cells[c] = enumerator.Current[c].GetString(); + } + rows.Add(cells); + } + return rows; + } + + private static byte[] Xlsx(string rows, string sharedItems) + { + const string Main = "http://schemas.openxmlformats.org/spreadsheetml/2006/main"; + const string Rel = "http://schemas.openxmlformats.org/officeDocument/2006/relationships"; + using MemoryStream ms = new(); + using (var zip = new ZipArchive(ms, ZipArchiveMode.Create, leaveOpen: true)) + { + Add(zip, "[Content_Types].xml", + "" + + "" + + ""); + Add(zip, "_rels/.rels", + "" + + $""); + Add(zip, "xl/workbook.xml", + $""); + Add(zip, "xl/_rels/workbook.xml.rels", + "" + + $"" + + $""); + Add(zip, "xl/sharedStrings.xml", $"{sharedItems}"); + Add(zip, "xl/worksheets/sheet1.xml", $"{rows}"); + } + return ms.ToArray(); + } + + private static void Add(ZipArchive zip, string name, string content) + { + using Stream entry = zip.CreateEntry(name).Open(); + entry.Write(Encoding.UTF8.GetBytes(content)); + } + } +} diff --git a/tests/ExcelReader.Tests/Reader/Xlsx/CorruptSheetEntryTests.cs b/tests/ExcelReader.Tests/Reader/Xlsx/CorruptSheetEntryTests.cs index 89f8c156..b6b165bc 100644 --- a/tests/ExcelReader.Tests/Reader/Xlsx/CorruptSheetEntryTests.cs +++ b/tests/ExcelReader.Tests/Reader/Xlsx/CorruptSheetEntryTests.cs @@ -28,7 +28,8 @@ private static int CountRows(XlsxWorkbook workbook) return count; } - // Damage at +4096 decodes as a stream that ends early: before the fix every reader stopped at 298 rows without an error. + // The damage either breaks the deflate stream or decodes as garbage XML that trips a limit. + // Either way the read must fail, not stop early without an error. [Theory] [InlineData(4096, false)] [InlineData(4096, true)] @@ -40,7 +41,8 @@ public async Task ADamagedSheetEntryThrowsInsteadOfEndingEarly(int offset, bool Array.Fill(bytes, (byte)0xFF, entry + offset, 512); using XlsxWorkbook workbook = Excel.FromXlsx(bytes, new ExcelReaderOptions { PrefetchDecompression = prefetch }); - Assert.Throws(() => CountRows(workbook)); + Exception failure = Assert.ThrowsAny(() => CountRows(workbook)); + Assert.True(failure is InvalidDataException or ExcelLimitExceededException, failure.GetType().Name); } } } diff --git a/tests/ExcelReader.Tests/Reader/Zip/ZipChecksumTests.cs b/tests/ExcelReader.Tests/Reader/Zip/ZipChecksumTests.cs new file mode 100644 index 00000000..0e104524 --- /dev/null +++ b/tests/ExcelReader.Tests/Reader/Zip/ZipChecksumTests.cs @@ -0,0 +1,153 @@ +using System.IO.Compression; +using System.Text; +using ExcelReader.Core.Reader; +using ExcelReader.Core.Writer.Xlsb; +using ExcelReader.Core.Writer.Xlsx; + +namespace ExcelReader.Tests.Reader.Zip +{ + /// A package whose entry data no longer matches its CRC-32 is refused, unless the check is turned off. + public sealed class ZipChecksumTests + { + private const int Rows = 20_000; + + private static readonly ExcelReaderOptions Unchecked = new() { VerifyZipChecksums = false }; + + public static TheoryData Cases + { + get + { + var data = new TheoryData(); + foreach (bool xlsb in new[] { false, true }) + { + foreach (bool stream in new[] { false, true }) + { + data.Add(xlsb, stream, false); + data.Add(xlsb, stream, true); + } + } + return data; + } + } + + [Theory] + [MemberData(nameof(Cases))] + public void ADamagedSheetIsRefused(bool xlsb, bool fromStream, bool prefetch) + { + byte[] package = Damage(Write(xlsb), xlsb, xlsb ? "xl/worksheets/sheet1.bin" : "xl/worksheets/sheet1.xml"); + var options = new ExcelReaderOptions { PrefetchDecompression = prefetch }; + + InvalidDataException error = Assert.Throws(() => Sum(package, fromStream, options)); + Assert.Contains("CRC-32", error.Message, StringComparison.Ordinal); + } + + [Theory] + [MemberData(nameof(Cases))] + public void TheCheckCanBeTurnedOff(bool xlsb, bool fromStream, bool prefetch) + { + byte[] package = Damage(Write(xlsb), xlsb, xlsb ? "xl/worksheets/sheet1.bin" : "xl/worksheets/sheet1.xml"); + + Sum(package, fromStream, Unchecked with { PrefetchDecompression = prefetch }); + } + + [Theory] + [MemberData(nameof(Cases))] + public void AnIntactPackageStillReads(bool xlsb, bool fromStream, bool prefetch) + { + long sum = Sum(Write(xlsb), fromStream, new ExcelReaderOptions { PrefetchDecompression = prefetch }); + + Assert.Equal((long)Rows * (Rows + 1) / 2, sum); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void ADamagedSharedStringTableIsRefused(bool xlsb) + { + byte[] package = Damage(Write(xlsb, sharedStrings: true), xlsb, xlsb ? "xl/sharedStrings.bin" : "xl/sharedStrings.xml"); + + Assert.Throws(() => Sum(package, fromStream: false, ExcelReaderOptions.Default)); + Assert.Throws(() => Sum(package, fromStream: true, ExcelReaderOptions.Default)); + } + + private static long Sum(byte[] package, bool fromStream, ExcelReaderOptions options) + { + using IExcelWorkbook workbook = fromStream + ? Excel.Open(new MemoryStream(package, writable: false), leaveOpen: false, options) + : Excel.Open(package, options); + long sum = 0; + using IExcelRowEnumerator rows = workbook.FirstSheet.GetEnumerator(); + while (rows.MoveNext()) + { + rows.Current[0].GetString(); + Assert.True(rows.Current[1].TryGetDouble(out double number)); + sum += (long)number; + } + return sum; + } + + private static byte[] Write(bool xlsb, bool sharedStrings = false) + { + using MemoryStream ms = new(); + if (xlsb) + { + using XlsbWorkbookWriter wb = XlsbWorkbookWriter.Create(ms, leaveOpen: true, new XlsbWriterOptions { Compression = CompressionLevel.NoCompression, UseSharedStrings = sharedStrings }); + XlsbSheetWriter sheet = wb.AddSheet("S1"); + for (int i = 1; i <= Rows; i++) + { + using XlsbRowWriter row = sheet.StartRow(); + row.Write("text" + (i % 50)); + row.Write(i); + } + sheet.End(); + wb.End(); + } + else + { + using XlsxWorkbookWriter wb = XlsxWorkbookWriter.Create(ms, leaveOpen: true, new XlsxWriterOptions { Compression = CompressionLevel.NoCompression, UseSharedStrings = sharedStrings }); + XlsxSheetWriter sheet = wb.AddSheet("S1"); + for (int i = 1; i <= Rows; i++) + { + using XlsxRowWriter row = sheet.StartRow(); + row.Write("text" + (i % 50)); + row.Write(i); + } + sheet.End(); + wb.End(); + } + return ms.ToArray(); + } + + // The package is stored, not deflated, so one changed letter of the cell text "text" stays valid + // XML or BIFF12 and only the checksum can tell. + private static byte[] Damage(byte[] package, bool xlsb, string entryName) + { + byte[] content; + using (var zip = new ZipArchive(new MemoryStream(package), ZipArchiveMode.Read)) + { + using Stream data = zip.GetEntry(entryName)!.Open(); + using var copy = new MemoryStream(); + data.CopyTo(copy); + content = copy.ToArray(); + } + // BIFF12 stores text as UTF-16. + byte[] needle = xlsb ? Encoding.Unicode.GetBytes("text") : Encoding.ASCII.GetBytes("text"); + int inEntry = Find(content, needle, content.Length / 2); + int inPackage = Find(package, content.AsSpan(inEntry - 16, 64), 0) + 16; + byte[] damaged = (byte[])package.Clone(); + damaged[inPackage] = (byte)'T'; + return damaged; + } + + private static int Find(ReadOnlySpan haystack, ReadOnlySpan needle, int from) + { + int at = haystack[from..].IndexOf(needle); + if (at < 0 && from > 0) + { + at = haystack.IndexOf(needle) - from; + } + Assert.True(from + at >= 0, "needle not found"); + return from + at; + } + } +} diff --git a/tests/ExcelReader.Tests/Reader/Zip/ZipIndexTests.cs b/tests/ExcelReader.Tests/Reader/Zip/ZipIndexTests.cs index 4328dfc1..3d9156bb 100644 --- a/tests/ExcelReader.Tests/Reader/Zip/ZipIndexTests.cs +++ b/tests/ExcelReader.Tests/Reader/Zip/ZipIndexTests.cs @@ -392,7 +392,7 @@ public void CompressedSizeEndingExactlyAtTheFileEndIsAcceptedAndOneMoreIsRejecte long exact = probe.Bytes.Length - probe.DataOffset; Zip64Layout fits = BuildStoredZip64(Zip64Payload, compressedSize: exact, uncompressedSize: exact); - using (ZipIndex index = ZipIndex.Create(fits.Bytes, ExcelReaderOptions.Default)) + using (ZipIndex index = ZipIndex.Create(fits.Bytes, Unchecked)) { Assert.True(index.TryGetEntry("a.txt"u8, out ZipEntryRef entry)); using ZipPart part = index.OpenPart(entry, new DecompressedByteCounter(0)); @@ -508,12 +508,16 @@ public void LocalExtraLengthPushingDataPastTheFileEndThrowsInvalidDataOnOpen() private static readonly bool[] Modes = [false, true]; - private static async Task CreatePositionalAsync(byte[] bytes, bool useAsync, CancellationToken ct) + // The size-boundary tests stretch a stored entry over bytes its recorded CRC-32 never covered. + private static readonly ExcelReaderOptions Unchecked = new() { VerifyZipChecksums = false }; + + private static async Task CreatePositionalAsync( + byte[] bytes, bool useAsync, CancellationToken ct, ExcelReaderOptions? options = null) { ByteSource source = ByteSource.FromStream(new MemoryStream(bytes, writable: false), leaveOpen: false); return useAsync - ? await ZipIndex.CreateAsync(source, ExcelReaderOptions.Default, ct) - : ZipIndex.Create(source, ExcelReaderOptions.Default); + ? await ZipIndex.CreateAsync(source, options ?? ExcelReaderOptions.Default, ct) + : ZipIndex.Create(source, options ?? ExcelReaderOptions.Default); } private static async Task CreateFailsAsync(byte[] bytes, bool useAsync, CancellationToken ct) @@ -675,7 +679,7 @@ public async Task PositionalCompressedSizeEndingExactlyAtTheFileEndIsAcceptedAnd long exact = probe.Bytes.Length - probe.DataOffset; Zip64Layout fits = BuildStoredZip64(Zip64Payload, compressedSize: exact, uncompressedSize: exact); - using (ZipIndex index = await CreatePositionalAsync(fits.Bytes, useAsync, ct)) + using (ZipIndex index = await CreatePositionalAsync(fits.Bytes, useAsync, ct, Unchecked)) { Assert.True(index.TryGetEntry("a.txt"u8, out ZipEntryRef entry)); using ZipPart part = await OpenPartPositionalAsync(index, entry, useAsync, ct); diff --git a/tests/ExcelReader.Tests/Writer/Parallel/BatchRunnerTests.cs b/tests/ExcelReader.Tests/Writer/Parallel/BatchRunnerTests.cs new file mode 100644 index 00000000..410597e4 --- /dev/null +++ b/tests/ExcelReader.Tests/Writer/Parallel/BatchRunnerTests.cs @@ -0,0 +1,196 @@ +using ExcelReader.Core.Reader; +using ExcelReader.Core.Writer; +using ExcelReader.Core.Writer.Parallel; + +namespace ExcelReader.Tests.Writer.Parallel +{ + public class BatchRunnerTests + { + private sealed class Probe + { + private int _running; + private int _peak; + private int _formats; + + internal List<(int RowsBefore, int Rows)> Emitted { get; } = []; + internal int Peak => _peak; + internal int Running => Volatile.Read(ref _running); + internal int Formats => Volatile.Read(ref _formats); + internal int FailAtRowsBefore { get; init; } = -1; + + internal BatchSegment Format(int[] batch, int count, int rowsBefore) + { + int now = Interlocked.Increment(ref _running); + Interlocked.Increment(ref _formats); + InterlockedMax(ref _peak, now); + try + { + Thread.Sleep(2); + if (rowsBefore == FailAtRowsBefore) + { + throw new FormatException("batch " + rowsBefore); + } + Assert.Equal(rowsBefore, batch[0]); + return new BatchSegment(count) { RowFirst = rowsBefore }; + } + finally + { + Interlocked.Decrement(ref _running); + } + } + + internal ValueTask Emit(BatchSegment segment, CancellationToken ct) + { + Emitted.Add((segment.RowFirst, segment.Rows)); + return ValueTask.CompletedTask; + } + + private static void InterlockedMax(ref int target, int value) + { + int seen = Volatile.Read(ref target); + while (value > seen) + { + int previous = Interlocked.CompareExchange(ref target, value, seen); + if (previous == seen) + { + return; + } + seen = previous; + } + } + } + + private static BatchRunner Runner(Probe probe, int degree, int batchRows, CancellationToken ct, int rowsBefore = 0) + { + var options = new ParallelWriteOptions { DegreeOfParallelism = degree, BatchRows = batchRows }; + return new BatchRunner(options, rowsBefore, probe.Format, probe.Emit, ct); + } + + private static async IAsyncEnumerable Async(int count) + { + for (int i = 0; i < count; i++) + { + await Task.Yield(); + yield return i; + } + } + + [Theory] + [InlineData(0, 0)] + [InlineData(1, 1)] + [InlineData(10, 1)] + [InlineData(11, 2)] + [InlineData(95, 10)] + public async Task BatchesComeOutInOrderWithAPartialLastOne(int records, int expectedBatches) + { + foreach (int degree in new[] { 1, 4 }) + { + var probe = new Probe(); + await Runner(probe, degree, 10, TestContext.Current.CancellationToken) + .RunAsync(new RecordSource(Enumerable.Range(0, records))); + + Assert.Equal(expectedBatches, probe.Emitted.Count); + Assert.Equal(records, probe.Emitted.Sum(static e => e.Rows)); + Assert.Equal(probe.Emitted.OrderBy(static e => e.RowsBefore), probe.Emitted); + Assert.True(probe.Peak <= degree); + } + } + + [Fact] + public async Task AnAsyncSourceGivesTheSameBatches() + { + var probe = new Probe(); + await Runner(probe, 3, 10, TestContext.Current.CancellationToken).RunAsync(new RecordSource(Async(45))); + + Assert.Equal([(0, 10), (10, 10), (20, 10), (30, 10), (40, 5)], probe.Emitted); + } + + [Fact] + public async Task RowsBeforeOffsetsEveryBatch() + { + var probe = new Probe(); + int[] source = [.. Enumerable.Range(7, 25)]; + await Runner(probe, 2, 10, TestContext.Current.CancellationToken, rowsBefore: 7).RunAsync(new RecordSource(source)); + + Assert.Equal([(7, 10), (17, 10), (27, 5)], probe.Emitted); + } + + [Fact] + public async Task AFailingBatchStopsTheRunAndLeavesNoWorker() + { + var probe = new Probe { FailAtRowsBefore = 30 }; + BatchRunner runner = Runner(probe, 4, 10, TestContext.Current.CancellationToken); + + FormatException failure = await Assert.ThrowsAsync( + async () => await runner.RunAsync(new RecordSource(Enumerable.Range(0, 500)))); + + Assert.Equal("batch 30", failure.Message); + Assert.Equal(0, probe.Running); + Assert.All(probe.Emitted, static e => Assert.True(e.RowsBefore < 30)); + } + + [Fact] + public async Task AThrowingSourceSurfacesItsOwnException() + { + static IEnumerable Source() + { + for (int i = 0; i < 25; i++) + { + yield return i; + } + throw new InvalidDataException("source"); + } + var probe = new Probe(); + + await Assert.ThrowsAsync( + async () => await Runner(probe, 4, 10, TestContext.Current.CancellationToken).RunAsync(new RecordSource(Source()))); + Assert.Equal(0, probe.Running); + } + + [Fact] + public async Task CancellationStopsBetweenBatches() + { + using var cts = new CancellationTokenSource(); + var probe = new Probe(); + IEnumerable Source() + { + for (int i = 0; i < 1_000; i++) + { + if (i == 35) + { + cts.Cancel(); + } + yield return i; + } + } + + await Assert.ThrowsAnyAsync( + async () => await Runner(probe, 2, 10, cts.Token).RunAsync(new RecordSource(Source()))); + Assert.Equal(0, probe.Running); + Assert.True(probe.Emitted.Count <= 3); + } + + [Fact] + public async Task TheRowLimitThrowsBeforeTheOverflowingBatchIsFormatted() + { + var probe = new Probe(); + int start = 1_048_576 - 15; + int[] source = [.. Enumerable.Range(start, 30)]; + + await Assert.ThrowsAsync( + async () => await Runner(probe, 1, 10, TestContext.Current.CancellationToken, rowsBefore: start).RunAsync(new RecordSource(source))); + Assert.Equal(1, probe.Formats); + Assert.DoesNotContain(probe.Emitted, e => e.RowsBefore > start); + } + + [Fact] + public void OptionsRejectValuesOutOfRange() + { + Assert.Throws(() => new ParallelWriteOptions { DegreeOfParallelism = -1 }.Validate()); + Assert.Throws(() => new ParallelWriteOptions { BatchRows = 0 }.Validate()); + ParallelWriteOptions.Default.Validate(); + Assert.Equal(8192, ParallelWriteOptions.Default.BatchRows); + Assert.Equal(0, ParallelWriteOptions.Default.DegreeOfParallelism); + } + } +} diff --git a/tests/ExcelReader.Tests/Writer/Parallel/ParallelRecordWriteTests.cs b/tests/ExcelReader.Tests/Writer/Parallel/ParallelRecordWriteTests.cs new file mode 100644 index 00000000..e49b50a4 --- /dev/null +++ b/tests/ExcelReader.Tests/Writer/Parallel/ParallelRecordWriteTests.cs @@ -0,0 +1,383 @@ +using System.Globalization; +using System.IO.Compression; +using ExcelReader.Core.Reader; +using ExcelReader.Core.Reader.Xlsx; +using ExcelReader.Core.Writer; +using ExcelReader.Core.Writer.Xlsx; +using ExcelReader.Tests.Writer.Zip; + +namespace ExcelReader.Tests.Writer.Parallel +{ + public class ParallelRecordWriteTests + { + internal sealed record Item(int Id, string Name, double Value, DateTime Day); + + internal const int Columns = 4; + + internal static List Items(int count, int distinctNames = 37) + { + return [.. Enumerable.Range(0, count).Select(i => new Item( + i, string.Create(CultureInfo.InvariantCulture, $"name {i % distinctNames}"), i * 1.5, new DateTime(2024, 1, 1).AddDays(i % 400)))]; + } + + internal static void Fill(TRow row, Item item) + where TRow : IRowWriter + { + row.Write(item.Id); + row.Write(item.Name); + row.Write(item.Value); + row.Write(item.Day); + } + + internal static List Cells(IExcelWorkbook workbook, int sheet = 0) + { + var rows = new List(); + using IExcelRowEnumerator enumerator = workbook.SheetAt(sheet).GetEnumerator(); + while (enumerator.MoveNext()) + { + string[] cells = new string[Columns]; + for (int c = 0; c < Columns; c++) + { + cells[c] = enumerator.Current[c].GetString(); + } + rows.Add(cells); + } + return rows; + } + + internal static byte[] Entry(byte[] package, string name) + { + using var zip = new ZipArchive(new MemoryStream(package), ZipArchiveMode.Read); + ZipArchiveEntry entry = zip.GetEntry(name)!; + using Stream data = entry.Open(); + using var copy = new MemoryStream(); + data.CopyTo(copy); + Assert.Equal(Crc32Tests.BitwiseCrc(copy.ToArray()), entry.Crc32); + return copy.ToArray(); + } + + public static TheoryData Modes() + { + var data = new TheoryData(); + foreach (CompressionLevel level in Enum.GetValues()) + { + foreach (bool prefetch in new[] { false, true }) + { + data.Add(level, prefetch, false); + data.Add(level, prefetch, true); + } + } + return data; + } + + private static async Task WriteAsync(XlsxWriterOptions options, Func body, bool seekable = true) + { + var ms = new MemoryStream(); + Stream target = seekable ? ms : new ForwardOnlyWriteStream(ms); + await using (XlsxWorkbookWriter wb = XlsxWorkbookWriter.Create(target, leaveOpen: true, options)) + { + XlsxSheetWriter sheet = wb.AddSheet("Data"); + await body(sheet); + await sheet.EndAsync(TestContext.Current.CancellationToken); + await wb.EndAsync(TestContext.Current.CancellationToken); + } + return ms.ToArray(); + } + + private static ParallelWriteOptions Parallel(int degree, int batchRows = 500) + { + return new ParallelWriteOptions { DegreeOfParallelism = degree, BatchRows = batchRows }; + } + + private static List Read(byte[] package) + { + using XlsxWorkbook workbook = Excel.FromXlsx(new MemoryStream(package)); + return Cells(workbook); + } + + [Theory] + [MemberData(nameof(Modes))] + public async Task ParallelGivesTheSameCellsAsSerial(CompressionLevel level, bool prefetch, bool sharedStrings) + { + CancellationToken ct = TestContext.Current.CancellationToken; + List items = Items(6_300); + var options = new XlsxWriterOptions { Compression = level, PrefetchWrite = prefetch, UseSharedStrings = sharedStrings }; + byte[] serial = await WriteAsync(options, async sheet => + await sheet.WriteRecordsAsync(items, (Action)Fill, ct)); + foreach (bool seekable in new[] { true, false }) + { + byte[] parallel = await WriteAsync(options, async sheet => + await sheet.WriteRecordsParallelAsync(items, Fill, Parallel(4), ct), seekable); + + Assert.Equal(Read(serial), Read(parallel)); + Entry(parallel, "xl/worksheets/sheet1.xml"); + } + } + + [Theory] + [InlineData(CompressionLevel.Fastest)] + [InlineData(CompressionLevel.Optimal)] + [InlineData(CompressionLevel.NoCompression)] + public async Task OneWorkerWritesTheSameSheetBytesAsSerial(CompressionLevel level) + { + CancellationToken ct = TestContext.Current.CancellationToken; + List items = Items(2_750); + var options = new XlsxWriterOptions { Compression = level }; + byte[] serial = await WriteAsync(options, async sheet => + await sheet.WriteRecordsAsync(items, (Action)Fill, ct)); + byte[] parallel = await WriteAsync(options, async sheet => + await sheet.WriteRecordsParallelAsync(items, Fill, Parallel(1), ct)); + + Assert.Equal(Entry(serial, "xl/worksheets/sheet1.xml"), Entry(parallel, "xl/worksheets/sheet1.xml")); + } + + [Theory] + [InlineData(0)] + [InlineData(1)] + [InlineData(500)] + [InlineData(501)] + [InlineData(1_733)] + public async Task BatchEdges(int count) + { + CancellationToken ct = TestContext.Current.CancellationToken; + List items = Items(count); + byte[] package = await WriteAsync(XlsxWriterOptions.Default, async sheet => + await sheet.WriteRecordsParallelAsync(items, Fill, Parallel(3), ct)); + + List rows = Read(package); + Assert.Equal(count, rows.Count); + for (int i = 0; i < count; i++) + { + Assert.Equal(i.ToString(CultureInfo.InvariantCulture), rows[i][0]); + Assert.Equal(items[i].Name, rows[i][1]); + } + } + + [Fact] + public async Task ParallelAsTheFirstWriteOnTheSheet() + { + CancellationToken ct = TestContext.Current.CancellationToken; + List items = Items(1_200); + byte[] package = await WriteAsync(XlsxWriterOptions.Default with { PrefetchWrite = true }, async sheet => + await sheet.WriteRecordsParallelAsync(items, Fill, Parallel(4), ct)); + + Assert.Equal(1_200, Read(package).Count); + } + + [Theory] + [InlineData(CompressionLevel.Fastest, false)] + [InlineData(CompressionLevel.Optimal, false)] + [InlineData(CompressionLevel.SmallestSize, false)] + [InlineData(CompressionLevel.NoCompression, false)] + [InlineData(CompressionLevel.Fastest, true)] + [InlineData(CompressionLevel.Optimal, true)] + [InlineData(CompressionLevel.SmallestSize, true)] + [InlineData(CompressionLevel.NoCompression, true)] + public async Task SerialRowsBeforeAndAfterSurvive(CompressionLevel level, bool prefetch) + { + CancellationToken ct = TestContext.Current.CancellationToken; + List items = Items(1_300); + byte[] package = await WriteAsync(new XlsxWriterOptions { Compression = level, PrefetchWrite = prefetch }, async sheet => + { + await sheet.WriteRecordsAsync(items, (Action)Fill, ct); + await sheet.WriteRecordsParallelAsync(items, Fill, Parallel(4), ct); + await sheet.WriteRecordsAsync(items, (Action)Fill, ct); + await sheet.WriteRecordsParallelAsync(items, Fill, Parallel(2), ct); + using XlsxRowWriter last = sheet.StartRow(); + last.Write("end"); + }); + + List rows = Read(package); + Assert.Equal((4 * 1_300) + 1, rows.Count); + for (int block = 0; block < 4; block++) + { + Assert.Equal("0", rows[block * 1_300][0]); + Assert.Equal("1299", rows[(block * 1_300) + 1_299][0]); + Assert.Equal(items[777].Name, rows[(block * 1_300) + 777][1]); + } + Assert.Equal("end", rows[^1][0]); + } + + [Fact] + public async Task ALayoutWritesItsHeaderFirst() + { + CancellationToken ct = TestContext.Current.CancellationToken; + List items = Items(900); + byte[] serial = await WriteAsync(XlsxWriterOptions.Default, async sheet => + await sheet.WriteRecordsAsync(items, ExcelRecordLayout.FromAttributes(), ct)); + byte[] parallel = await WriteAsync(XlsxWriterOptions.Default, async sheet => + await sheet.WriteRecordsParallelAsync(items, ExcelRecordLayout.FromAttributes(), Parallel(4, 100), ct)); + + Assert.Equal(Read(serial), Read(parallel)); + Assert.Equal(901, Read(parallel).Count); + } + + [Fact] + public async Task AnAsyncSourceWorks() + { + CancellationToken ct = TestContext.Current.CancellationToken; + List items = Items(1_050); + async IAsyncEnumerable Source() + { + foreach (Item item in items) + { + await Task.Yield(); + yield return item; + } + } + byte[] package = await WriteAsync(XlsxWriterOptions.Default, async sheet => + await sheet.WriteRecordsParallelAsync(Source(), Fill, Parallel(4, 100), ct)); + + Assert.Equal(1_050, Read(package).Count); + } + + [Fact] + public async Task AParallelTableCoversItsRows() + { + CancellationToken ct = TestContext.Current.CancellationToken; + List items = Items(820); + byte[] serial = await WriteAsync(XlsxWriterOptions.Default, async sheet => + await sheet.WriteTableAsync(items, "Records", ExcelRecordLayout.FromAttributes(), ct: ct)); + byte[] parallel = await WriteAsync(XlsxWriterOptions.Default, async sheet => + await sheet.WriteTableParallelAsync(items, "Records", ExcelRecordLayout.FromAttributes(), options: Parallel(4, 100), ct: ct)); + + Assert.Equal(Read(serial), Read(parallel)); + Assert.Equal(Entry(serial, "xl/tables/table1.xml"), Entry(parallel, "xl/tables/table1.xml")); + } + + [Fact] + public async Task SharedStringsGiveEveryCellItsOwnText() + { + CancellationToken ct = TestContext.Current.CancellationToken; + List items = Items(4_000, distinctNames: 613); + for (int round = 0; round < 25; round++) + { + byte[] package = await WriteAsync(new XlsxWriterOptions { UseSharedStrings = true }, async sheet => + await sheet.WriteRecordsParallelAsync(items, Fill, Parallel(8, 50), ct)); + + List rows = Read(package); + Assert.Equal(items.Count, rows.Count); + for (int i = 0; i < items.Count; i++) + { + Assert.Equal(items[i].Name, rows[i][1]); + } + string sst = System.Text.Encoding.UTF8.GetString(Entry(package, "xl/sharedStrings.xml")); + Assert.Contains("count=\"4000\" uniqueCount=\"613\"", sst, StringComparison.Ordinal); + } + } + + [Fact] + public async Task AThrowingRowFaultsTheSheetAndStopsTheWorkers() + { + CancellationToken ct = TestContext.Current.CancellationToken; + List items = Items(5_000); + int running = 0; + void Throwing(XlsxRowWriter row, Item item) + { + Interlocked.Increment(ref running); + try + { + row.Write(item.Id); + if (item.Id == 2_345) + { + throw new FormatException("row 2345"); + } + row.Write(item.Name); + } + finally + { + Interlocked.Decrement(ref running); + } + } + var ms = new MemoryStream(); + XlsxWorkbookWriter wb = XlsxWorkbookWriter.Create(ms, leaveOpen: true); + XlsxSheetWriter sheet = wb.AddSheet("Data"); + + FormatException failure = await Assert.ThrowsAsync( + async () => await sheet.WriteRecordsParallelAsync(items, Throwing, Parallel(4, 100), ct)); + + Assert.Equal("row 2345", failure.Message); + Assert.Equal(0, Volatile.Read(ref running)); + Assert.True(sheet.ResourcesReleased); + Assert.Throws(() => sheet.StartRow()); + await wb.DisposeAsync(); + } + + [Fact] + public async Task TheSheetRefusesRowsWhileAParallelWriteRuns() + { + CancellationToken ct = TestContext.Current.CancellationToken; + var ms = new MemoryStream(); + await using XlsxWorkbookWriter wb = XlsxWorkbookWriter.Create(ms, leaveOpen: true); + XlsxSheetWriter sheet = wb.AddSheet("Data"); + InvalidOperationException? seen = null; + IEnumerable Source() + { + foreach (Item item in Items(50)) + { + if (item.Id == 20) + { + seen = Assert.Throws(() => sheet.StartRow()); + } + yield return item; + } + } + + await sheet.WriteRecordsParallelAsync(Source(), Fill, Parallel(2, 10), ct); + + Assert.NotNull(seen); + await sheet.EndAsync(ct); + await wb.EndAsync(ct); + } + + [Fact] + public async Task InvalidArgumentsThrowBeforeAnythingIsWritten() + { + CancellationToken ct = TestContext.Current.CancellationToken; + var ms = new MemoryStream(); + await using XlsxWorkbookWriter wb = XlsxWorkbookWriter.Create(ms, leaveOpen: true); + XlsxSheetWriter sheet = wb.AddSheet("Data"); + List items = Items(3); + + await Assert.ThrowsAsync(async () => + await sheet.WriteRecordsParallelAsync(items, ExcelRecordLayout.FromAttributes(), new ParallelWriteOptions { BatchRows = 0 }, ct)); + await Assert.ThrowsAsync(async () => + await sheet.WriteRecordsParallelAsync((IEnumerable)null!, Fill, ct: ct)); + await sheet.WriteRecordsParallelAsync(items, Fill, ct: ct); + await sheet.EndAsync(ct); + await wb.EndAsync(ct); + + Assert.Equal(3, Read(ms.ToArray()).Count); + } + + [Fact] + public async Task ABatchSizeBeyondTheRowLimitStillWrites() + { + CancellationToken ct = TestContext.Current.CancellationToken; + List items = Items(7); + byte[] package = await WriteAsync(XlsxWriterOptions.Default, async sheet => + await sheet.WriteRecordsParallelAsync(items, Fill, Parallel(2, int.MaxValue), ct)); + + Assert.Equal(7, Read(package).Count); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task AFailingOutputSurfacesItsOwnException(bool prefetch) + { + CancellationToken ct = TestContext.Current.CancellationToken; + List items = Items(20_000); + using var target = new FailingWriteStream(limit: 4096); + XlsxWorkbookWriter wb = XlsxWorkbookWriter.Create(target, leaveOpen: true, new XlsxWriterOptions { PrefetchWrite = prefetch }); + XlsxSheetWriter sheet = wb.AddSheet("Data"); + + IOException failure = await Assert.ThrowsAsync( + async () => await sheet.WriteRecordsParallelAsync(items, Fill, Parallel(4), ct)); + + Assert.Equal("disk full", failure.Message); + Assert.True(sheet.ResourcesReleased); + await wb.DisposeAsync(); + } + } +} diff --git a/tests/ExcelReader.Tests/Writer/Parallel/ParallelXlsbRecordWriteTests.cs b/tests/ExcelReader.Tests/Writer/Parallel/ParallelXlsbRecordWriteTests.cs new file mode 100644 index 00000000..693f117a --- /dev/null +++ b/tests/ExcelReader.Tests/Writer/Parallel/ParallelXlsbRecordWriteTests.cs @@ -0,0 +1,230 @@ +using System.Globalization; +using System.IO.Compression; +using ExcelReader.Core.Reader; +using ExcelReader.Core.Reader.Xlsb; +using ExcelReader.Core.Writer; +using ExcelReader.Core.Writer.Xlsb; +using ExcelReader.Tests.Writer.Zip; +using Item = ExcelReader.Tests.Writer.Parallel.ParallelRecordWriteTests.Item; + +namespace ExcelReader.Tests.Writer.Parallel +{ + public class ParallelXlsbRecordWriteTests + { + public static TheoryData Modes() + { + return ParallelRecordWriteTests.Modes(); + } + + private static async Task WriteAsync(XlsbWriterOptions options, Func body, bool seekable = true) + { + var ms = new MemoryStream(); + Stream target = seekable ? ms : new ForwardOnlyWriteStream(ms); + await using (XlsbWorkbookWriter wb = XlsbWorkbookWriter.Create(target, leaveOpen: true, options)) + { + XlsbSheetWriter sheet = wb.AddSheet("Data"); + await body(sheet); + await sheet.EndAsync(TestContext.Current.CancellationToken); + await wb.EndAsync(TestContext.Current.CancellationToken); + } + return ms.ToArray(); + } + + private static ParallelWriteOptions Parallel(int degree, int batchRows = 500) + { + return new ParallelWriteOptions { DegreeOfParallelism = degree, BatchRows = batchRows }; + } + + private static List Read(byte[] package) + { + using XlsbWorkbook workbook = Excel.FromXlsb(new MemoryStream(package)); + return ParallelRecordWriteTests.Cells(workbook); + } + + private static void Fill(XlsbRowWriter row, Item item) + { + ParallelRecordWriteTests.Fill(row, item); + } + + [Theory] + [MemberData(nameof(Modes))] + public async Task ParallelGivesTheSameCellsAsSerial(CompressionLevel level, bool prefetch, bool sharedStrings) + { + CancellationToken ct = TestContext.Current.CancellationToken; + List items = ParallelRecordWriteTests.Items(6_300); + var options = new XlsbWriterOptions { Compression = level, PrefetchWrite = prefetch, UseSharedStrings = sharedStrings }; + byte[] serial = await WriteAsync(options, async sheet => await sheet.WriteRecordsAsync(items, Fill, ct)); + foreach (bool seekable in new[] { true, false }) + { + byte[] parallel = await WriteAsync(options, async sheet => + await sheet.WriteRecordsParallelAsync(items, Fill, Parallel(4), ct), seekable); + + Assert.Equal(Read(serial), Read(parallel)); + ParallelRecordWriteTests.Entry(parallel, "xl/worksheets/sheet1.bin"); + } + } + + [Theory] + [InlineData(0)] + [InlineData(1)] + [InlineData(500)] + [InlineData(501)] + [InlineData(1_733)] + public async Task BatchEdges(int count) + { + CancellationToken ct = TestContext.Current.CancellationToken; + List items = ParallelRecordWriteTests.Items(count); + byte[] package = await WriteAsync(XlsbWriterOptions.Default, async sheet => + await sheet.WriteRecordsParallelAsync(items, Fill, Parallel(3), ct)); + + List rows = Read(package); + Assert.Equal(count, rows.Count); + for (int i = 0; i < count; i++) + { + Assert.Equal(i.ToString(CultureInfo.InvariantCulture), rows[i][0]); + Assert.Equal(items[i].Name, rows[i][1]); + } + } + + [Fact] + public async Task ParallelAsTheFirstWriteOnTheSheet() + { + CancellationToken ct = TestContext.Current.CancellationToken; + List items = ParallelRecordWriteTests.Items(1_200); + byte[] package = await WriteAsync(XlsbWriterOptions.Default with { PrefetchWrite = true }, async sheet => + await sheet.WriteRecordsParallelAsync(items, Fill, Parallel(4), ct)); + + Assert.Equal(1_200, Read(package).Count); + } + + [Theory] + [InlineData(CompressionLevel.Fastest, false)] + [InlineData(CompressionLevel.Optimal, false)] + [InlineData(CompressionLevel.NoCompression, false)] + [InlineData(CompressionLevel.Fastest, true)] + [InlineData(CompressionLevel.Optimal, true)] + [InlineData(CompressionLevel.NoCompression, true)] + public async Task SerialRowsBeforeAndAfterSurvive(CompressionLevel level, bool prefetch) + { + CancellationToken ct = TestContext.Current.CancellationToken; + List items = ParallelRecordWriteTests.Items(1_300); + byte[] package = await WriteAsync(new XlsbWriterOptions { Compression = level, PrefetchWrite = prefetch }, async sheet => + { + await sheet.WriteRecordsAsync(items, Fill, ct); + await sheet.WriteRecordsParallelAsync(items, Fill, Parallel(4), ct); + await sheet.WriteRecordsAsync(items, Fill, ct); + await sheet.WriteRecordsParallelAsync(items, Fill, Parallel(2), ct); + using XlsbRowWriter last = sheet.StartRow(); + last.Write("end"); + }); + + List rows = Read(package); + Assert.Equal((4 * 1_300) + 1, rows.Count); + for (int block = 0; block < 4; block++) + { + Assert.Equal("0", rows[block * 1_300][0]); + Assert.Equal("1299", rows[(block * 1_300) + 1_299][0]); + } + Assert.Equal("end", rows[^1][0]); + } + + [Fact] + public async Task ALayoutAndATableMatchSerial() + { + CancellationToken ct = TestContext.Current.CancellationToken; + List items = ParallelRecordWriteTests.Items(900); + byte[] serial = await WriteAsync(XlsbWriterOptions.Default, async sheet => + await sheet.WriteTableAsync(items, "Records", ExcelRecordLayout.FromAttributes(), ct: ct)); + byte[] parallel = await WriteAsync(XlsbWriterOptions.Default, async sheet => + await sheet.WriteTableParallelAsync(items, "Records", ExcelRecordLayout.FromAttributes(), options: Parallel(4, 100), ct: ct)); + + Assert.Equal(Read(serial), Read(parallel)); + Assert.Equal(ParallelRecordWriteTests.Entry(serial, "xl/tables/table1.bin"), ParallelRecordWriteTests.Entry(parallel, "xl/tables/table1.bin")); + } + + [Fact] + public async Task SharedStringsGiveEveryCellItsOwnText() + { + CancellationToken ct = TestContext.Current.CancellationToken; + List items = ParallelRecordWriteTests.Items(4_000, distinctNames: 613); + for (int round = 0; round < 25; round++) + { + byte[] package = await WriteAsync(new XlsbWriterOptions { UseSharedStrings = true }, async sheet => + await sheet.WriteRecordsParallelAsync(items, Fill, Parallel(8, 50), ct)); + + List rows = Read(package); + Assert.Equal(items.Count, rows.Count); + for (int i = 0; i < items.Count; i++) + { + Assert.Equal(items[i].Name, rows[i][1]); + } + Assert.Equal((4_000u, 613u), SharedStringCounts(package)); + } + } + + // BrtBeginSst opens the part: a two-byte record id, a one-byte size, then the reference count + // and the unique count as 32-bit integers. + private static (uint References, uint Unique) SharedStringCounts(byte[] package) + { + byte[] part = ParallelRecordWriteTests.Entry(package, "xl/sharedStrings.bin"); + return (BitConverter.ToUInt32(part, 3), BitConverter.ToUInt32(part, 7)); + } + + [Fact] + public async Task SharedStringCountsMatchASerialWrite() + { + CancellationToken ct = TestContext.Current.CancellationToken; + List items = ParallelRecordWriteTests.Items(4_000, distinctNames: 613); + var options = new XlsbWriterOptions { UseSharedStrings = true }; + byte[] serial = await WriteAsync(options, async sheet => await sheet.WriteRecordsAsync(items, Fill, ct)); + byte[] parallel = await WriteAsync(options, async sheet => + await sheet.WriteRecordsParallelAsync(items, Fill, Parallel(8, 50), ct)); + + Assert.Equal(SharedStringCounts(serial), SharedStringCounts(parallel)); + } + + [Fact] + public async Task AThrowingRowFaultsTheSheet() + { + CancellationToken ct = TestContext.Current.CancellationToken; + List items = ParallelRecordWriteTests.Items(5_000); + static void Throwing(XlsbRowWriter row, Item item) + { + row.Write(item.Id); + if (item.Id == 2_345) + { + throw new FormatException("row 2345"); + } + } + var ms = new MemoryStream(); + XlsbWorkbookWriter wb = XlsbWorkbookWriter.Create(ms, leaveOpen: true); + XlsbSheetWriter sheet = wb.AddSheet("Data"); + + await Assert.ThrowsAsync( + async () => await sheet.WriteRecordsParallelAsync(items, Throwing, Parallel(4, 100), ct)); + + Assert.True(sheet.ResourcesReleased); + Assert.Throws(() => sheet.StartRow()); + await wb.DisposeAsync(); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task AFailingOutputSurfacesItsOwnException(bool prefetch) + { + CancellationToken ct = TestContext.Current.CancellationToken; + List items = ParallelRecordWriteTests.Items(20_000); + using var target = new FailingWriteStream(limit: 4096); + XlsbWorkbookWriter wb = XlsbWorkbookWriter.Create(target, leaveOpen: true, new XlsbWriterOptions { PrefetchWrite = prefetch }); + XlsbSheetWriter sheet = wb.AddSheet("Data"); + + IOException failure = await Assert.ThrowsAsync( + async () => await sheet.WriteRecordsParallelAsync(items, Fill, Parallel(4), ct)); + + Assert.Equal("disk full", failure.Message); + Assert.True(sheet.ResourcesReleased); + await wb.DisposeAsync(); + } + } +} diff --git a/tests/ExcelReader.Tests/Writer/WriteOffloadDrainTests.cs b/tests/ExcelReader.Tests/Writer/WriteOffloadDrainTests.cs new file mode 100644 index 00000000..6c0097d3 --- /dev/null +++ b/tests/ExcelReader.Tests/Writer/WriteOffloadDrainTests.cs @@ -0,0 +1,60 @@ +using ExcelReader.Core.Writer.Internal; +using ExcelReader.Tests.Writer.Zip; + +namespace ExcelReader.Tests.Writer +{ + public class WriteOffloadDrainTests + { + private sealed class SlowStream : MemoryStream + { + public override async ValueTask WriteAsync(ReadOnlyMemory buffer, CancellationToken cancellationToken = default) + { + await Task.Delay(15, cancellationToken); + await base.WriteAsync(buffer, cancellationToken); + } + } + + [Fact] + public async Task DrainWaitsForEverythingWrittenBefore() + { + CancellationToken ct = TestContext.Current.CancellationToken; + var inner = new SlowStream(); + await using var offload = new WriteOffloadStream(inner); + for (int i = 0; i < 6; i++) + { + await offload.WriteAsync(new byte[] { (byte)i, (byte)i }, ct); + } + + await offload.DrainAsync(ct); + + Assert.Equal([0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5], inner.ToArray()); + } + + [Fact] + public async Task TheStreamStaysWritableAfterADrain() + { + CancellationToken ct = TestContext.Current.CancellationToken; + var inner = new MemoryStream(); + var offload = new WriteOffloadStream(inner); + await offload.WriteAsync("ab"u8.ToArray(), ct); + await offload.DrainAsync(ct); + await offload.DrainAsync(ct); + await offload.WriteAsync("cd"u8.ToArray(), ct); + await offload.FlushAsync(ct); + + Assert.Equal("abcd"u8.ToArray(), inner.ToArray()); + await offload.DisposeAsync(); + } + + [Fact] + public async Task DrainSurfacesAConsumerFailureWithItsType() + { + CancellationToken ct = TestContext.Current.CancellationToken; + var offload = new WriteOffloadStream(new FailingWriteStream(0)); + await offload.WriteAsync("ab"u8.ToArray(), ct); + + await Assert.ThrowsAsync(async () => await offload.DrainAsync(ct)); + await Assert.ThrowsAsync(async () => await offload.DisposeAsync()); + } + } +} diff --git a/tests/ExcelReader.Tests/Writer/WriterEdgeValidationTests.cs b/tests/ExcelReader.Tests/Writer/WriterEdgeValidationTests.cs new file mode 100644 index 00000000..a2a0ba7d --- /dev/null +++ b/tests/ExcelReader.Tests/Writer/WriterEdgeValidationTests.cs @@ -0,0 +1,250 @@ +using System.IO.Compression; +using System.Text; +using ExcelReader.Core.Reader; +using ExcelReader.Core.Writer; +using ExcelReader.Core.Writer.Xls; +using ExcelReader.Core.Writer.Xlsb; +using ExcelReader.Core.Writer.Xlsx; + +namespace ExcelReader.Tests.Writer +{ + /// + /// Inputs the writers used to accept and Excel then refused or altered. Each was confirmed against + /// Excel 16; these tests pin the writer side. + /// + public sealed class WriterEdgeValidationTests + { + public static TheoryData Formats => ["xlsx", "xlsb", "xls"]; + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void XlsxEscapesCarriageReturnsAndNonCharacters(bool sharedStrings) + { + string[] texts = ["a\rb", "a\r\nb", "\r", "a￾b", "a￿b", "plain"]; + using MemoryStream ms = new(); + using (XlsxWorkbookWriter wb = XlsxWorkbookWriter.Create(ms, leaveOpen: true, new XlsxWriterOptions { UseSharedStrings = sharedStrings })) + { + XlsxSheetWriter sheet = wb.AddSheet("S1"); + using (XlsxRowWriter row = sheet.StartRow()) + { + foreach (string text in texts) + { + row.Write(text); + } + row.WriteUtf8(Encoding.UTF8.GetBytes("u￾v")); + } + sheet.End(); + wb.End(); + } + + byte[] xml = Entry(ms.ToArray(), sharedStrings ? "xl/sharedStrings.xml" : "xl/worksheets/sheet1.xml"); + Assert.DoesNotContain((byte)'\r', xml); + Assert.Equal(-1, xml.AsSpan().IndexOf("￾"u8)); + Assert.Equal(-1, xml.AsSpan().IndexOf("￿"u8)); + Assert.True(xml.AsSpan().IndexOf("a_x000D_b"u8) >= 0); + Assert.True(xml.AsSpan().IndexOf("a_xFFFE_b"u8) >= 0); + Assert.True(xml.AsSpan().IndexOf("a_xFFFF_b"u8) >= 0); + + using IExcelWorkbook read = Excel.Open(ms.ToArray()); + using IExcelRowEnumerator rows = read.FirstSheet.GetEnumerator(); + Assert.True(rows.MoveNext()); + for (int i = 0; i < texts.Length; i++) + { + Assert.Equal(texts[i], rows.Current[i].GetString()); + } + Assert.Equal("u￾v", rows.Current[texts.Length].GetString()); + } + + [Theory] + [MemberData(nameof(Formats))] + public void SheetNamesExcelRefusesAreRejected(string format) + { + foreach (string name in new[] { "'a", "a'", "'", "a\u0001b", "a\tb", "a\nb", "a￾b", "a￿b" }) + { + Assert.Throws(() => WithWorkbook(format, wb => wb.AddSheet(name))); + } + WithWorkbook(format, wb => wb.AddSheet("it's")); + } + + [Theory] + [MemberData(nameof(Formats))] + public void ColumnSettingsBeyondTheFormatAreRejected(string format) + { + int columns = string.Equals(format, "xls", StringComparison.Ordinal) ? 256 : 16_384; + Assert.Throws(() => WithWorkbook(format, wb => wb.Width(columns, 10))); + Assert.Throws(() => WithWorkbook(format, wb => wb.Style(columns))); + Assert.Throws(() => WithWorkbook(format, wb => wb.Width(0, 255.5))); + Assert.Throws(() => WithWorkbook(format, wb => wb.Width(0, double.PositiveInfinity))); + WithWorkbook(format, wb => + { + wb.Width(columns - 1, 255); + wb.Style(columns - 1); + }); + } + + [Theory] + [MemberData(nameof(Formats))] + public void NumberFormatsStopAtWhatExcelOpens(string format) + { + WithWorkbook(format, wb => + { + for (int i = 0; i < 200; i++) + { + wb.AddStyle(new CellStyle { NumberFormat = $"0.00\"u{i}\"" }); + } + wb.AddStyle(new CellStyle { NumberFormat = "0.00\"u7\"", Bold = true }); + ExcelLimitExceededException error = Assert.Throws( + () => wb.AddStyle(new CellStyle { NumberFormat = "0.000" })); + Assert.Equal(200, error.Limit); + wb.AddSheet("S1"); + }); + Assert.Throws( + () => WithWorkbook(format, wb => wb.AddStyle(new CellStyle { NumberFormat = new string('0', 256) }))); + WithWorkbook(format, wb => + { + wb.AddStyle(new CellStyle { NumberFormat = new string('0', 255) }); + wb.AddSheet("S1"); + }); + } + + [Fact] + public void AValueWithNoNumericFormIsRefusedTheSameWayEverywhere() + { + using MemoryStream ms = new(); + using (XlsxWorkbookWriter wb = XlsxWorkbookWriter.Create(ms, leaveOpen: true)) + { + XlsxSheetWriter sheet = wb.AddSheet("S1"); + using (XlsxRowWriter row = sheet.StartRow()) + { + Assert.Throws(() => row.Write('c')); + } + sheet.End(); + } + using (XlsbWorkbookWriter wb = XlsbWorkbookWriter.Create(ms, leaveOpen: true)) + { + XlsbSheetWriter sheet = wb.AddSheet("S1"); + using (XlsbRowWriter row = sheet.StartRow()) + { + Assert.Throws(() => row.Write('c')); + } + sheet.End(); + } + using (XlsWorkbookWriter wb = XlsWorkbookWriter.Create(ms, leaveOpen: true)) + { + XlsSheetWriter sheet = wb.AddSheet("S1"); + using (XlsRowWriter row = sheet.StartRow()) + { + Assert.Throws(() => row.Write('c')); + } + sheet.End(); + } + } + + [Fact] + public void ALimitMessageNamesNoUnit() + { + var error = new ExcelLimitExceededException("Rows", 1_048_576, 1_048_577); + + Assert.Equal("Rows limit exceeded: 1048577 exceeds 1048576.", error.Message); + } + + private static void WithWorkbook(string format, Action body) + { + using MemoryStream ms = new(); + Probe probe = format switch + { + "xlsx" => Probe.Of(XlsxWorkbookWriter.Create(ms, leaveOpen: true)), + "xlsb" => Probe.Of(XlsbWorkbookWriter.Create(ms, leaveOpen: true)), + _ => Probe.Of(XlsWorkbookWriter.Create(ms, leaveOpen: true)), + }; + try + { + body(probe); + } + finally + { + probe.Abandon(); + } + } + + private static byte[] Entry(byte[] package, string name) + { + using var zip = new ZipArchive(new MemoryStream(package), ZipArchiveMode.Read); + using Stream data = zip.GetEntry(name)!.Open(); + using var copy = new MemoryStream(); + data.CopyTo(copy); + return copy.ToArray(); + } + + /// The few workbook calls these tests make, over any of the three writers. + private sealed class Probe + { + private Action _addSheet = static _ => { }; + private Action _width = static (_, _) => { }; + private Action _style = static _ => { }; + private Func _addStyle = static _ => 0; + private Action _abandon = static () => { }; + + internal static Probe Of(TWorkbook workbook) + where TWorkbook : IWorkbookWriter, IDisposable + where TSheet : ISheetWriter + where TRow : IRowWriter + { + return new Probe + { + _addSheet = name => workbook.AddSheet(name).End(), + _width = (column, width) => + { + TSheet sheet = workbook.AddSheet("S1"); + sheet.SetColumnWidth(column, width); + sheet.End(); + }, + _style = column => + { + TSheet sheet = workbook.AddSheet("S2"); + sheet.SetColumnStyle(column, 1); + sheet.End(); + }, + _addStyle = workbook.AddStyle, + _abandon = () => + { + try + { + workbook.Dispose(); + } + catch (InvalidOperationException) + { + // A workbook left without a finished sheet refuses to end; the probe only checks arguments. + } + }, + }; + } + + internal void AddSheet(string name) + { + _addSheet(name); + } + + internal void Width(int column, double width) + { + _width(column, width); + } + + internal void Style(int column) + { + _style(column); + } + + internal int AddStyle(CellStyle style) + { + return _addStyle(style); + } + + internal void Abandon() + { + _abandon(); + } + } + } +} diff --git a/tests/ExcelReader.Tests/Writer/Xls/XlsConformanceTests.cs b/tests/ExcelReader.Tests/Writer/Xls/XlsConformanceTests.cs new file mode 100644 index 00000000..40dc4248 --- /dev/null +++ b/tests/ExcelReader.Tests/Writer/Xls/XlsConformanceTests.cs @@ -0,0 +1,299 @@ +using System.Buffers.Binary; +using System.Diagnostics.CodeAnalysis; +using ExcelReader.Core.Crypto; +using ExcelReader.Core.Reader; +using ExcelReader.Core.Writer.Xls; + +namespace ExcelReader.Tests.Writer.Xls +{ + /// + /// Excel refuses a BIFF8 file whose records break the [MS-XLS] grammar, while this library's own + /// reader accepts it, so a round trip proves nothing here. These pin the structure Excel checks. + /// + public class XlsConformanceTests + { + private const int Bof = 0x0809; + private const int Eof = 0x000A; + private const int Index = 0x020B; + private const int DefColWidth = 0x0055; + private const int Row = 0x0208; + private const int DbCell = 0x00D7; + private const int Window1 = 0x003D; + private const int Window2 = 0x023E; + private const int WriteAccess = 0x005C; + private const int Date1904 = 0x0022; + private const int Label = 0x0204; + private const int LabelSst = 0x00FD; + private const int Sst = 0x00FC; + private const int ExtSst = 0x00FF; + private const int Rk = 0x027E; + private const int MulRk = 0x00BD; + private const int Number = 0x0203; + + private readonly record struct Record(int Position, int Id, byte[] Payload); + + private static byte[] Write(Action build) + { + var ms = new MemoryStream(); + using (var wb = XlsWorkbookWriter.Create(ms, leaveOpen: true)) + { + build(wb); + wb.End(); + } + return ms.ToArray(); + } + + private static List Records(byte[] xls) + { + using var cfb = CfbContainer.Parse(new MemoryStream(xls), ownsSource: true, ExcelReaderOptions.Default); + byte[] stream = cfb.ReadStream("Workbook", maxBytes: 64 * 1024 * 1024); + var records = new List(); + int pos = 0; + while (pos + 4 <= stream.Length) + { + int id = BinaryPrimitives.ReadUInt16LittleEndian(stream.AsSpan(pos)); + if (id == 0) + { + break; + } + int length = BinaryPrimitives.ReadUInt16LittleEndian(stream.AsSpan(pos + 2)); + records.Add(new Record(pos, id, stream.AsSpan(pos + 4, length).ToArray())); + pos += 4 + length; + } + return records; + } + + private static List Worksheet(List records) + { + int start = records.FindLastIndex(r => r.Id == Bof); + return records.GetRange(start, records.Count - start); + } + + [Fact] + public void WorksheetSubstreamIndexesItsRowBlocks() + { + byte[] xls = Write(wb => + { + using var sheet = wb.AddSheet("S"); + for (int i = 0; i < 40; i++) + { + using var row = sheet.StartRow(); + row.Write(i); + row.Write("text"); + } + }); + + List sheet = Worksheet(Records(xls)); + Dictionary byPosition = sheet.ToDictionary(r => r.Position); + + Assert.Equal(Bof, sheet[0].Id); + Assert.Equal(Index, sheet[1].Id); + Assert.Equal(Eof, sheet[^1].Id); + Assert.Equal(Window2, sheet[^2].Id); + + byte[] index = sheet[1].Payload; + Assert.Equal(40, BinaryPrimitives.ReadInt32LittleEndian(index.AsSpan(8))); + Assert.Equal(DefColWidth, byPosition[BinaryPrimitives.ReadInt32LittleEndian(index.AsSpan(12))].Id); + + int blocks = (index.Length - 16) / 4; + Assert.Equal(2, blocks); + for (int block = 0; block < blocks; block++) + { + Record dbCell = byPosition[BinaryPrimitives.ReadInt32LittleEndian(index.AsSpan(16 + (4 * block)))]; + Assert.Equal(DbCell, dbCell.Id); + int firstRow = dbCell.Position - BinaryPrimitives.ReadInt32LittleEndian(dbCell.Payload); + Assert.Equal(Row, byPosition[firstRow].Id); + Assert.Equal(block * 32, BinaryPrimitives.ReadUInt16LittleEndian(byPosition[firstRow].Payload)); + } + + Assert.Equal(40, sheet.Count(r => r.Id == Row)); + Assert.True(sheet.FindIndex(r => r.Id == Window2) > sheet.FindLastIndex(r => r.Id == DbCell)); + } + + [Fact] + public void GlobalsRecordsHaveTheirSpecifiedShape() + { + byte[] xls = Write(wb => + { + using var sheet = wb.AddSheet("S"); + using var row = sheet.StartRow(); + row.Write(1); + }); + + List records = Records(xls); + Assert.Equal(0x07CC, BinaryPrimitives.ReadUInt16LittleEndian(records[0].Payload.AsSpan(6))); + + byte[] writeAccess = records.Single(r => r.Id == WriteAccess).Payload; + Assert.Equal(112, writeAccess.Length); + Assert.Equal("ExcelReader".Length, BinaryPrimitives.ReadUInt16LittleEndian(writeAccess)); + Assert.Equal(0, writeAccess[2]); + + Assert.Equal(18, records.Single(r => r.Id == Window1).Payload.Length); + Assert.Equal([0, 0], records.Single(r => r.Id == Date1904).Payload); + } + + [Theory] + [InlineData(0.0, true)] + [InlineData(42.0, true)] + [InlineData(-536870912.0, true)] + [InlineData(536870912.0, true)] + [InlineData(1.5, true)] + [InlineData(0.1, true)] + [InlineData(45292.25, true)] + [InlineData(1234.56, true)] + [InlineData(0.5, true)] + [InlineData(Math.PI, false)] + [InlineData(1e-300, false)] + [InlineData(123456789.123, false)] + public void RkEncodesOnlyNumbersItGivesBackUnchanged(double value, bool encodable) + { + Assert.Equal(encodable, XlsRk.TryEncode(value, out int rk)); + if (encodable) + { + Assert.Equal(BitConverter.DoubleToInt64Bits(value), BitConverter.DoubleToInt64Bits(XlsRk.Decode(rk))); + } + } + + [Fact] + [SuppressMessage("Security", "CA5394:Do not use insecure randomness", + Justification = "A seeded generator picks reproducible test inputs; nothing here is secret.")] + public void RkNeverChangesAValue() + { + Assert.True(XlsRk.TryEncode(-0.0, out int negativeZero)); + Assert.Equal(BitConverter.DoubleToInt64Bits(-0.0), BitConverter.DoubleToInt64Bits(XlsRk.Decode(negativeZero))); + + var random = new Random(1234); + for (int i = 0; i < 100_000; i++) + { + double value = (i % 4) switch + { + 0 => random.Next(-1_000_000_000, 1_000_000_000), + 1 => Math.Round(random.NextDouble() * 1e7, 2), + 2 => random.NextDouble() * Math.Pow(10, random.Next(-20, 20)), + _ => BitConverter.Int64BitsToDouble(random.NextInt64() & ~0x3_FFFF_FFFFL), + }; + if (double.IsFinite(value) && XlsRk.TryEncode(value, out int rk)) + { + Assert.Equal(BitConverter.DoubleToInt64Bits(value), BitConverter.DoubleToInt64Bits(XlsRk.Decode(rk))); + } + } + } + + [Fact] + public void ConsecutiveNumbersShareOneMulRkRecord() + { + byte[] xls = Write(wb => + { + using var sheet = wb.AddSheet("S"); + using var row = sheet.StartRow(); + row.Write(1); + row.Write(2.5); + row.Write(new DateTime(2026, 1, 2)); + row.Write("x"); + row.Write(3); + row.Skip(1); + row.Write(4); + row.Write(Math.PI); + }); + + List cells = Worksheet(Records(xls)) + .Where(r => r.Id is MulRk or Rk or Number or Label) + .Select(r => r.Id) + .ToList(); + Assert.Equal([MulRk, Label, Rk, Rk, Number], cells); + + using var reader = Excel.FromXls(new MemoryStream(xls)); + using var rows = reader.FirstSheet.GetEnumerator(); + Assert.True(rows.MoveNext()); + Row read = rows.Current; + Assert.Equal("1", read[0].GetString()); + Assert.Equal("2.5", read[1].GetString()); + Assert.Equal(CellType.Date, read[2].Type); + Assert.True(read[2].TryGetDateTime(isDate1904: false, out DateTime date)); + Assert.Equal(new DateTime(2026, 1, 2), date); + Assert.Equal("x", read[3].GetString()); + Assert.Equal("3", read[4].GetString()); + Assert.Equal(CellType.Empty, read[5].Type); + Assert.Equal("4", read[6].GetString()); + Assert.True(read[7].TryParse(System.Globalization.CultureInfo.InvariantCulture, out double pi)); + Assert.Equal(Math.PI, pi); + } + + [Fact] + public void LongTextDifferingOnlyBetweenHashSamplesIsStillStoredOnce() + { + static string Text(int i) + { + return new string('a', 97) + i.ToString("D6", System.Globalization.CultureInfo.InvariantCulture) + new string('b', 197); + } + + const int distinct = 200; + byte[] xls = Write(wb => + { + using var sheet = wb.AddSheet("S"); + for (int i = 0; i < 2 * distinct; i++) + { + using var row = sheet.StartRow(); + row.Write(Text(i % distinct)); + } + }); + + byte[] sst = Records(xls).Single(r => r.Id == Sst).Payload; + Assert.Equal(2 * distinct, BinaryPrimitives.ReadInt32LittleEndian(sst)); + Assert.Equal(distinct, BinaryPrimitives.ReadInt32LittleEndian(sst.AsSpan(4))); + + using var reader = Excel.FromXls(new MemoryStream(xls)); + using var rows = reader.FirstSheet.GetEnumerator(); + for (int i = 0; i < 2 * distinct; i++) + { + Assert.True(rows.MoveNext()); + Assert.Equal(Text(i % distinct), rows.Current[0].GetString()); + } + Assert.False(rows.MoveNext()); + } + + [Theory] + [InlineData(255, 'x', Label)] + [InlineData(256, 'x', LabelSst)] + [InlineData(20_000, 'x', LabelSst)] + [InlineData(9_000, 'é', LabelSst)] + [InlineData(5_000, '日', LabelSst)] + public void TextKeepsItsFullLength(int length, char fill, int expectedRecord) + { + string text = new(fill, length); + byte[] xls = Write(wb => + { + using var sheet = wb.AddSheet("S"); + using (var row = sheet.StartRow()) + { + row.Write(text); + row.Write(text); + } + using (var row = sheet.StartRow()) + { + row.Write("short"); + } + }); + + List records = Records(xls); + // "short" is a LABEL too. + int expectedCount = expectedRecord == Label ? 3 : 2; + Assert.Equal(expectedCount, Worksheet(records).Count(r => r.Id == expectedRecord)); + if (expectedRecord == LabelSst) + { + Assert.Single(records, r => r.Id == Sst); + Assert.Single(records, r => r.Id == ExtSst); + Assert.Equal([2, 0, 0, 0, 1, 0, 0, 0], records.Single(r => r.Id == Sst).Payload.AsSpan(0, 8).ToArray()); + } + + using var reader = Excel.FromXls(new MemoryStream(xls)); + using var rows = reader.FirstSheet.GetEnumerator(); + Assert.True(rows.MoveNext()); + Assert.Equal(text, rows.Current[0].GetString()); + Assert.Equal(text, rows.Current[1].GetString()); + Assert.True(rows.MoveNext()); + Assert.Equal("short", rows.Current[0].GetString()); + Assert.False(rows.MoveNext()); + } + } +} diff --git a/tests/ExcelReader.Tests/Writer/Xls/XlsWriterTests.cs b/tests/ExcelReader.Tests/Writer/Xls/XlsWriterTests.cs index 7fa3c62d..dfa1087d 100644 --- a/tests/ExcelReader.Tests/Writer/Xls/XlsWriterTests.cs +++ b/tests/ExcelReader.Tests/Writer/Xls/XlsWriterTests.cs @@ -342,7 +342,8 @@ public async Task LargeWorkbookForcesDifatSectors() r.Write(i); for (int c = 1; c < cols; c++) { - r.Write(i + (c * 0.5)); + // Thirds fit no RK form, so each cell stays an 8-byte NUMBER. + r.Write(i + (c / 3.0)); } } s.End(); @@ -360,7 +361,7 @@ public async Task LargeWorkbookForcesDifatSectors() Assert.True(e.Current[0].TryParse(Inv, out int first)); Assert.Equal(read, first); Assert.True(e.Current[cols - 1].TryParse(Inv, out double last)); - Assert.Equal(read + ((cols - 1) * 0.5), last); + Assert.Equal(read + ((cols - 1) / 3.0), last); } read++; } diff --git a/tests/ExcelReader.Tests/Writer/Xlsb/XlsbWriterTests.cs b/tests/ExcelReader.Tests/Writer/Xlsb/XlsbWriterTests.cs index 1e21e32c..b9dadaea 100644 --- a/tests/ExcelReader.Tests/Writer/Xlsb/XlsbWriterTests.cs +++ b/tests/ExcelReader.Tests/Writer/Xlsb/XlsbWriterTests.cs @@ -375,6 +375,20 @@ public async Task SkipBeyondColumnLimitThrows() Assert.Throws(() => row.Skip(16_385)); } + [Fact] + public void ARowBeyondTheRowLimitThrows() + { + using XlsbWorkbookWriter wb = XlsbWorkbookWriter.Create(Stream.Null, leaveOpen: true); + XlsbSheetWriter sheet = wb.AddSheet("Sheet1"); + for (int i = 0; i < 1_048_576; i++) + { + sheet.StartRow().Dispose(); + } + + ExcelLimitExceededException error = Assert.Throws(() => sheet.StartRow()); + Assert.Equal(1_048_577, error.Actual); + } + [Fact] public async Task WriteNonNumericFormattableThrowsArgumentException() { @@ -406,7 +420,11 @@ public async Task WriteOverflowingFormattableThrowsNamingTheSourceType() [InlineData(-1d)] [InlineData(536_870_911d)] [InlineData(-536_870_912d)] - public async Task IntegralNumberInRkRangeUsesFourByteRkCell(double value) + [InlineData(536_870_912d)] + [InlineData(0.5d)] + [InlineData(1234.56d)] + [InlineData(45_292.25d)] + public async Task NumberRkGivesBackExactlyUsesFourByteRkCell(double value) { (int Id, int PayloadLength)[] cells; double readBack; @@ -417,11 +435,11 @@ public async Task IntegralNumberInRkRangeUsesFourByteRkCell(double value) } [Theory] - [InlineData(536_870_912d)] [InlineData(-536_870_913d)] - [InlineData(0.5d)] + [InlineData(Math.PI)] + [InlineData(123_456_789.123d)] [InlineData(1e300d)] - public async Task NumberOutsideRkRangeKeepsEightByteRealCell(double value) + public async Task NumberRkWouldChangeKeepsEightByteRealCell(double value) { (int Id, int PayloadLength)[] cells; double readBack; @@ -432,13 +450,13 @@ public async Task NumberOutsideRkRangeKeepsEightByteRealCell(double value) } [Fact] - public async Task NegativeZeroKeepsRealCellSoItsSignSurvives() + public async Task NegativeZeroKeepsItsSignInAnRkCell() { (int Id, int PayloadLength)[] cells; double readBack; (cells, readBack) = await NumericCellAsync(-0.0); - Assert.Equal([(Brt.CellReal, CellHeaderLength + 8)], cells); + Assert.Equal([(Brt.CellRk, CellHeaderLength + 4)], cells); Assert.True(double.IsNegative(readBack)); } diff --git a/tests/ExcelReader.Tests/Writer/Zip/Crc32Tests.cs b/tests/ExcelReader.Tests/Writer/Zip/Crc32Tests.cs new file mode 100644 index 00000000..d019d5c4 --- /dev/null +++ b/tests/ExcelReader.Tests/Writer/Zip/Crc32Tests.cs @@ -0,0 +1,139 @@ +using System.Diagnostics.CodeAnalysis; +using System.Runtime.Intrinsics.X86; +using ExcelReader.Core.Writer.Zip; +using ArmCrc32 = System.Runtime.Intrinsics.Arm.Crc32; + +namespace ExcelReader.Tests.Writer.Zip +{ + public class Crc32Tests + { + private delegate uint AppendFunction(uint crc, ReadOnlySpan data); + + internal static uint BitwiseCrc(ReadOnlySpan data) + { + uint register = uint.MaxValue; + foreach (byte value in data) + { + register ^= value; + for (int bit = 0; bit < 8; bit++) + { + uint mask = 0u - (register & 1); + register = (register >> 1) ^ (0xEDB8_8320 & mask); + } + } + return ~register; + } + + private static List<(string Name, AppendFunction Append)> AvailablePaths() + { + var paths = new List<(string Name, AppendFunction Append)> + { + ("dispatch", Crc32.Append), + ("table", Crc32.AppendTable), + }; + if (Pclmulqdq.IsSupported) + { + paths.Add(("fold128", Crc32.AppendFold128)); + } + if (Pclmulqdq.V256.IsSupported) + { + paths.Add(("fold256", Crc32.AppendFold256)); + } + if (ArmCrc32.Arm64.IsSupported) + { + paths.Add(("arm64", Crc32.AppendArm64)); + } + return paths; + } + + [Fact] + public void EveryPathGivesTheStandardCheckValue() + { + foreach ((string name, AppendFunction append) in AvailablePaths()) + { + uint actual = append(0, "123456789"u8); + Assert.True(actual == 0xCBF4_3926, $"{name}: {actual:X8}"); + } + } + + [Fact] + public void EmptyInputLeavesTheCrcUnchanged() + { + foreach ((string name, AppendFunction append) in AvailablePaths()) + { + uint actual = append(0x1234_5678, []); + Assert.True(actual == 0x1234_5678, $"{name}: {actual:X8}"); + } + } + + [Fact] + [SuppressMessage("Security", "CA5394:Do not use insecure randomness", + Justification = "A seeded generator picks reproducible test inputs; nothing here is secret.")] + public void EveryPathMatchesTheBitwiseDefinitionWholeAndSplit() + { + var random = new Random(5); + byte[] buffer = new byte[70_064]; + random.NextBytes(buffer); + List<(string Name, AppendFunction Append)> paths = AvailablePaths(); + for (int trial = 0; trial < 900; trial++) + { + int start = random.Next(0, 64); + int length = trial < 600 ? trial : random.Next(600, 70_000); + ReadOnlySpan slice = buffer.AsSpan(start, length); + uint expected = BitwiseCrc(slice); + int cut = random.Next(0, length + 1); + foreach ((string name, AppendFunction append) in paths) + { + uint whole = append(0, slice); + uint split = append(append(0, slice[..cut]), slice[cut..]); + Assert.True(whole == expected, $"{name} whole, length {length}: {whole:X8} != {expected:X8}"); + Assert.True(split == expected, $"{name} split at {cut} of {length}: {split:X8} != {expected:X8}"); + } + } + } + + [Fact] + [SuppressMessage("Security", "CA5394:Do not use insecure randomness", + Justification = "A seeded generator picks reproducible test inputs; nothing here is secret.")] + public void CombineMatchesTheCrcOfTheConcatenation() + { + var random = new Random(11); + byte[] buffer = new byte[200_000]; + random.NextBytes(buffer); + for (int trial = 0; trial < 400; trial++) + { + int total = trial < 40 ? trial : random.Next(40, buffer.Length); + int cut = random.Next(0, total + 1); + ReadOnlySpan whole = buffer.AsSpan(0, total); + uint first = Crc32.Append(0, whole[..cut]); + uint second = Crc32.Append(0, whole[cut..]); + uint combined = Crc32.Combine(first, second, total - cut); + Assert.True(combined == BitwiseCrc(whole), $"total {total}, cut {cut}: {combined:X8}"); + } + } + + [Fact] + public void CombineWithNothingKeepsTheCrc() + { + Assert.Equal(0x1234_5678u, Crc32.Combine(0x1234_5678, 0, 0)); + Assert.Equal(0u, Crc32.Combine(0, 0, 0)); + } + + [Fact] + public void CombineChainsOverThreeParts() + { + byte[] a = "first part "u8.ToArray(); + byte[] b = "second, longer part of the input "u8.ToArray(); + byte[] c = "third"u8.ToArray(); + uint ab = Crc32.Combine(Crc32.Append(0, a), Crc32.Append(0, b), b.Length); + uint abc = Crc32.Combine(ab, Crc32.Append(0, c), c.Length); + Assert.Equal(BitwiseCrc([.. a, .. b, .. c]), abc); + } + + [Fact] + public void CombineRejectsANegativeLength() + { + Assert.Throws(() => Crc32.Combine(1, 2, -1)); + } + } +} diff --git a/tests/ExcelReader.Tests/Writer/Zip/DeflateEncoderTests.cs b/tests/ExcelReader.Tests/Writer/Zip/DeflateEncoderTests.cs new file mode 100644 index 00000000..a49eb6bb --- /dev/null +++ b/tests/ExcelReader.Tests/Writer/Zip/DeflateEncoderTests.cs @@ -0,0 +1,208 @@ +using System.Diagnostics.CodeAnalysis; +using System.Globalization; +using System.IO.Compression; +using System.Text; +using ExcelReader.Core.Reader.Zip.Inflate; +using ExcelReader.Core.Writer.Internal; +using ExcelReader.Core.Writer.Zip; + +namespace ExcelReader.Tests.Writer.Zip +{ + public class DeflateEncoderTests + { + internal static byte[] Encode(ReadOnlySpan input, Func nextChunk) + { + using var encoder = new DeflateEncoder(); + using var output = new BiffBuffer(4096); + while (!input.IsEmpty) + { + int take = Math.Clamp(nextChunk(input.Length), 1, input.Length); + encoder.Write(input[..take], output); + input = input[take..]; + } + encoder.Finish(output); + return output.Span.ToArray(); + } + + internal static byte[] SheetXml(int rows) + { + var builder = new StringBuilder(); + for (int row = 1; row <= rows; row++) + { + builder.Append(CultureInfo.InvariantCulture, + $"item {row % 97}{row}{row * 1.25}"); + } + return Encoding.UTF8.GetBytes(builder.ToString()); + } + + private static byte[] Inflate(byte[] compressed, bool library) + { + using var source = new MemoryStream(compressed); + Stream inflate; + if (library) + { + inflate = new InflateStream(source); + } + else + { + inflate = new DeflateStream(source, CompressionMode.Decompress); + } + using (inflate) + { + using var output = new MemoryStream(); + inflate.CopyTo(output); + return output.ToArray(); + } + } + + private static void AssertRoundTrips(byte[] input, byte[] compressed) + { + byte[] viaZlib = Inflate(compressed, library: false); + byte[] viaLibrary = Inflate(compressed, library: true); + Assert.True(viaZlib.AsSpan().SequenceEqual(input), $"DeflateStream gave {viaZlib.Length} bytes for {input.Length}"); + Assert.True(viaLibrary.AsSpan().SequenceEqual(input), $"InflateStream gave {viaLibrary.Length} bytes for {input.Length}"); + } + + [SuppressMessage("Security", "CA5394:Do not use insecure randomness", + Justification = "A seeded generator picks reproducible test inputs; nothing here is secret.")] + private static byte[] RandomBytes(int length, int seed) + { + byte[] bytes = new byte[length]; + new Random(seed).NextBytes(bytes); + return bytes; + } + + [Fact] + public void EmptyInputIsOneEmptyFinalBlock() + { + byte[] compressed = Encode([], static _ => 1); + Assert.Equal([0x03, 0x00], compressed); + AssertRoundTrips([], compressed); + } + + [Fact] + public void SingleByteRoundTrips() + { + byte[] input = [(byte)'x']; + AssertRoundTrips(input, Encode(input, static _ => 1)); + } + + [Fact] + public void SheetXmlRoundTripsAndShrinks() + { + byte[] input = SheetXml(50_000); + byte[] compressed = Encode(input, static _ => 64 * 1024); + AssertRoundTrips(input, compressed); + Assert.True(compressed.Length < input.Length / 4, $"{compressed.Length} of {input.Length}"); + } + + [Fact] + public void TenMegabytesOfZerosUseLongMatches() + { + byte[] input = new byte[10 << 20]; + byte[] compressed = Encode(input, static _ => 1 << 20); + AssertRoundTrips(input, compressed); + Assert.True(compressed.Length < input.Length / 50, $"{compressed.Length} of {input.Length}"); + } + + [Fact] + public void RandomBytesFallBackToStoredBlocks() + { + byte[] input = RandomBytes(1 << 20, seed: 3); + byte[] compressed = Encode(input, static _ => 64 * 1024); + AssertRoundTrips(input, compressed); + long bound = input.Length + (5L * ((input.Length / DeflateEncoder.MaxBlockLength) + 1)) + 16; + Assert.True(compressed.Length <= bound, $"{compressed.Length} > {bound}"); + } + + [Fact] + public void CjkTextRoundTrips() + { + var builder = new StringBuilder(); + for (int i = 0; i < 20_000; i++) + { + builder.Append(CultureInfo.InvariantCulture, $"日本語のテキスト{i % 313}、中文文本。"); + } + byte[] input = Encoding.UTF8.GetBytes(builder.ToString()); + byte[] compressed = Encode(input, static _ => 4096); + AssertRoundTrips(input, compressed); + Assert.True(compressed.Length < input.Length, $"{compressed.Length} of {input.Length}"); + } + + [Fact] + [SuppressMessage("Security", "CA5394:Do not use insecure randomness", + Justification = "A seeded generator picks reproducible test inputs; nothing here is secret.")] + public void RandomWriteSizesRoundTrip() + { + byte[] input = [.. SheetXml(30_000), .. RandomBytes(300_000, seed: 4), .. SheetXml(30_000)]; + var random = new Random(9); + AssertRoundTrips(input, Encode(input, _ => random.Next(1, 1 << 20))); + } + + [Fact] + public void OneHugeWriteSlidesTheWindowMidWrite() + { + byte[] input = SheetXml(120_000); + Assert.True(input.Length > 4 << 20); + AssertRoundTrips(input, Encode(input, static remaining => remaining)); + } + + [Fact] + public void ByteAtATimeWritesRoundTrip() + { + byte[] input = SheetXml(3_000); + AssertRoundTrips(input, Encode(input, static _ => 1)); + } + + [Fact] + public void MatchesAtAndPastTheWindowEdgeRoundTrip() + { + byte[] history = RandomBytes(32_768, seed: 6); + byte[] input = [.. history, .. history.AsSpan(0, 1_000), .. history]; + AssertRoundTrips(input, Encode(input, static _ => 64 * 1024)); + } + + [Fact] + public void FarMatchAtBlockStartSurvivesPendingBits() + { + byte[] run = RandomBytes(300, seed: 1); + run[0] = 0xAB; + for (int prefix = 0; prefix < 256; prefix++) + { + foreach (int distance in new[] { 30_000, 22_000, 18_000 }) + { + byte[] input = new byte[DeflateEncoder.MaxBlockLength + 2_000]; + RandomBytes(prefix, seed: prefix).CopyTo(input, 0); + run.CopyTo(input, DeflateEncoder.MaxBlockLength - distance); + run.AsSpan(0, 200).CopyTo(input.AsSpan(DeflateEncoder.MaxBlockLength)); + RandomBytes(1_800, seed: 1_000 + prefix).CopyTo(input, DeflateEncoder.MaxBlockLength + 200); + input[DeflateEncoder.MaxBlockLength + 200] = (byte)(run[200] ^ 0xFF); + AssertRoundTrips(input, Encode(input, static remaining => remaining)); + } + } + } + + [Fact] + public void FixedCodesFollowRfc1951() + { + Assert.Equal(0x188u, FixedHuffmanCodes.Literals[0]); + Assert.Equal(0x269u, FixedHuffmanCodes.Literals[144]); + Assert.Equal(0x807u, FixedHuffmanCodes.Lengths[3]); + Assert.Equal(0x1468u, FixedHuffmanCodes.Lengths[258]); + Assert.Equal(0x5u, FixedHuffmanCodes.Distances[1]); + Assert.Equal(0x7FFEF2u, FixedHuffmanCodes.Distances[32_768]); + } + + [Fact] + public void UsingADisposedEncoderThrows() + { + using var output = new BiffBuffer(4096); + var encoder = new DeflateEncoder(); + encoder.Write(new byte[100], output); + encoder.Dispose(); + Assert.Throws(() => encoder.Write(new byte[1], output)); + Assert.Throws(() => encoder.Write(new byte[DeflateEncoder.MaxBlockLength], output)); + Assert.Throws(() => encoder.Finish(output)); + } + } +} diff --git a/tests/ExcelReader.Tests/Writer/Zip/PrefetchAbandonTests.cs b/tests/ExcelReader.Tests/Writer/Zip/PrefetchAbandonTests.cs new file mode 100644 index 00000000..64ad00e7 --- /dev/null +++ b/tests/ExcelReader.Tests/Writer/Zip/PrefetchAbandonTests.cs @@ -0,0 +1,167 @@ +using System.Diagnostics; +using ExcelReader.Core.Writer.Xlsb; +using ExcelReader.Core.Writer.Xlsx; + +namespace ExcelReader.Tests.Writer.Zip +{ + public class PrefetchAbandonTests + { + private const int Iterations = 200; + private const int Rows = 4_000; + private static readonly string[] Texts = BuildTexts(); + + private static string[] BuildTexts() + { + uint state = 11; + var texts = new string[Rows]; + var chars = new char[120]; + for (int i = 0; i < texts.Length; i++) + { + for (int c = 0; c < chars.Length; c++) + { + state = (state * 1_664_525) + 1_013_904_223; + chars[c] = (char)('0' + (state >> 26)); + } + texts[i] = new string(chars); + } + return texts; + } + + private static (IDisposable Workbook, Func Released) StartXlsx(Stream output) + { + XlsxWorkbookWriter wb = XlsxWorkbookWriter.Create(output, leaveOpen: true, new XlsxWriterOptions { PrefetchWrite = true }); + XlsxSheetWriter sheet = wb.AddSheet("Data"); + for (int i = 0; i < Rows; i++) + { + using XlsxRowWriter row = sheet.StartRow(); + row.Write(i); + row.Write(Texts[i]); + } + sheet.StartRow().Write("left open"); + return (wb, () => sheet.ResourcesReleased); + } + + private static (IDisposable Workbook, Func Released) StartXlsb(Stream output) + { + XlsbWorkbookWriter wb = XlsbWorkbookWriter.Create(output, leaveOpen: true, new XlsbWriterOptions { PrefetchWrite = true }); + XlsbSheetWriter sheet = wb.AddSheet("Data"); + for (int i = 0; i < Rows; i++) + { + using XlsbRowWriter row = sheet.StartRow(); + row.Write(i); + row.Write(Texts[i]); + } + sheet.StartRow().Write("left open"); + return (wb, () => sheet.ResourcesReleased); + } + + [Theory] + [InlineData(false, false)] + [InlineData(false, true)] + [InlineData(true, false)] + [InlineData(true, true)] + public async Task DisposingWithARowLeftOpenStopsTheBackgroundWriterFirst(bool xlsb, bool async) + { + var outputs = new List(Iterations); + for (int iteration = 0; iteration < Iterations; iteration++) + { + var output = new ProbeStream(); + outputs.Add(output); + (IDisposable workbook, Func released) = xlsb ? StartXlsb(output) : StartXlsx(output); + if (async) + { + await Assert.ThrowsAsync(() => ((IAsyncDisposable)workbook).DisposeAsync().AsTask()); + } + else + { + Assert.Throws(workbook.Dispose); + } + output.Seal(); + Assert.True(released()); + } + await Task.Delay(100, TestContext.Current.CancellationToken); + Assert.All(outputs, static output => + { + Assert.True(output.Writes > 1); + Assert.Equal(0, output.Overlaps); + Assert.Equal(0, output.LateWrites); + }); + } + + /// A slow forward-only output that counts overlapping writes and writes made after . + private sealed class ProbeStream : Stream + { + private int _active; + private int _overlaps; + private int _lateWrites; + private int _writes; + private volatile bool _sealed; + + internal int Overlaps => Volatile.Read(ref _overlaps); + internal int LateWrites => Volatile.Read(ref _lateWrites); + internal int Writes => Volatile.Read(ref _writes); + public override bool CanRead => false; + public override bool CanSeek => false; + public override bool CanWrite => true; + public override long Length => throw new NotSupportedException(); + public override long Position { get => throw new NotSupportedException(); set => throw new NotSupportedException(); } + + internal void Seal() + { + _sealed = true; + } + + public override void Flush() { } + + public override int Read(byte[] buffer, int offset, int count) + { + throw new NotSupportedException(); + } + + public override long Seek(long offset, SeekOrigin origin) + { + throw new NotSupportedException(); + } + + public override void SetLength(long value) + { + throw new NotSupportedException(); + } + + public override void Write(byte[] buffer, int offset, int count) + { + Record(); + } + + public override void Write(ReadOnlySpan buffer) + { + Record(); + } + + public override ValueTask WriteAsync(ReadOnlyMemory buffer, CancellationToken cancellationToken = default) + { + Record(); + return ValueTask.CompletedTask; + } + + private void Record() + { + Interlocked.Increment(ref _writes); + if (_sealed) + { + Interlocked.Increment(ref _lateWrites); + } + if (Interlocked.Increment(ref _active) != 1) + { + Interlocked.Increment(ref _overlaps); + } + long until = Stopwatch.GetTimestamp() + (Stopwatch.Frequency / 5_000); + while (Stopwatch.GetTimestamp() < until) + { + Thread.SpinWait(20); + } + Interlocked.Decrement(ref _active); + } + } + } +} diff --git a/tests/ExcelReader.Tests/Writer/Zip/SegmentCompressorTests.cs b/tests/ExcelReader.Tests/Writer/Zip/SegmentCompressorTests.cs new file mode 100644 index 00000000..a6baa300 --- /dev/null +++ b/tests/ExcelReader.Tests/Writer/Zip/SegmentCompressorTests.cs @@ -0,0 +1,94 @@ +using System.Diagnostics.CodeAnalysis; +using System.IO.Compression; +using ExcelReader.Core.Reader.Zip.Inflate; +using ExcelReader.Core.Writer.Internal; +using ExcelReader.Core.Writer.Zip; + +namespace ExcelReader.Tests.Writer.Zip +{ + public class SegmentCompressorTests + { + public static TheoryData DeflateLevels() + { + return [CompressionLevel.Fastest, CompressionLevel.Optimal, CompressionLevel.SmallestSize]; + } + + private static byte[] Inflate(Func open, byte[] compressed) + { + using Stream inflate = open(new MemoryStream(compressed)); + using var copy = new MemoryStream(); + inflate.CopyTo(copy); + return copy.ToArray(); + } + + internal static void AssertInflatesTo(byte[] expected, byte[] compressed) + { + Assert.Equal(expected, Inflate(static s => new DeflateStream(s, CompressionMode.Decompress), compressed)); + Assert.Equal(expected, Inflate(static s => new InflateStream(s), compressed)); + } + + [Theory] + [MemberData(nameof(DeflateLevels))] + [SuppressMessage("Security", "CA5394:Do not use insecure randomness", + Justification = "A seeded generator picks reproducible test inputs; nothing here is secret.")] + public void SegmentsConcatenateIntoOneStream(CompressionLevel level) + { + var random = new Random(3); + byte[] noise = new byte[70_000]; + random.NextBytes(noise); + byte[][] parts = + [ + DeflateEncoderTests.SheetXml(2_000), + [], + noise, + [42], + DeflateEncoderTests.SheetXml(9_000), + ]; + using var output = new BiffBuffer(4096); + foreach (byte[] part in parts) + { + SegmentCompressor.Compress(part, level, output); + } + output.Write(SegmentCompressor.FinalBlock); + + AssertInflatesTo([.. parts.SelectMany(static p => p)], output.Span.ToArray()); + } + + [Fact] + public void NoCompressionCopiesTheBytes() + { + using var output = new BiffBuffer(64); + SegmentCompressor.Compress("plain"u8, CompressionLevel.NoCompression, output); + Assert.Equal("plain"u8.ToArray(), output.Span.ToArray()); + } + + [Fact] + public void FlushSyncLeavesTheEncoderUsable() + { + byte[] first = DeflateEncoderTests.SheetXml(3_000); + byte[] second = DeflateEncoderTests.SheetXml(1_000); + using var encoder = new DeflateEncoder(); + using var output = new BiffBuffer(4096); + encoder.Write(first, output); + encoder.FlushSync(output); + int aligned = output.Length; + encoder.FlushSync(output); + encoder.Write(second, output); + encoder.Finish(output); + + Assert.Equal([0x00, 0x00, 0xFF, 0xFF], output.Span.Slice(aligned - 4, 4).ToArray()); + AssertInflatesTo([.. first, .. second], output.Span.ToArray()); + } + + [Fact] + public void BufferSinkDropsWritesWhileDiscarding() + { + using var target = new BiffBuffer(64); + using var sink = new BufferSink(target); + sink.Write("kept"u8); + sink.Discard = true; + sink.Write("dropped"u8); + Assert.Equal("kept"u8.ToArray(), target.Span.ToArray()); + } + } +} diff --git a/tests/ExcelReader.Tests/Writer/Zip/WriterZipIntegrationTests.cs b/tests/ExcelReader.Tests/Writer/Zip/WriterZipIntegrationTests.cs new file mode 100644 index 00000000..8f36cb9d --- /dev/null +++ b/tests/ExcelReader.Tests/Writer/Zip/WriterZipIntegrationTests.cs @@ -0,0 +1,127 @@ +using System.Globalization; +using System.IO.Compression; +using ExcelReader.Core.Reader; +using ExcelReader.Core.Reader.Xlsb; +using ExcelReader.Core.Reader.Xlsx; +using ExcelReader.Core.Writer.Xlsb; +using ExcelReader.Core.Writer.Xlsx; + +namespace ExcelReader.Tests.Writer.Zip +{ + public class WriterZipIntegrationTests + { + private const int Rows = 30_000; + + public static TheoryData Cases() + { + var data = new TheoryData(); + foreach (CompressionLevel level in Enum.GetValues()) + { + foreach (bool seekable in new[] { true, false }) + { + data.Add(level, seekable, false); + data.Add(level, seekable, true); + } + } + return data; + } + + private static Stream Target(MemoryStream ms, bool seekable) + { + if (seekable) + { + return ms; + } + return new ForwardOnlyWriteStream(ms); + } + + private static void AssertEveryEntryCrc(byte[] package) + { + using var zip = new ZipArchive(new MemoryStream(package), ZipArchiveMode.Read); + Assert.NotEmpty(zip.Entries); + foreach (ZipArchiveEntry entry in zip.Entries) + { + using Stream data = entry.Open(); + using var copy = new MemoryStream(); + data.CopyTo(copy); + Assert.Equal(Crc32Tests.BitwiseCrc(copy.ToArray()), entry.Crc32); + } + } + + private static void AssertRows(IExcelWorkbook reader) + { + using IExcelRowEnumerator rows = reader.SheetAt(0).GetEnumerator(); + for (int i = 0; i < Rows; i++) + { + Assert.True(rows.MoveNext()); + Assert.Equal(i.ToString(CultureInfo.InvariantCulture), rows.Current[0].GetString()); + Assert.Equal($"text {i % 101}", rows.Current[1].GetString()); + } + Assert.False(rows.MoveNext()); + using IExcelRowEnumerator second = reader.SheetAt(1).GetEnumerator(); + Assert.True(second.MoveNext()); + Assert.Equal("second", second.Current[0].GetString()); + } + + [Theory] + [MemberData(nameof(Cases))] + public async Task XlsxReadsBackFromEveryLevelAndOutput(CompressionLevel level, bool seekable, bool prefetch) + { + CancellationToken ct = TestContext.Current.CancellationToken; + var ms = new MemoryStream(); + var options = new XlsxWriterOptions { Compression = level, PrefetchWrite = prefetch, UseSharedStrings = true }; + await using (XlsxWorkbookWriter wb = XlsxWorkbookWriter.Create(Target(ms, seekable), leaveOpen: true, options)) + { + XlsxSheetWriter sheet = wb.AddSheet("Data"); + for (int i = 0; i < Rows; i++) + { + await using XlsxRowWriter row = await sheet.StartRowAsync(ct); + row.Write(i); + row.Write($"text {i % 101}"); + row.Write(i * 1.5); + } + await sheet.EndAsync(ct); + XlsxSheetWriter second = wb.AddSheet("Second"); + await using (XlsxRowWriter row = await second.StartRowAsync(ct)) + { + row.Write("second"); + } + await second.EndAsync(ct); + await wb.EndAsync(ct); + } + AssertEveryEntryCrc(ms.ToArray()); + using XlsxWorkbook reader = Excel.FromXlsx(new MemoryStream(ms.ToArray())); + AssertRows(reader); + } + + [Theory] + [MemberData(nameof(Cases))] + public void XlsbReadsBackFromEveryLevelAndOutput(CompressionLevel level, bool seekable, bool prefetch) + { + var ms = new MemoryStream(); + var options = new XlsbWriterOptions { Compression = level, PrefetchWrite = prefetch, UseSharedStrings = true }; + using (XlsbWorkbookWriter wb = XlsbWorkbookWriter.Create(Target(ms, seekable), leaveOpen: true, options)) + { + XlsbSheetWriter sheet = wb.AddSheet("Data"); + for (int i = 0; i < Rows; i++) + { + using XlsbRowWriter row = sheet.StartRow(); + row.Write(i); + row.Write($"text {i % 101}"); + row.Write(i * 1.5); + } + sheet.End(); + XlsbSheetWriter second = wb.AddSheet("Second"); + using (XlsbRowWriter row = second.StartRow()) + { + row.Write("second"); + } + second.End(); + wb.End(); + } + AssertEveryEntryCrc(ms.ToArray()); + using XlsbWorkbook reader = Excel.FromXlsb(new MemoryStream(ms.ToArray())); + AssertRows(reader); + } + } +} diff --git a/tests/ExcelReader.Tests/Writer/Zip/ZipEntrySegmentTests.cs b/tests/ExcelReader.Tests/Writer/Zip/ZipEntrySegmentTests.cs new file mode 100644 index 00000000..d53d8a89 --- /dev/null +++ b/tests/ExcelReader.Tests/Writer/Zip/ZipEntrySegmentTests.cs @@ -0,0 +1,145 @@ +using System.IO.Compression; +using ExcelReader.Core.Writer.Internal; +using ExcelReader.Core.Writer.Zip; + +namespace ExcelReader.Tests.Writer.Zip +{ + public class ZipEntrySegmentTests + { + public static TheoryData Cases() + { + var data = new TheoryData(); + foreach (CompressionLevel level in Enum.GetValues()) + { + data.Add(level, true); + data.Add(level, false); + } + return data; + } + + private static Stream Target(MemoryStream ms, bool seekable) + { + if (seekable) + { + return ms; + } + return new ForwardOnlyWriteStream(ms); + } + + private static (byte[] Segment, uint Crc) Segment(byte[] raw, CompressionLevel level) + { + using var buffer = new BiffBuffer(4096); + SegmentCompressor.Compress(raw, level, buffer); + return (buffer.Span.ToArray(), Crc32.Append(0, raw)); + } + + private static void AssertEntry(byte[] package, byte[] expected) + { + using var zip = new ZipArchive(new MemoryStream(package), ZipArchiveMode.Read); + ZipArchiveEntry entry = Assert.Single(zip.Entries); + using Stream data = entry.Open(); + using var copy = new MemoryStream(); + data.CopyTo(copy); + Assert.Equal(expected, copy.ToArray()); + Assert.Equal(Crc32Tests.BitwiseCrc(expected), entry.Crc32); + Assert.Equal(expected.Length, entry.Length); + } + + [Theory] + [MemberData(nameof(Cases))] + public void SerialWritesAndSegmentsInterleave(CompressionLevel level, bool seekable) + { + byte[] a = DeflateEncoderTests.SheetXml(1_500); + byte[] b = DeflateEncoderTests.SheetXml(4_000); + byte[] c = ""u8.ToArray(); + byte[] d = DeflateEncoderTests.SheetXml(2_500); + byte[] e = DeflateEncoderTests.SheetXml(700); + var ms = new MemoryStream(); + var zip = new ZipPackageWriter(Target(ms, seekable)); + using (ZipEntryStream entry = zip.OpenEntry("sheet.xml", level)) + { + entry.Write(a); + (byte[] segment, uint crc) = Segment(b, level); + entry.AppendSegment(segment, crc, b.Length); + entry.Write(c); + (segment, crc) = Segment(d, level); + entry.AppendSegment(segment, crc, d.Length); + (segment, crc) = Segment([], level); + entry.AppendSegment(segment, crc, 0); + entry.Write(e); + } + zip.Finish(); + + AssertEntry(ms.ToArray(), [.. a, .. b, .. c, .. d, .. e]); + } + + [Theory] + [MemberData(nameof(Cases))] + public async Task ASegmentCanBeTheFirstAndOnlyContent(CompressionLevel level, bool seekable) + { + CancellationToken ct = TestContext.Current.CancellationToken; + byte[] raw = DeflateEncoderTests.SheetXml(3_000); + var ms = new MemoryStream(); + var zip = new ZipPackageWriter(Target(ms, seekable)); + ZipEntryStream entry = zip.OpenEntry("sheet.xml", level); + await using (entry) + { + (byte[] segment, uint crc) = Segment(raw, level); + await entry.AppendSegmentAsync(segment, crc, raw.Length, ct); + } + await zip.FinishAsync(ct); + + AssertEntry(ms.ToArray(), raw); + } + + // A zlib stream kept across a segment could point back at bytes the reader sees before the segment. + [Fact] + public void SerialWritesAfterASegmentStartAFreshStream() + { + byte[] repeated = DeflateEncoderTests.SheetXml(2_000); + var ms = new MemoryStream(); + var zip = new ZipPackageWriter(ms); + using (ZipEntryStream entry = zip.OpenEntry("sheet.xml", CompressionLevel.Optimal)) + { + entry.Write(repeated); + (byte[] segment, uint crc) = Segment(""u8.ToArray(), CompressionLevel.Optimal); + entry.AppendSegment(segment, crc, 8); + entry.Write(repeated); + } + zip.Finish(); + + AssertEntry(ms.ToArray(), [.. repeated, .. ""u8, .. repeated]); + } + + [Fact] + public void AFailedAppendKillsTheEntryAndTheArchive() + { + byte[] raw = DeflateEncoderTests.SheetXml(20_000); + var zip = new ZipPackageWriter(new FailingWriteStream(limit: 0)); + ZipEntryStream entry = zip.OpenEntry("sheet.xml", CompressionLevel.Fastest); + (byte[] segment, uint crc) = Segment(raw, CompressionLevel.Fastest); + + Assert.Throws(() => entry.AppendSegment(segment, crc, raw.Length)); + entry.Dispose(); + Assert.Throws(() => entry.AppendSegment(segment, crc, raw.Length)); + Assert.Throws(zip.Finish); + zip.Abandon(); + } + + [Fact] + public async Task AFailedAsyncAppendKillsTheEntryAndTheArchive() + { + CancellationToken ct = TestContext.Current.CancellationToken; + byte[] raw = DeflateEncoderTests.SheetXml(20_000); + var zip = new ZipPackageWriter(new FailingWriteStream(limit: 0)); + ZipEntryStream entry = zip.OpenEntry("sheet.xml", CompressionLevel.Fastest); + (byte[] segment, uint crc) = Segment(raw, CompressionLevel.Fastest); + + await Assert.ThrowsAsync(async () => await entry.AppendSegmentAsync(segment, crc, raw.Length, ct)); + await entry.DisposeAsync(); + await Assert.ThrowsAsync(async () => await entry.AppendSegmentAsync(segment, crc, raw.Length, ct)); + await Assert.ThrowsAsync(async () => await zip.FinishAsync(ct)); + zip.Abandon(); + } + } +} diff --git a/tests/ExcelReader.Tests/Writer/Zip/ZipPackageWriterTests.cs b/tests/ExcelReader.Tests/Writer/Zip/ZipPackageWriterTests.cs new file mode 100644 index 00000000..e7e6db48 --- /dev/null +++ b/tests/ExcelReader.Tests/Writer/Zip/ZipPackageWriterTests.cs @@ -0,0 +1,738 @@ +using System.Diagnostics.CodeAnalysis; +using System.Globalization; +using System.IO.Compression; +using System.Text; +using ExcelReader.Core.Writer.Zip; + +namespace ExcelReader.Tests.Writer.Zip +{ + public class ZipPackageWriterTests + { + private const int ChunkSize = 8192; + private static readonly (string Name, byte[] Content)[] Entries = BuildEntries(); + + [SuppressMessage("Security", "CA5394:Do not use insecure randomness", + Justification = "A seeded generator picks reproducible test inputs; nothing here is secret.")] + private static (string Name, byte[] Content)[] BuildEntries() + { + var random = new Random(11); + var digits = new StringBuilder(); + while (digits.Length < 1_000_000) + { + digits.Append(random.Next().ToString(CultureInfo.InvariantCulture)).Append(','); + } + return + [ + ("a.xml", "hello hello hello"u8.ToArray()), + ("empty.xml", []), + ("big.xml", Encoding.ASCII.GetBytes(digits.ToString())), + ("ünïcode.xml", ""u8.ToArray()), + ]; + } + + public static TheoryData LayoutCases() + { + var data = new TheoryData(); + foreach (CompressionLevel level in Enum.GetValues()) + { + data.Add(level, true, 0, false); + data.Add(level, true, 0, true); + data.Add(level, false, 0, false); + data.Add(level, false, 0, true); + data.Add(level, true, 100, false); + } + return data; + } + + private static Stream Target(MemoryStream ms, bool seekable) + { + if (seekable) + { + return ms; + } + return new ForwardOnlyWriteStream(ms); + } + + private static void WriteInChunks(Stream entry, byte[] content) + { + for (int offset = 0; offset < content.Length; offset += ChunkSize) + { + entry.Write(content, offset, Math.Min(ChunkSize, content.Length - offset)); + } + } + + private static byte[] WriteWithZipArchive(CompressionLevel level, bool seekable, int prefix) + { + var ms = new MemoryStream(); + ms.Write(new byte[prefix]); + using (var zip = new ZipArchive(Target(ms, seekable), ZipArchiveMode.Create, leaveOpen: true)) + { + foreach ((string name, byte[] content) in Entries) + { + using Stream entry = zip.CreateEntry(name, level).Open(); + WriteInChunks(entry, content); + } + } + return ms.ToArray(); + } + + private static async Task WriteWithPackageAsync(CompressionLevel level, bool seekable, int prefix, bool useAsync) + { + CancellationToken ct = TestContext.Current.CancellationToken; + var ms = new MemoryStream(); + ms.Write(new byte[prefix]); + var zip = new ZipPackageWriter(Target(ms, seekable)); + foreach ((string name, byte[] content) in Entries) + { + Stream entry = zip.OpenEntry(name, level); + if (!useAsync) + { + WriteInChunks(entry, content); + entry.Dispose(); + continue; + } + for (int offset = 0; offset < content.Length; offset += ChunkSize) + { + await entry.WriteAsync(content.AsMemory(offset, Math.Min(ChunkSize, content.Length - offset)), ct); + } + await entry.DisposeAsync(); + } + if (useAsync) + { + await zip.FinishAsync(ct); + } + else + { + zip.Finish(); + } + return ms.ToArray(); + } + + private static void AssertContentAndCrc(byte[] zipBytes) + { + using var zip = new ZipArchive(new MemoryStream(zipBytes), ZipArchiveMode.Read); + Assert.Equal(Entries.Length, zip.Entries.Count); + foreach ((string name, byte[] content) in Entries) + { + ZipArchiveEntry entry = zip.GetEntry(name)!; + using Stream data = entry.Open(); + using var copy = new MemoryStream(); + data.CopyTo(copy); + Assert.True(copy.ToArray().AsSpan().SequenceEqual(content), name); + Assert.Equal(Crc32Tests.BitwiseCrc(content), entry.Crc32); + } + } + + private static void AssertSameShape(ZipCentral expected, ZipCentral actual, byte[] expectedZip, byte[] actualZip) + { + Assert.Equal(expected.Name, actual.Name); + // ZipArchive puts the operating system in the high byte; the package writer always writes 0, + // so a file does not depend on the machine that wrote it. + Assert.Equal((byte)expected.MadeBy, (byte)actual.MadeBy); + Assert.Equal(0, actual.MadeBy >> 8); + Assert.Equal(expected.Version, actual.Version); + Assert.Equal(expected.Flags, actual.Flags); + Assert.Equal(expected.Method, actual.Method); + Assert.Equal(expected.Crc, actual.Crc); + Assert.Equal(expected.UncompressedSize, actual.UncompressedSize); + Assert.Equal(expected.Extra, actual.Extra); + ZipLocal expectedLocal = ZipLayout.ReadLocal(expectedZip, expected.LocalOffset); + ZipLocal actualLocal = ZipLayout.ReadLocal(actualZip, actual.LocalOffset); + Assert.Equal(expectedLocal.Version, actualLocal.Version); + Assert.Equal(expectedLocal.Flags, actualLocal.Flags); + Assert.Equal(expectedLocal.Method, actualLocal.Method); + Assert.Equal(expectedLocal.Crc, actualLocal.Crc); + Assert.Equal(expectedLocal.UncompressedSize, actualLocal.UncompressedSize); + Assert.Equal(expectedLocal.CompressedSize == 0, actualLocal.CompressedSize == 0); + if (actualLocal.CompressedSize != 0) + { + Assert.Equal(actual.CompressedSize, actualLocal.CompressedSize); + } + } + + [Theory] + [MemberData(nameof(LayoutCases))] + public async Task LayoutMatchesZipArchive(CompressionLevel level, bool seekable, int prefix, bool useAsync) + { + byte[] expected = WriteWithZipArchive(level, seekable, prefix); + byte[] actual = await WriteWithPackageAsync(level, seekable, prefix, useAsync); + List expectedEntries = ZipLayout.ReadCentral(expected); + List actualEntries = ZipLayout.ReadCentral(actual); + Assert.Equal(expectedEntries.Count, actualEntries.Count); + for (int i = 0; i < expectedEntries.Count; i++) + { + AssertSameShape(expectedEntries[i], actualEntries[i], expected, actual); + } + if (level == CompressionLevel.NoCompression) + { + Assert.Equal(ZipLayout.WithoutTimestampsAndHost(expected), ZipLayout.WithoutTimestampsAndHost(actual)); + } + AssertContentAndCrc(actual); + } + + [Fact] + public void ZeroEntriesWriteTheEmptyArchive() + { + var expected = new MemoryStream(); + using (new ZipArchive(expected, ZipArchiveMode.Create, leaveOpen: true)) + { + } + var actual = new MemoryStream(); + new ZipPackageWriter(actual).Finish(); + Assert.Equal(expected.ToArray(), actual.ToArray()); + using var read = new ZipArchive(new MemoryStream(actual.ToArray()), ZipArchiveMode.Read); + Assert.Empty(read.Entries); + } + + [Fact] + public void OpeningASecondEntryWhileOneIsOpenThrows() + { + var zip = new ZipPackageWriter(new MemoryStream()); + using Stream first = zip.OpenEntry("a.xml", CompressionLevel.Fastest); + Assert.Throws(() => zip.OpenEntry("b.xml", CompressionLevel.Fastest)); + Assert.Throws(zip.Finish); + } + + [Fact] + public void UndefinedCompressionLevelIsRejected() + { + var zip = new ZipPackageWriter(new MemoryStream()); + Assert.Throws(() => zip.OpenEntry("a.xml", (CompressionLevel)42)); + } + + [Fact] + public async Task DisposingAnEntryAgainIsHarmless() + { + var ms = new MemoryStream(); + var zip = new ZipPackageWriter(ms); + Stream entry = zip.OpenEntry("a.xml", CompressionLevel.Fastest); + entry.Write(""u8); + await entry.DisposeAsync(); + entry.Dispose(); + await entry.DisposeAsync(); + Assert.Throws(() => entry.Write("x"u8)); + zip.Finish(); + Assert.Single(ZipLayout.ReadCentral(ms.ToArray())); + } + + [Fact] + public void WriteByteBuildsTheSameEntry() + { + byte[] content = DeflateEncoderTests.SheetXml(200); + var ms = new MemoryStream(); + var zip = new ZipPackageWriter(ms); + using (Stream entry = zip.OpenEntry("bytes.xml", CompressionLevel.Fastest)) + { + foreach (byte value in content) + { + entry.WriteByte(value); + } + } + zip.Finish(); + using var read = new ZipArchive(new MemoryStream(ms.ToArray()), ZipArchiveMode.Read); + ZipArchiveEntry stored = read.GetEntry("bytes.xml")!; + using var copy = new MemoryStream(); + using (Stream data = stored.Open()) + { + data.CopyTo(copy); + } + Assert.True(copy.ToArray().AsSpan().SequenceEqual(content)); + Assert.Equal(Crc32Tests.BitwiseCrc(content), stored.Crc32); + } + + [Fact] + public void AFailedWriteKillsTheArchiveAndAbandonListsTheCompletedEntries() + { + var output = new FailOnceWriteStream(failOnCall: 2); + var zip = new ZipPackageWriter(output); + CompleteEntry(zip, "done.xml", Done); + Stream entry = zip.OpenEntry("big.xml", CompressionLevel.NoCompression); + Assert.Throws(() => entry.Write(new byte[64 * 1024])); + long length = AssertFailedArchiveRejectsEverything(zip, entry, output.Written); + + zip.Abandon(); + byte[] package = output.Written.ToArray(); + Assert.True(package.Length > length); + Assert.Equal(1, EndRecordCount(package)); + AssertEntries(package, ("done.xml", Done)); + zip.Abandon(); + Assert.Equal(package.Length, output.Written.Length); + Assert.Throws(() => zip.OpenEntry("next.xml", CompressionLevel.Fastest)); + Assert.Throws(zip.Finish); + } + + [Fact] + public async Task AFailedAsyncWriteKillsTheArchiveAndAbandonListsTheCompletedEntries() + { + CancellationToken ct = TestContext.Current.CancellationToken; + var output = new FailOnceWriteStream(failOnCall: 2); + var zip = new ZipPackageWriter(output); + CompleteEntry(zip, "done.xml", Done); + Stream entry = zip.OpenEntry("big.xml", CompressionLevel.NoCompression); + await Assert.ThrowsAsync(async () => await entry.WriteAsync(new byte[64 * 1024], ct)); + await entry.DisposeAsync(); + await Assert.ThrowsAsync(async () => await zip.FinishAsync(ct)); + AssertFailedArchiveRejectsEverything(zip, entry, output.Written); + + zip.Abandon(); + byte[] package = output.Written.ToArray(); + Assert.Equal(1, EndRecordCount(package)); + AssertEntries(package, ("done.xml", Done)); + } + + [Fact] + public async Task ACancelledWriteKillsTheArchiveEvenWhenDisposedWithNoToken() + { + CancellationToken ct = TestContext.Current.CancellationToken; + var output = new MemoryStream(); + var zip = new ZipPackageWriter(output); + CompleteEntry(zip, "done.xml", Done); + Stream entry = zip.OpenEntry("big.xml", CompressionLevel.NoCompression); + using var cancelled = new CancellationTokenSource(); + await cancelled.CancelAsync(); + await Assert.ThrowsAnyAsync( + async () => await entry.WriteAsync(new byte[64 * 1024], cancelled.Token)); + await entry.DisposeAsync(); + await Assert.ThrowsAsync(async () => await zip.FinishAsync(ct)); + AssertFailedArchiveRejectsEverything(zip, entry, output); + + zip.Abandon(); + byte[] package = output.ToArray(); + Assert.Equal(1, EndRecordCount(package)); + AssertEntries(package, ("done.xml", Done)); + } + + [Fact] + public void AFailedFlushKillsTheArchiveAndAbandonListsTheCompletedEntries() + { + var output = new FlushFailingStream(); + var zip = new ZipPackageWriter(output); + CompleteEntry(zip, "done.xml", Done); + Stream entry = zip.OpenEntry("next.xml", CompressionLevel.NoCompression); + entry.Write(Done); + output.FailFlush = true; + Assert.Throws(entry.Flush); + output.FailFlush = false; + AssertFailedArchiveRejectsEverything(zip, entry, output); + + zip.Abandon(); + byte[] package = output.ToArray(); + Assert.Equal(1, EndRecordCount(package)); + AssertEntries(package, ("done.xml", Done)); + } + + [Fact] + public async Task AFailedAsyncFlushKillsTheArchive() + { + CancellationToken ct = TestContext.Current.CancellationToken; + var output = new FlushFailingStream { FailFlush = true }; + var zip = new ZipPackageWriter(output); + Stream entry = zip.OpenEntry("a.xml", CompressionLevel.NoCompression); + await Assert.ThrowsAsync(() => entry.FlushAsync(ct)); + output.FailFlush = false; + AssertFailedArchiveRejectsEverything(zip, entry, output); + } + + [Fact] + public void FlushingAnEntryFlushesTheOutputAndFlushingAClosedEntryDoesNot() + { + var output = new FlushFailingStream(); + var zip = new ZipPackageWriter(output); + Stream entry = zip.OpenEntry("a.xml", CompressionLevel.NoCompression); + entry.Flush(); + Assert.Equal(1, output.Flushes); + entry.Dispose(); + entry.Flush(); + Assert.Equal(1, output.Flushes); + } + + [Theory] + [InlineData(CompressionLevel.Fastest)] + [InlineData(CompressionLevel.Optimal)] + [InlineData(CompressionLevel.NoCompression)] + public void AbandonWithAnOpenEntryListsTheCompletedEntriesAndReleasesTheOpenOne(CompressionLevel level) + { + var output = new MemoryStream(); + var zip = new ZipPackageWriter(output); + CompleteEntry(zip, "done.xml", Done); + Stream entry = zip.OpenEntry("a.xml", level); + entry.Write(DeflateEncoderTests.SheetXml(2000)); + zip.Abandon(); + + byte[] package = output.ToArray(); + Assert.Equal(1, EndRecordCount(package)); + AssertEntries(package, ("done.xml", Done)); + Assert.Throws(() => entry.Write(new byte[1])); + entry.Dispose(); + zip.Abandon(); + Assert.Equal(package.Length, output.Length); + Assert.Throws(zip.Finish); + Assert.Throws(() => zip.OpenEntry("next.xml", level)); + } + + [Fact] + public void AbandonOnAnOutputThatAlwaysFailsDoesNotThrow() + { + var output = new FailingWriteStream(limit: 0); + var zip = new ZipPackageWriter(output); + Stream entry = zip.OpenEntry("big.xml", CompressionLevel.NoCompression); + Assert.Throws(() => entry.Write(new byte[64 * 1024])); + zip.Abandon(); + zip.Abandon(); + Assert.Equal(0, output.Written.Length); + } + + [Fact] + public async Task AbandonAsyncWritesTheDirectoryOverAnOutputThatRejectsSyncWrites() + { + CancellationToken ct = TestContext.Current.CancellationToken; + var output = new AsyncOnlyStream(); + var zip = new ZipPackageWriter(output); + await CompleteEntryAsync(zip, "done.xml", Done, ct); + Stream entry = zip.OpenEntry("open.xml", CompressionLevel.NoCompression); + await entry.WriteAsync(Done, ct); + + await zip.AbandonAsync(); + + byte[] package = output.ToArray(); + Assert.Equal(1, EndRecordCount(package)); + AssertEntries(package, ("done.xml", Done)); + Assert.Throws(() => entry.Write(new byte[1])); + Assert.Throws(() => zip.OpenEntry("next.xml", CompressionLevel.Fastest)); + } + + [Fact] + public async Task AbandonAsyncIsIdempotentAndBlocksALaterAbandon() + { + CancellationToken ct = TestContext.Current.CancellationToken; + var output = new MemoryStream(); + var zip = new ZipPackageWriter(output); + await CompleteEntryAsync(zip, "done.xml", Done, ct); + + await zip.AbandonAsync(); + long length = output.Length; + await zip.AbandonAsync(); + zip.Abandon(); + + Assert.Equal(length, output.Length); + Assert.Equal(1, EndRecordCount(output.ToArray())); + } + + [Fact] + public async Task AbandonAsyncOnAnOutputThatAlwaysFailsDoesNotThrow() + { + var output = new FailingWriteStream(limit: 0); + var zip = new ZipPackageWriter(output); + Stream entry = zip.OpenEntry("big.xml", CompressionLevel.NoCompression); + await Assert.ThrowsAsync(async () => await entry.WriteAsync(new byte[64 * 1024], TestContext.Current.CancellationToken)); + await zip.AbandonAsync(); + await zip.AbandonAsync(); + Assert.Equal(0, output.Written.Length); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task AbandonSeeksBackToTheEndAfterAFailedHeaderPatch(bool useAsync) + { + var output = new FailOnBackwardWriteStream(); + var zip = new ZipPackageWriter(output); + CompleteEntry(zip, "done.xml", Done); + Stream entry = zip.OpenEntry("big.xml", CompressionLevel.NoCompression); + entry.Write(new byte[70_000]); + output.Armed = true; + Assert.Throws(entry.Dispose); + + if (useAsync) + { + await zip.AbandonAsync(); + } + else + { + zip.Abandon(); + } + + byte[] package = output.ToArray(); + Assert.Equal(1, EndRecordCount(package)); + AssertEntries(package, ("done.xml", Done)); + } + + private static readonly byte[] Done = "content"u8.ToArray(); + + private static void CompleteEntry(ZipPackageWriter zip, string name, byte[] content) + { + using Stream entry = zip.OpenEntry(name, CompressionLevel.NoCompression); + entry.Write(content); + } + + private static async Task CompleteEntryAsync(ZipPackageWriter zip, string name, byte[] content, CancellationToken ct) + { + Stream entry = zip.OpenEntry(name, CompressionLevel.NoCompression); + await using (entry.ConfigureAwait(false)) + { + await entry.WriteAsync(content, ct); + } + } + + private static long AssertFailedArchiveRejectsEverything(ZipPackageWriter zip, Stream entry, MemoryStream written) + { + long length = written.Length; + entry.Dispose(); + Assert.Throws(() => entry.Write(new byte[1])); + Assert.Throws(zip.Finish); + Assert.Throws(() => zip.OpenEntry("next.xml", CompressionLevel.Fastest)); + Assert.Equal(length, written.Length); + return length; + } + + private static int EndRecordCount(byte[] package) + { + int count = 0; + int from = 0; + ReadOnlySpan signature = [0x50, 0x4B, 0x05, 0x06]; + while (true) + { + int found = package.AsSpan(from).IndexOf(signature); + if (found < 0) + { + return count; + } + count++; + from += found + signature.Length; + } + } + + private static void AssertEntries(byte[] package, params (string Name, byte[] Content)[] expected) + { + using var archive = new ZipArchive(new MemoryStream(package), ZipArchiveMode.Read); + Assert.Equal(expected.Select(static e => e.Name), archive.Entries.Select(static e => e.FullName), StringComparer.Ordinal); + for (int i = 0; i < expected.Length; i++) + { + using Stream data = archive.Entries[i].Open(); + using var copy = new MemoryStream(); + data.CopyTo(copy); + Assert.True(copy.ToArray().AsSpan().SequenceEqual(expected[i].Content)); + Assert.Equal(Crc32Tests.BitwiseCrc(expected[i].Content), archive.Entries[i].Crc32); + } + } + + private sealed class AsyncOnlyStream : MemoryStream + { + private bool _inAsync; + + public override void Write(byte[] buffer, int offset, int count) + { + RejectSync(); + base.Write(buffer, offset, count); + } + + public override void Write(ReadOnlySpan buffer) + { + RejectSync(); + base.Write(buffer); + } + + public override async ValueTask WriteAsync(ReadOnlyMemory buffer, CancellationToken cancellationToken = default) + { + _inAsync = true; + try + { + Write(buffer.Span); + } + finally + { + _inAsync = false; + } + await Task.Yield(); + } + + private void RejectSync() + { + if (!_inAsync) + { + throw new NotSupportedException("sync write"); + } + } + } + + private sealed class FailOnBackwardWriteStream : MemoryStream + { + internal bool Armed { get; set; } + + public override void Write(ReadOnlySpan buffer) + { + if (Armed && Position < Length) + { + Armed = false; + throw new IOException("patch failed"); + } + base.Write(buffer); + } + + public override ValueTask WriteAsync(ReadOnlyMemory buffer, CancellationToken cancellationToken = default) + { + Write(buffer.Span); + return ValueTask.CompletedTask; + } + } + + private sealed class FlushFailingStream : MemoryStream + { + internal bool FailFlush { get; set; } + internal int Flushes { get; private set; } + + public override void Flush() + { + Flushes++; + if (FailFlush) + { + throw new IOException("flush failed"); + } + } + + public override Task FlushAsync(CancellationToken cancellationToken) + { + Flush(); + return Task.CompletedTask; + } + } + + [Theory] + [Trait("Category", "Slow")] + [InlineData(true)] + [InlineData(false)] + public void AnEntryOver4GiBUsesZip64AndSoDoLaterOffsets(bool seekable) + { + const long Size = (1L << 32) + (1 << 20); + var sink = new CapturingStream(seekable); + var zip = new ZipPackageWriter(sink); + byte[] chunk = new byte[1 << 20]; + uint crc = 0; + using (Stream big = zip.OpenEntry("big.bin", CompressionLevel.NoCompression)) + { + for (long written = 0; written < Size; written += chunk.Length) + { + big.Write(chunk); + crc = Crc32.Append(crc, chunk); + } + } + using (Stream after = zip.OpenEntry("after.xml", CompressionLevel.Fastest)) + { + after.Write(""u8); + } + zip.Finish(); + + ZipLocal local = ZipLayout.ReadLocal(sink.Head, 0); + Assert.Equal(ZipRecords.FlagDataDescriptor, local.Flags & ZipRecords.FlagDataDescriptor); + Assert.Equal(seekable ? 45 : 20, local.Version); + + byte[] tail = sink.Tail; + long descriptorAt = ZipRecords.LocalHeaderLength + "big.bin".Length + Size; + long d = descriptorAt - CapturingStream.TailStart; + Assert.Equal(0x0807_4B50u, ZipLayout.U32(tail, d)); + Assert.Equal(crc, ZipLayout.U32(tail, d + 4)); + Assert.Equal((ulong)Size, ZipLayout.U64(tail, d + 8)); + Assert.Equal((ulong)Size, ZipLayout.U64(tail, d + 16)); + + long end = tail.Length - 22; + Assert.Equal(0x0605_4B50u, ZipLayout.U32(tail, end)); + Assert.Equal(uint.MaxValue, ZipLayout.U32(tail, end + 16)); + long locator = end - 20; + Assert.Equal(0x0706_4B50u, ZipLayout.U32(tail, locator)); + long zip64End = (long)ZipLayout.U64(tail, locator + 8) - CapturingStream.TailStart; + Assert.Equal(0x0606_4B50u, ZipLayout.U32(tail, zip64End)); + Assert.Equal(2UL, ZipLayout.U64(tail, zip64End + 32)); + long central = (long)ZipLayout.U64(tail, zip64End + 48) - CapturingStream.TailStart; + + Assert.Equal(0x0201_4B50u, ZipLayout.U32(tail, central)); + Assert.Equal(45, ZipLayout.U16(tail, central + 6)); + Assert.Equal(uint.MaxValue, ZipLayout.U32(tail, central + 20)); + Assert.Equal(uint.MaxValue, ZipLayout.U32(tail, central + 24)); + Assert.Equal(20, ZipLayout.U16(tail, central + 30)); + Assert.Equal(0u, ZipLayout.U32(tail, central + 42)); + long extra = central + 46 + "big.bin".Length; + Assert.Equal(1, ZipLayout.U16(tail, extra)); + Assert.Equal(16, ZipLayout.U16(tail, extra + 2)); + Assert.Equal((ulong)Size, ZipLayout.U64(tail, extra + 4)); + Assert.Equal((ulong)Size, ZipLayout.U64(tail, extra + 12)); + + long second = extra + 20; + Assert.Equal(0x0201_4B50u, ZipLayout.U32(tail, second)); + Assert.Equal(45, ZipLayout.U16(tail, second + 6)); + Assert.Equal(uint.MaxValue, ZipLayout.U32(tail, second + 42)); + long secondExtra = second + 46 + "after.xml".Length; + Assert.Equal(1, ZipLayout.U16(tail, secondExtra)); + Assert.Equal(8, ZipLayout.U16(tail, secondExtra + 2)); + Assert.Equal((ulong)(descriptorAt + 24), ZipLayout.U64(tail, secondExtra + 4)); + } + + [Fact] + public void MoreThan65534EntriesUseZip64EndRecords() + { + var ms = new MemoryStream(); + var zip = new ZipPackageWriter(ms); + for (int i = 0; i < 70_000; i++) + { + using Stream entry = zip.OpenEntry($"e{i}.bin", CompressionLevel.NoCompression); + entry.WriteByte((byte)i); + } + zip.Finish(); + byte[] bytes = ms.ToArray(); + int end = bytes.Length - 22; + Assert.Equal(0xFFFF, ZipLayout.U16(bytes, end + 10)); + int locator = end - 20; + Assert.Equal(0x0706_4B50u, ZipLayout.U32(bytes, locator)); + long zip64End = (long)ZipLayout.U64(bytes, locator + 8); + Assert.Equal(0x0606_4B50u, ZipLayout.U32(bytes, zip64End)); + Assert.Equal(70_000UL, ZipLayout.U64(bytes, zip64End + 32)); + using var read = new ZipArchive(new MemoryStream(bytes), ZipArchiveMode.Read); + Assert.Equal(70_000, read.Entries.Count); + using Stream last = read.GetEntry("e69999.bin")!.Open(); + Assert.Equal(69_999 & 0xFF, last.ReadByte()); + } + + public static TheoryData HugeWriteCases() + { + var data = new TheoryData(); + foreach (CompressionLevel level in Enum.GetValues()) + { + data.Add(level, false); + data.Add(level, true); + } + return data; + } + + [Theory] + [MemberData(nameof(HugeWriteCases))] + [SuppressMessage("Security", "CA5394:Do not use insecure randomness", + Justification = "A seeded generator picks reproducible test inputs; nothing here is secret.")] + public async Task OneHugeWriteReachesTheOutputInBoundedPieces(CompressionLevel level, bool useAsync) + { + byte[] content = new byte[5 << 20]; + new Random(7).NextBytes(content); + var output = new LargestWriteStream(); + var zip = new ZipPackageWriter(output); + Stream entry = zip.OpenEntry("huge.bin", level); + if (useAsync) + { + await entry.WriteAsync(content, TestContext.Current.CancellationToken); + await entry.DisposeAsync(); + } + else + { + entry.Write(content); + entry.Dispose(); + } + zip.Finish(); + + Assert.InRange(output.Largest, 1, (1 << 20) + (64 * 1024)); + using var read = new ZipArchive(new MemoryStream(output.ToArray()), ZipArchiveMode.Read); + ZipArchiveEntry stored = read.GetEntry("huge.bin")!; + using Stream data = stored.Open(); + using var copy = new MemoryStream(); + await data.CopyToAsync(copy, TestContext.Current.CancellationToken); + Assert.True(copy.ToArray().AsSpan().SequenceEqual(content)); + Assert.Equal(Crc32Tests.BitwiseCrc(content), stored.Crc32); + } + } +} diff --git a/tests/ExcelReader.Tests/Writer/Zip/ZipTestSupport.cs b/tests/ExcelReader.Tests/Writer/Zip/ZipTestSupport.cs new file mode 100644 index 00000000..70d03c98 --- /dev/null +++ b/tests/ExcelReader.Tests/Writer/Zip/ZipTestSupport.cs @@ -0,0 +1,289 @@ +using System.Buffers.Binary; +using System.Text; + +namespace ExcelReader.Tests.Writer.Zip +{ + internal sealed record ZipCentral( + int Position, ushort MadeBy, ushort Version, ushort Flags, ushort Method, uint Crc, + uint CompressedSize, uint UncompressedSize, uint LocalOffset, string Name, byte[] Extra); + + internal sealed record ZipLocal(ushort Version, ushort Flags, ushort Method, uint Crc, uint CompressedSize, uint UncompressedSize); + + internal static class ZipLayout + { + internal static ushort U16(ReadOnlySpan data, long offset) + { + return BinaryPrimitives.ReadUInt16LittleEndian(data[(int)offset..]); + } + + internal static uint U32(ReadOnlySpan data, long offset) + { + return BinaryPrimitives.ReadUInt32LittleEndian(data[(int)offset..]); + } + + internal static ulong U64(ReadOnlySpan data, long offset) + { + return BinaryPrimitives.ReadUInt64LittleEndian(data[(int)offset..]); + } + + /// Reads a classic (non-Zip64) central directory through the end record at the end of the file. + internal static List ReadCentral(byte[] zip) + { + int end = zip.Length - 22; + Assert.Equal(0x0605_4B50u, U32(zip, end)); + int count = U16(zip, end + 10); + int position = (int)U32(zip, end + 16); + var entries = new List(count); + for (int i = 0; i < count; i++) + { + Assert.Equal(0x0201_4B50u, U32(zip, position)); + int nameLength = U16(zip, position + 28); + int extraLength = U16(zip, position + 30); + int commentLength = U16(zip, position + 32); + entries.Add(new ZipCentral( + position, U16(zip, position + 4), U16(zip, position + 6), U16(zip, position + 8), U16(zip, position + 10), + U32(zip, position + 16), U32(zip, position + 20), U32(zip, position + 24), U32(zip, position + 42), + Encoding.UTF8.GetString(zip, position + 46, nameLength), + zip.AsSpan(position + 46 + nameLength, extraLength).ToArray())); + position += 46 + nameLength + extraLength + commentLength; + } + return entries; + } + + internal static ZipLocal ReadLocal(ReadOnlySpan zip, long offset) + { + Assert.Equal(0x0403_4B50u, U32(zip, offset)); + return new ZipLocal(U16(zip, offset + 4), U16(zip, offset + 6), U16(zip, offset + 8), + U32(zip, offset + 14), U32(zip, offset + 18), U32(zip, offset + 22)); + } + + /// + /// Zeroes what legitimately differs between two writers of the same content: the DOS time and date + /// of every local and central header, and the two central fields + /// fills from the operating system (the host byte of "version made by", and the external attributes, + /// which hold Unix permissions there). + /// + internal static byte[] WithoutTimestampsAndHost(byte[] zip) + { + byte[] copy = (byte[])zip.Clone(); + foreach (ZipCentral entry in ReadCentral(zip)) + { + copy.AsSpan((int)entry.LocalOffset + 10, 4).Clear(); + copy.AsSpan(entry.Position + 12, 4).Clear(); + copy[entry.Position + 5] = 0; + copy.AsSpan(entry.Position + 38, 4).Clear(); + } + return copy; + } + } + + /// A write-only stream that refuses to seek, like a network or pipe stream. + internal sealed class ForwardOnlyWriteStream(Stream inner) : Stream + { + public override bool CanRead => false; + public override bool CanSeek => false; + public override bool CanWrite => true; + public override long Length => throw new NotSupportedException(); + public override long Position { get => throw new NotSupportedException(); set => throw new NotSupportedException(); } + public override void Flush() + { + inner.Flush(); + } + + public override int Read(byte[] buffer, int offset, int count) + { + throw new NotSupportedException(); + } + + public override long Seek(long offset, SeekOrigin origin) + { + throw new NotSupportedException(); + } + + public override void SetLength(long value) + { + throw new NotSupportedException(); + } + + public override void Write(byte[] buffer, int offset, int count) + { + inner.Write(buffer, offset, count); + } + + public override void Write(ReadOnlySpan buffer) + { + inner.Write(buffer); + } + } + + /// Accepts writes until bytes, then throws on every write. + internal sealed class FailingWriteStream(long limit) : Stream + { + internal MemoryStream Written { get; } = new(); + public override bool CanRead => false; + public override bool CanSeek => false; + public override bool CanWrite => true; + public override long Length => throw new NotSupportedException(); + public override long Position { get => throw new NotSupportedException(); set => throw new NotSupportedException(); } + public override void Flush() { } + public override int Read(byte[] buffer, int offset, int count) + { + throw new NotSupportedException(); + } + + public override long Seek(long offset, SeekOrigin origin) + { + throw new NotSupportedException(); + } + + public override void SetLength(long value) + { + throw new NotSupportedException(); + } + + public override void Write(byte[] buffer, int offset, int count) + { + Write(buffer.AsSpan(offset, count)); + } + + public override void Write(ReadOnlySpan buffer) + { + if (Written.Length + buffer.Length > limit) + { + throw new IOException("disk full"); + } + Written.Write(buffer); + } + } + + /// Throws on write number without storing it, then accepts every later write. + internal sealed class FailOnceWriteStream(int failOnCall) : Stream + { + private int _calls; + + internal MemoryStream Written { get; } = new(); + public override bool CanRead => false; + public override bool CanSeek => false; + public override bool CanWrite => true; + public override long Length => throw new NotSupportedException(); + public override long Position { get => throw new NotSupportedException(); set => throw new NotSupportedException(); } + public override void Flush() { } + public override int Read(byte[] buffer, int offset, int count) + { + throw new NotSupportedException(); + } + + public override long Seek(long offset, SeekOrigin origin) + { + throw new NotSupportedException(); + } + + public override void SetLength(long value) + { + throw new NotSupportedException(); + } + + public override void Write(byte[] buffer, int offset, int count) + { + Write(buffer.AsSpan(offset, count)); + } + + public override void Write(ReadOnlySpan buffer) + { + _calls++; + if (_calls == failOnCall) + { + throw new IOException("transient"); + } + Written.Write(buffer); + } + } + + /// + /// Counts every byte but keeps only the first 4 KB and everything at or past 4 GiB, so a Zip64 + /// archive can be inspected without holding it. + /// + internal sealed class CapturingStream(bool seekable) : Stream + { + internal const long TailStart = 1L << 32; + private readonly MemoryStream _tail = new(); + private long _position; + private long _length; + + internal byte[] Head { get; } = new byte[4096]; + internal byte[] Tail => _tail.ToArray(); + public override bool CanRead => false; + public override bool CanSeek => seekable; + public override bool CanWrite => true; + public override long Length => _length; + + public override long Position + { + get => _position; + set => Seek(value, SeekOrigin.Begin); + } + + public override void Flush() { } + public override int Read(byte[] buffer, int offset, int count) + { + throw new NotSupportedException(); + } + + public override void SetLength(long value) + { + throw new NotSupportedException(); + } + + public override void Write(byte[] buffer, int offset, int count) + { + Write(buffer.AsSpan(offset, count)); + } + + public override long Seek(long offset, SeekOrigin origin) + { + if (!seekable || origin != SeekOrigin.Begin) + { + throw new NotSupportedException(); + } + _position = offset; + return _position; + } + + public override void Write(ReadOnlySpan buffer) + { + long start = _position; + if (start < Head.Length) + { + int headPart = (int)Math.Min(buffer.Length, Head.Length - start); + buffer[..headPart].CopyTo(Head.AsSpan((int)start)); + } + long end = start + buffer.Length; + if (end > TailStart) + { + long from = Math.Max(start, TailStart); + _tail.Position = from - TailStart; + _tail.Write(buffer[(int)(from - start)..]); + } + _position = end; + _length = Math.Max(_length, end); + } + } + + /// A seekable in-memory output that remembers the size of its largest single write. + internal sealed class LargestWriteStream : MemoryStream + { + internal int Largest { get; private set; } + + public override void Write(byte[] buffer, int offset, int count) + { + Largest = Math.Max(Largest, count); + base.Write(buffer, offset, count); + } + + public override void Write(ReadOnlySpan buffer) + { + Largest = Math.Max(Largest, buffer.Length); + base.Write(buffer); + } + } +} diff --git a/tests/ExcelReader.Tests/data/excel-formula-strings.xls b/tests/ExcelReader.Tests/data/excel-formula-strings.xls new file mode 100644 index 00000000..36010934 Binary files /dev/null and b/tests/ExcelReader.Tests/data/excel-formula-strings.xls differ diff --git a/tests/ExcelReader.Tests/data/excel-formula-strings.xlsx b/tests/ExcelReader.Tests/data/excel-formula-strings.xlsx new file mode 100644 index 00000000..51e40241 Binary files /dev/null and b/tests/ExcelReader.Tests/data/excel-formula-strings.xlsx differ