diff --git a/_posts/2026-09-08-arrow-go-18.8.0.md b/_posts/2026-09-08-arrow-go-18.8.0.md new file mode 100644 index 000000000000..86582e113b3e --- /dev/null +++ b/_posts/2026-09-08-arrow-go-18.8.0.md @@ -0,0 +1,232 @@ +--- +layout: post +title: "Apache Arrow Go 18.8.0 Release" +date: "2026-09-08 00:00:00" +author: pmc +categories: [release] +--- + + +The Apache Arrow team is pleased to announce the v18.8.0 release of Apache Arrow Go. +This minor release covers 283 commits from 16 distinct contributors. + +## Contributors +```console +$ git shortlog -sn v18.7.0..v18.8.0 + 251 Minh Vu + 10 Matt Topol + 4 Colton Loftus + 4 Digvijay + 4 Tom Frank + 1 Derek Perkins + 1 Lucas Valente + 1 Madan kumar + 1 Neelesh Salian + 1 Nick Ripley + 1 Pedro Matias + 1 Raúl Cumplido + 1 Ruihao Chen + 1 Tobias Pütz + 1 Tomáš Karela Procházka + 1 Willem Jan +``` + +## Highlights + +### Arrow + +#### Features + +- Added `VariantGet` for efficient path extraction from variant arrays [#1206](https://github.com/apache/arrow-go/pull/1206) +- Added `list_element`, `cumulative_sum`, `cumulative_sum_checked`, and `dictionary_encode` compute functions [#1138](https://github.com/apache/arrow-go/pull/1138), +[#1139](https://github.com/apache/arrow-go/pull/1139), [#1140](https://github.com/apache/arrow-go/pull/1140) +- Added `ValueAsAny` for accessing Arrow array values as native Go values [#1170](https://github.com/apache/arrow-go/pull/1170) +- Strengthened IPC file validation for malformed block framing and lengths [#1097](https://github.com/apache/arrow-go/pull/1097) + +#### Bug Fixes +- Fixed a memory leak when calling `Resize(0)` on non-empty builders [#995](https://github.com/apache/arrow-go/pull/995) +- Preserved exact precision when casting strings to Decimal128 and Decimal256 scalars [#1107](https://github.com/apache/arrow-go/pull/1107) + +#### Performance Improvements + +This is a performance-heavy release: 32 of the merged pull requests are dedicated +optimizations. Unless noted otherwise, the figures below are the PR-reported +benchmarks on an Apple M1 Pro, comparing the new path against the scalar +baseline it replaces. + +- Expanded SIMD acceleration for compute arithmetic, comparisons, and filtering on ARM64 and AMD64. Arithmetic kernels run 4.1-4.4x faster on int64 and float64 +adds, int64 array/array comparisons drop from 313.6us to 29.3us per op (10.7x), narrow 8- and 16-bit integer comparisons reach 12.8-15.1x, and null-free +32-bit fixed-width filters compress up to 11.8x faster on 1M-element inputs [#1265](https://github.com/apache/arrow-go/pull/1265), [#1266](https://github.com/apache/arrow-go/pull/1266), +[#1284](https://github.com/apache/arrow-go/pull/1284), [#1286](https://github.com/apache/arrow-go/pull/1286), [#1287](https://github.com/apache/arrow-go/pull/1287), [#1288](https://github.com/apache/arrow-go/pull/1288) +- Building take indices from fragmented selection vectors is no longer quadratic. For a 1M-element alternating filter it drops from 4.31s to 515us, and a +serial `FilterRecordBatch` over the same filter falls from 2.06s to 831us [#1287](https://github.com/apache/arrow-go/pull/1287) +- Added SIMD-accelerated packing of boolean slices and validity data into Arrow bitmaps: `BooleanBuilder` packing of 65,536 values improves roughly 4-5x +(23.4us to 3.9us), batched validity packing is up to 4.4x faster, and ARM64 NEON bitmap operations gain 2.2-2.9x [#1283](https://github.com/apache/arrow-go/pull/1283), +[#1185](https://github.com/apache/arrow-go/pull/1185), [#1264](https://github.com/apache/arrow-go/pull/1264), [#1090](https://github.com/apache/arrow-go/pull/1090) +- Batched run-end encoded appends turn a millisecond-scale operation into a microsecond-scale one for null and empty-value runs, cutting 1.16MB across 55 +allocations per op down to 1.7KB across 19 [#1214](https://github.com/apache/arrow-go/pull/1214) +- Concatenation avoids temporary buffers and copies bitmaps directly: 65,536 int64 values spread over 8,192 chunks go from 571.7us to 214.5us (-62%) with +allocations falling from 8,199 to 6 per op [#1194](https://github.com/apache/arrow-go/pull/1194), [#1224](https://github.com/apache/arrow-go/pull/1224) +- Serial and small-batch execution skips goroutine and span overhead: single-span kernel dispatch nearly halves latency at length 1 (342.0ns to 189.0ns), +serial record filtering over 128 columns goes from 709.8us to 420.9us, and serial `Take` plus IPC encoder scratch reuse cut per-op allocations across the +board [#1233](https://github.com/apache/arrow-go/pull/1233), [#1245](https://github.com/apache/arrow-go/pull/1245), [#1226](https://github.com/apache/arrow-go/pull/1226), [#1221](https://github.com/apache/arrow-go/pull/1221) +- CSV writing no longer materializes rows first, cutting a 1,000-row by 16-column write from 4.78ms to 0.84ms (-82%) and allocated bytes by 81% [#1190](https://github.com/apache/arrow-go/pull/1190) + +### Parquet + +#### Features + +- Added read and write support for GeoArrow extension types using Parquet geometry and geography logical types [#960](https://github.com/apache/arrow-go/pull/960), +[#969](https://github.com/apache/arrow-go/pull/969) +- Added codec-aware dictionary cost fallback and per-column writer controls [#1036](https://github.com/apache/arrow-go/pull/1036) + +#### Bug Fixes + +- Correctly populated Bloom filters from dictionary-encoded values [#1164](https://github.com/apache/arrow-go/pull/1164) +- Rejected decimal values that overflow Parquet INT32 or INT64 columns instead of silently narrowing them [#1122](https://github.com/apache/arrow-go/pull/1122), +[#1161](https://github.com/apache/arrow-go/pull/1161) +- Fixed reading fixed-size lists whose parent values are null [#1099](https://github.com/apache/arrow-go/pull/1099) + +#### Performance Improvements + +- Added SIMD `BYTE_STREAM_SPLIT` encoding on ARM64 and AMD64: 16.9x faster for 4-byte values (113,877ns to 6,745ns per op) and 7.6x for 8-byte values [#1260](https://github.com/apache/arrow-go/pull/1260) +- Boolean decoding now targets Arrow bitmaps directly. All-true RLE boolean pages of 1M values decode in 2.45us instead of 2.65ms, and dense boolean column +reads through pqarrow drop 90% in time and 65% in heap use [#1246](https://github.com/apache/arrow-go/pull/1246), [#1182](https://github.com/apache/arrow-go/pull/1182) +- Batched encoding and decoding across the core codecs: plain `BYTE_ARRAY` encoding is 45-62% faster, RLE level encoding is 79% faster for all-defined +levels, and `DELTA_BINARY_PACKED` decoding is 34-66% faster [#1193](https://github.com/apache/arrow-go/pull/1193), [#1177](https://github.com/apache/arrow-go/pull/1177), [#1169](https://github.com/apache/arrow-go/pull/1169) +- Dictionary encoding allocates far less: numeric insertion of 65,535 float32 values goes from 64,905 allocations per op to 13, and typed byte-array memo +insertion cuts time 24% and allocations 50% (Apple M3 Max). Fixed-width dictionary materialization is also vectorized on AVX2 systems [#1178](https://github.com/apache/arrow-go/pull/1178), +[#1272](https://github.com/apache/arrow-go/pull/1272), [#1269](https://github.com/apache/arrow-go/pull/1269) +- Reduced copying when writing binary and fixed-size binary Arrow values to Parquet: 16-byte `FixedSizeBinary` writes are 72% faster for required columns +with statistics disabled, and 64K String writes go from 1.798ms to 1.260ms [#1262](https://github.com/apache/arrow-go/pull/1262), [#1263](https://github.com/apache/arrow-go/pull/1263) +- Compressors are reused across pages: random 64KiB gzip page encoding is 6.1x faster (94.7us to 15.6us) with allocations down from 15 to 1 per op, and +Brotli encoding drops from about 22 allocations per op to 1 [#1238](https://github.com/apache/arrow-go/pull/1238), [#1254](https://github.com/apache/arrow-go/pull/1254) +- Serial reads skip goroutine orchestration, trimming 18% from single-column row-group reads [#1235](https://github.com/apache/arrow-go/pull/1235) + +## Changelog + +### Selected Changes + +A selection of the user-facing changes in this release. See the full changelog link +below for all 285 merged pull requests. + +#### Arrow + +* fix: remove superfluous struct field index compare by @Willem-J-an in [#970](https://github.com/apache/arrow-go/pull/970) +* fix(ipc): reject truncated message frames by @fallintoplace in [#991](https://github.com/apache/arrow-go/pull/991) +* fix: Fix arm64 assembly stack frame manipulation by @nsrip-dd in [#984](https://github.com/apache/arrow-go/pull/984) +* fix(arrow/array): `Resize(0)` on non-empty builder causes memory leak by @serramatutu in [#995](https://github.com/apache/arrow-go/pull/995) +* fix(arrow/csv): preserve decimal precision by @fallintoplace in [#1022](https://github.com/apache/arrow-go/pull/1022) +* refactor(arrow/avro): migrate from hamba/avro to twmb/avro by @prochac in [#830](https://github.com/apache/arrow-go/pull/830) +* feat(flightsql): allow configuring the driver memory allocator by @fallintoplace in [#1076](https://github.com/apache/arrow-go/pull/1076) +* feat(arrow/flight/sql): Add is_update field to ActionCreatePreparedStatementResult by @ennuite in [#732](https://github.com/apache/arrow-go/pull/732) +* feat(arrow/array): roll back JSON builder state after failed rows by @fallintoplace in [#1113](https://github.com/apache/arrow-go/pull/1113) +* feat(arrow/array): add ValueAsAny for native Go values by @dgvj-work in [#1170](https://github.com/apache/arrow-go/pull/1170) +* feat(extensions): add VariantGet for path extraction from variant arrays by @nssalian in [#1206](https://github.com/apache/arrow-go/pull/1206) +* fix(arrow/extensions): use canonical Variant extension name by @dgvj-work in [#1241](https://github.com/apache/arrow-go/pull/1241) +* fix(arrow/extensions): reject Null Variant typed_value by @dgvj-work in [#1243](https://github.com/apache/arrow-go/pull/1243) +* fix(arrow/ipc): validate file block framing and lengths by @fallintoplace in [#1097](https://github.com/apache/arrow-go/pull/1097) +* fix(arrow/ipc): avoid deadlock on compression errors by @fallintoplace in [#1142](https://github.com/apache/arrow-go/pull/1142) +* fix(arrow/scalar): preserve exact decimal string casts by @fallintoplace in [#1107](https://github.com/apache/arrow-go/pull/1107) +* fix(arrow/scalar): align timestamp timezone parsing by @fallintoplace in [#1109](https://github.com/apache/arrow-go/pull/1109) +* fix(arrow): validate precision in NewDecimalType by @fallintoplace in [#1162](https://github.com/apache/arrow-go/pull/1162) +* fix(arrow): reject duplicate union type codes by @fallintoplace in [#1131](https://github.com/apache/arrow-go/pull/1131) +* fix(arrow/csv): preserve first row during headerless inference by @fallintoplace in [#1145](https://github.com/apache/arrow-go/pull/1145) +* fix(arrow/csv): append null for later parse failures by @fallintoplace in [#1149](https://github.com/apache/arrow-go/pull/1149) +* fix(arrow/array): clamp run ends when concatenating a sliced RunEndEncoded array by @winklemad in [#1219](https://github.com/apache/arrow-go/pull/1219) +* fix(arrow/cdata): validate imported schema topology by @fallintoplace in [#1046](https://github.com/apache/arrow-go/pull/1046) + +#### Compute + +* feat(compute): add list_element by @fallintoplace in [#1138](https://github.com/apache/arrow-go/pull/1138) +* feat(compute): add cumulative_sum and cumulative_sum_checked by @fallintoplace in [#1139](https://github.com/apache/arrow-go/pull/1139) +* feat(compute): add dictionary_encode by @fallintoplace in [#1140](https://github.com/apache/arrow-go/pull/1140) +* fix(compute): round negative HalfToOdd ties correctly by @fallintoplace in [#1094](https://github.com/apache/arrow-go/pull/1094) +* fix(compute): handle odd temporal multiples by @fallintoplace in [#1119](https://github.com/apache/arrow-go/pull/1119) +* fix(compute): preserve caller context during execution by @fallintoplace in [#1154](https://github.com/apache/arrow-go/pull/1154) +* fix(compute): discard results after cancellation by @zeroshade in [#1278](https://github.com/apache/arrow-go/pull/1278) +* fix(compute/exprs): avoid empty field reference panic by @fallintoplace in [#1130](https://github.com/apache/arrow-go/pull/1130) + +#### Parquet + +* feat(parquet/schema): Wire up logical type functionality for writing geo types by @C-Loftus in [#960](https://github.com/apache/arrow-go/pull/960) +* feat(parquet/pqarrow): Read support for extension types / geo by @C-Loftus in [#969](https://github.com/apache/arrow-go/pull/969) +* feat(parquet): add WithDictionaryCostFallback writer property by @tom-eon in [#1036](https://github.com/apache/arrow-go/pull/1036) +* fix(parquet): wrap DELTA_BINARY_PACKED deltas at the physical type width by @tom-eon in [#1027](https://github.com/apache/arrow-go/pull/1027) +* fix(parquet/encoding): read ahead in the streaming value buffer by @joechenrh in [#937](https://github.com/apache/arrow-go/pull/937) +* fix(parquet/pqarrow): prefer stored schema extension type on read by @twuebi in [#1051](https://github.com/apache/arrow-go/pull/1051) +* fix(parquet/pqarrow): require Variant value field when writing by @dgvj-work in [#1242](https://github.com/apache/arrow-go/pull/1242) +* fix(parquet): guard byte-stream-split encoder release against never-flushed buffer by @tom-eon in [#1273](https://github.com/apache/arrow-go/pull/1273) +* fix(parquet): stop BYTE_STREAM_SPLIT FLBA spaced decode aliasing output by @zeroshade in [#1256](https://github.com/apache/arrow-go/pull/1256) +* fix(parquet): read null fixed-size-list parents by @fallintoplace in [#1099](https://github.com/apache/arrow-go/pull/1099) +* fix(parquet/pqarrow): reject decimal overflow in integer columns by @fallintoplace in [#1122](https://github.com/apache/arrow-go/pull/1122) +* fix(parquet/pqarrow): reject decimal overflow when writing integers by @fallintoplace in [#1161](https://github.com/apache/arrow-go/pull/1161) +* fix(parquet): build bloom filters from dictionary entries by @fallintoplace in [#1164](https://github.com/apache/arrow-go/pull/1164) +* fix(parquet): preserve dictionary index types by @fallintoplace in [#1098](https://github.com/apache/arrow-go/pull/1098) +* fix(parquet): restore Arrow duration units from schema metadata by @fallintoplace in [#1095](https://github.com/apache/arrow-go/pull/1095) +* fix(parquet/compress): honor LZ4 raw destination contract by @fallintoplace in [#1147](https://github.com/apache/arrow-go/pull/1147) +* fix(parquet): row-align V1 pages with offset indexes by @fallintoplace in [#1075](https://github.com/apache/arrow-go/pull/1075) +* fix(parquet/variant): validate metadata offset tables by @fallintoplace in [#1063](https://github.com/apache/arrow-go/pull/1063) + +#### Performance + +* perf(arrow/bitutil): add SetBitSwap/ClearBitSwap, use in builder.SetNull by @tom-eon in [#1090](https://github.com/apache/arrow-go/pull/1090) +* perf(bitutil): add ARM64 NEON bitmap operations by @fallintoplace in [#1264](https://github.com/apache/arrow-go/pull/1264) +* perf(arrow/array): pack validity values in batches by @fallintoplace in [#1185](https://github.com/apache/arrow-go/pull/1185) +* perf(arrow/array): SIMD-pack boolean slices into bitmaps by @fallintoplace in [#1283](https://github.com/apache/arrow-go/pull/1283) +* perf(arrow/array): batch run-end encoded appends by @fallintoplace in [#1214](https://github.com/apache/arrow-go/pull/1214) +* perf(arrow/array): avoid temporary buffers when concatenating by @fallintoplace in [#1194](https://github.com/apache/arrow-go/pull/1194) +* perf(arrow/array): concatenate bitmaps directly by @fallintoplace in [#1224](https://github.com/apache/arrow-go/pull/1224) +* perf(compute): add ARM64 NEON arithmetic kernels by @fallintoplace in [#1265](https://github.com/apache/arrow-go/pull/1265) +* perf(compute): add ARM64 NEON comparison kernels by @fallintoplace in [#1266](https://github.com/apache/arrow-go/pull/1266) +* perf(compute): add ARM64 NEON comparisons for 8- and 16-bit integers by @fallintoplace in [#1288](https://github.com/apache/arrow-go/pull/1288) +* perf(compute): add ARM64 NEON compression for null-free 32-bit fixed-width filters by @fallintoplace in [#1286](https://github.com/apache/arrow-go/pull/1286) +* perf(compute): vectorize mixed int32 filters with AVX2 by @fallintoplace in [#1284](https://github.com/apache/arrow-go/pull/1284) +* perf(compute): SIMD-compact fragmented filters into take indices by @fallintoplace in [#1287](https://github.com/apache/arrow-go/pull/1287) +* perf(compute): fast-path single-span execution by @fallintoplace in [#1233](https://github.com/apache/arrow-go/pull/1233) +* perf(arrow/compute): avoid goroutines for serial record filtering by @fallintoplace in [#1245](https://github.com/apache/arrow-go/pull/1245) +* perf(arrow/compute): avoid goroutines for serial take by @fallintoplace in [#1226](https://github.com/apache/arrow-go/pull/1226) +* perf(parquet): SIMD BYTE_STREAM_SPLIT encoding by @fallintoplace in [#1260](https://github.com/apache/arrow-go/pull/1260) +* perf(parquet): vectorize fixed-width dictionary materialization by @fallintoplace in [#1269](https://github.com/apache/arrow-go/pull/1269) +* perf(parquet): optimize numeric dictionary insertion by @fallintoplace in [#1178](https://github.com/apache/arrow-go/pull/1178) +* perf(parquet): use typed memo insertion for byte-array dictionaries by @derekperkins in [#1272](https://github.com/apache/arrow-go/pull/1272) +* perf(parquet): batch plain BYTE_ARRAY encoding by @fallintoplace in [#1193](https://github.com/apache/arrow-go/pull/1193) +* perf(parquet): encode RLE levels in batches by @fallintoplace in [#1177](https://github.com/apache/arrow-go/pull/1177) +* perf(parquet): batch delta binary packed decoding by @fallintoplace in [#1169](https://github.com/apache/arrow-go/pull/1169) +* perf(parquet): decode RLE booleans into bitmaps by @fallintoplace in [#1246](https://github.com/apache/arrow-go/pull/1246) +* perf(parquet/pqarrow): decode booleans directly to bitmaps by @fallintoplace in [#1182](https://github.com/apache/arrow-go/pull/1182) +* perf(parquet/pqarrow): write binary values from Arrow offsets by @fallintoplace in [#1262](https://github.com/apache/arrow-go/pull/1262) +* perf(parquet/pqarrow): write FixedSizeBinary values directly by @fallintoplace in [#1263](https://github.com/apache/arrow-go/pull/1263) +* perf(parquet/pqarrow): use synchronous path for serial reads by @fallintoplace in [#1235](https://github.com/apache/arrow-go/pull/1235) +* perf(parquet/compress): reuse gzip writers by @fallintoplace in [#1238](https://github.com/apache/arrow-go/pull/1238) +* perf(parquet/compress): reuse Brotli writers across encodes by @fallintoplace in [#1254](https://github.com/apache/arrow-go/pull/1254) +* perf(arrow/csv): avoid materializing rows before writing by @fallintoplace in [#1190](https://github.com/apache/arrow-go/pull/1190) +* perf(arrow/ipc): reuse record encoder scratch by @fallintoplace in [#1221](https://github.com/apache/arrow-go/pull/1221) + +### New Contributors +* @C-Loftus made their first contribution in [#960](https://github.com/apache/arrow-go/pull/960) +* @tom-eon made their first contribution in [#1027](https://github.com/apache/arrow-go/pull/1027) +* @prochac made their first contribution in [#830](https://github.com/apache/arrow-go/pull/830) +* @dgvj-work made their first contribution in [#1170](https://github.com/apache/arrow-go/pull/1170) +* @ennuite made their first contribution in [#732](https://github.com/apache/arrow-go/pull/732) +* @derekperkins made their first contribution in [#1272](https://github.com/apache/arrow-go/pull/1272) +* @winklemad made their first contribution in [#1219](https://github.com/apache/arrow-go/pull/1219) + +**Full Changelog**: https://github.com/apache/arrow-go/compare/v18.7.0...v18.8.0