From 95e65002fed91d2ee0a7d9fec8770146cc869c45 Mon Sep 17 00:00:00 2001 From: Matt Topol Date: Mon, 16 Nov 2026 17:17:04 +0000 Subject: [PATCH 1/3] [Release] Add release notes for Arrow Go 18.9.0 --- _posts/2026-11-16-arrow-go-18.9.0.md | 359 +++++++++++++++++++++++++++ 1 file changed, 359 insertions(+) create mode 100644 _posts/2026-11-16-arrow-go-18.9.0.md diff --git a/_posts/2026-11-16-arrow-go-18.9.0.md b/_posts/2026-11-16-arrow-go-18.9.0.md new file mode 100644 index 000000000000..2d9cf3081718 --- /dev/null +++ b/_posts/2026-11-16-arrow-go-18.9.0.md @@ -0,0 +1,359 @@ +--- +layout: post +title: "Apache Arrow Go 18.9.0 Release" +date: "2026-11-16 00:00:00" +author: pmc +categories: [release] +--- + + +The Apache Arrow team is pleased to announce the v18.9.0 release of Apache Arrow Go. +This minor release covers 152 commits from 27 distinct contributors. + +## Contributors +```console +$ git shortlog -sn v18.8.0..v18.9.0 + 45 Minh Vu + 37 Harrison Crosse + 31 Matt Topol + 6 singhpratech + 3 Sankalp Thakur + 2 Animesh Kumar + 2 Colton Loftus + 2 Edward Shen + 2 Samuele Resca + 2 Steve O'Brien + 2 Stochastic + 2 Willem Jan + 2 autoCAT + 2 matanper + 1 Arnaud + 1 Arnold Wakim + 1 Ayaan + 1 Jake Thomas + 1 Kyon + 1 Madan Kumar + 1 Nirajan Sah + 1 Ondřej Pavela + 1 Tobias Pütz + 1 hp + 1 peterolkhov + 1 prtkgaur + 1 ump45nose +``` + +## Changelog + +### What's Changed +* feat(parquet/schema): Wire up logical type functionality for writing geo types by @C-Loftus in [#960](https://github.com/apache/arrow-go/pull/960) +* chore(dev/release): filter out non numeric go releases (rc's and beta's) when finding latest go by @raulcd in [#968](https://github.com/apache/arrow-go/pull/968) +* ci: fix release workflow discussions by @zeroshade in [#980](https://github.com/apache/arrow-go/pull/980) +* ci: fix lint on release workflow by @zeroshade in [#982](https://github.com/apache/arrow-go/pull/982) +* fix: remove superfluous struct field index compare by @Willem-J-an in [#970](https://github.com/apache/arrow-go/pull/970) +* fix(ipc): reject truncated message frames by @fallintoplace in [#991](https://github.com/apache/arrow-go/pull/991) +* fix(internal/utils): handle reader reset and short peeks by @fallintoplace in [#987](https://github.com/apache/arrow-go/pull/987) +* fix(arrow/ipc): propagate CLI stream errors by @fallintoplace in [#986](https://github.com/apache/arrow-go/pull/986) +* fix(arrow/flight): avoid blocking after stream cancellation by @fallintoplace in [#989](https://github.com/apache/arrow-go/pull/989) +* fix(parquet/variant): handle empty metadata keys by @fallintoplace in [#985](https://github.com/apache/arrow-go/pull/985) +* fix(flight/flightsql/example): propagate row iteration errors by @fallintoplace in [#990](https://github.com/apache/arrow-go/pull/990) +* fix(parquet): reject truncated scalar bit reads by @fallintoplace in [#944](https://github.com/apache/arrow-go/pull/944) +* fix: Fix arm64 assembly stack frame manipulation by @nsrip-dd in [#984](https://github.com/apache/arrow-go/pull/984) +* fix(arrow/array): support zero-column tables by @fallintoplace in [#988](https://github.com/apache/arrow-go/pull/988) +* docs(parquet): Add thrift generation contribution info by @C-Loftus in [#971](https://github.com/apache/arrow-go/pull/971) +* fix(array): reject mismatched known null counts during full validation by @fallintoplace in [#993](https://github.com/apache/arrow-go/pull/993) +* fix(arrow/array): `Resize(0)` on non-empty builder causes memory leak by @serramatutu in [#995](https://github.com/apache/arrow-go/pull/995) +* feat(parquet/pqarrow): Read support for extension types / geo by @C-Loftus in [#969](https://github.com/apache/arrow-go/pull/969) +* fix(array): validate run-end encoded invariants by @fallintoplace in [#992](https://github.com/apache/arrow-go/pull/992) +* fix(array): validate binary_view and string_view payloads by @fallintoplace in [#994](https://github.com/apache/arrow-go/pull/994) +* fix(parquet): wrap DELTA_BINARY_PACKED deltas at the physical type width by @tom-eon in [#1027](https://github.com/apache/arrow-go/pull/1027) +* fix(arrow/compute): use is_null/is_not_null in expressions by @fallintoplace in [#997](https://github.com/apache/arrow-go/pull/997) +* fix(arrow/ipc): reject NaN minSpaceSavings values by @fallintoplace in [#998](https://github.com/apache/arrow-go/pull/998) +* fix(arrow/tensor): default nil dimension names to empty strings by @fallintoplace in [#999](https://github.com/apache/arrow-go/pull/999) +* fix(arrow/scalar): allow null extension scalars to keep null storage by @fallintoplace in [#1000](https://github.com/apache/arrow-go/pull/1000) +* fix(arrow/scalar): handle nil child values in run-end scalars by @fallintoplace in [#1001](https://github.com/apache/arrow-go/pull/1001) +* fix(arrow/array): ignore zero-count union appends by @fallintoplace in [#1026](https://github.com/apache/arrow-go/pull/1026) +* fix(arrow/csv): preserve decimal precision by @fallintoplace in [#1022](https://github.com/apache/arrow-go/pull/1022) +* fix(parquet/metadata): compare bloom filter caps in bits by @fallintoplace in [#996](https://github.com/apache/arrow-go/pull/996) +* fix(parquet/pqarrow): reject timestamp coercion overflow by @fallintoplace in [#1008](https://github.com/apache/arrow-go/pull/1008) +* fix(parquet/file): reject records starting with nonzero repetition levels by @fallintoplace in [#1009](https://github.com/apache/arrow-go/pull/1009) +* fix(parquet/variant): reject truncated scalar values by @fallintoplace in [#1017](https://github.com/apache/arrow-go/pull/1017) +* fix(parquet/encoding): reject invalid delta byte prefixes by @fallintoplace in [#1014](https://github.com/apache/arrow-go/pull/1014) +* fix(parquet/metadata): preserve infinite floating-point statistics by @fallintoplace in [#1007](https://github.com/apache/arrow-go/pull/1007) +* fix(parquet/encoding): bound lazy dictionary growth by @fallintoplace in [#1011](https://github.com/apache/arrow-go/pull/1011) +* fix(parquet/encoding): respect declared RLE payload lengths by @fallintoplace in [#1015](https://github.com/apache/arrow-go/pull/1015) +* fix(parquet/encoding): validate delta binary packed headers by @fallintoplace in [#1012](https://github.com/apache/arrow-go/pull/1012) +* fix(parquet/encoding): reject invalid delta length payloads by @fallintoplace in [#1013](https://github.com/apache/arrow-go/pull/1013) +* fix(parquet/pqarrow): preserve INT96 timestamps across units by @fallintoplace in [#1006](https://github.com/apache/arrow-go/pull/1006) +* fix(parquet/pqarrow): Change interface name for arrow type on read for extension types by @C-Loftus in [#1028](https://github.com/apache/arrow-go/pull/1028) +* fix(arrow/csv): respect timestamp timezones by @fallintoplace in [#1025](https://github.com/apache/arrow-go/pull/1025) +* fix(parquet/compress): avoid closing caller streams for uncompressed wrappers by @fallintoplace in [#1003](https://github.com/apache/arrow-go/pull/1003) +* fix(parquet/compress): allocate uncompressed output for short destinations by @fallintoplace in [#1004](https://github.com/apache/arrow-go/pull/1004) +* fix(internal/utils): keep byteReader position after failed seeks by @fallintoplace in [#1005](https://github.com/apache/arrow-go/pull/1005) +* fix(parquet/file): count repeated skips in logical rows by @fallintoplace in [#1002](https://github.com/apache/arrow-go/pull/1002) +* fix(parquet/metadata): reject malformed fixed-width statistics by @fallintoplace in [#1010](https://github.com/apache/arrow-go/pull/1010) +* fix(arrow/array): skip empty table chunks by @fallintoplace in [#1023](https://github.com/apache/arrow-go/pull/1023) +* fix(arrow/scalar): reject negative array lengths by @fallintoplace in [#1024](https://github.com/apache/arrow-go/pull/1024) +* fix(arrow/array): validate fixed-size list child lengths by @fallintoplace in [#1019](https://github.com/apache/arrow-go/pull/1019) +* fix(parquet/pqarrow): calculate large decimal widths by @fallintoplace in [#1034](https://github.com/apache/arrow-go/pull/1034) +* fix(parquet/schema): reject invalid fixed lengths by @fallintoplace in [#1035](https://github.com/apache/arrow-go/pull/1035) +* fix(parquet/encoding): reject truncated plain boolean data by @fallintoplace in [#1031](https://github.com/apache/arrow-go/pull/1031) +* fix(parquet/metadata): return page index decode errors by @fallintoplace in [#1032](https://github.com/apache/arrow-go/pull/1032) +* fix(parquet/metadata): validate adaptive bloom options by @fallintoplace in [#1033](https://github.com/apache/arrow-go/pull/1033) +* fix(tmpl): report generated output write errors by @fallintoplace in [#1037](https://github.com/apache/arrow-go/pull/1037) +* fix(arrow/array): propagate embedded pointer nullability by @fallintoplace in [#1038](https://github.com/apache/arrow-go/pull/1038) +* fix(flight/flightsql/driver): convert Date32 values by @fallintoplace in [#1039](https://github.com/apache/arrow-go/pull/1039) +* fix(arrow/scalar): support same-width time casts by @fallintoplace in [#1040](https://github.com/apache/arrow-go/pull/1040) +* fix(arrow/scalar): use duration type for units by @fallintoplace in [#1041](https://github.com/apache/arrow-go/pull/1041) +* fix(arrow/array): validate union child upper bound by @fallintoplace in [#1043](https://github.com/apache/arrow-go/pull/1043) +* refactor(arrow/avro): migrate from hamba/avro to twmb/avro by @prochac in [#830](https://github.com/apache/arrow-go/pull/830) +* fix(arrow/array): validate struct child lengths by @fallintoplace in [#1021](https://github.com/apache/arrow-go/pull/1021) +* fix(arrow/cdata): validate imported schema topology by @fallintoplace in [#1046](https://github.com/apache/arrow-go/pull/1046) +* fix(parquet/metadata): validate row-group ordinals in PageIndexReader.WillNeed by @fallintoplace in [#1057](https://github.com/apache/arrow-go/pull/1057) +* fix(arrow/flight): limit descriptors to next payload by @fallintoplace in [#1042](https://github.com/apache/arrow-go/pull/1042) +* fix(parquet): propagate nonce generation failures by @fallintoplace in [#1045](https://github.com/apache/arrow-go/pull/1045) +* fix(parquet/metadata): reject negative row-group indices in Subset by @fallintoplace in [#1055](https://github.com/apache/arrow-go/pull/1055) +* fix(parquet/metadata): use default writer properties when props is nil by @fallintoplace in [#1058](https://github.com/apache/arrow-go/pull/1058) +* fix(parquet/metadata): return schema initialization errors by @fallintoplace in [#1056](https://github.com/apache/arrow-go/pull/1056) +* fix(parquet/pqarrow): propagate level builder errors by @fallintoplace in [#1049](https://github.com/apache/arrow-go/pull/1049) +* fix(arrow/flight/session): clone option values by @fallintoplace in [#1044](https://github.com/apache/arrow-go/pull/1044) +* fix(parquet/variant): validate metadata offset tables by @fallintoplace in [#1063](https://github.com/apache/arrow-go/pull/1063) +* fix(arrow/csv): release Reader record builders by @fallintoplace in [#1059](https://github.com/apache/arrow-go/pull/1059) +* fix(parquet/pqarrow): release partially constructed field readers by @fallintoplace in [#1061](https://github.com/apache/arrow-go/pull/1061) +* fix(arrow/array): make dictionary suffix null indexes relative by @fallintoplace in [#1064](https://github.com/apache/arrow-go/pull/1064) +* fix(parquet/pqarrow): prefer stored schema extension type on read by @twuebi in [#1051](https://github.com/apache/arrow-go/pull/1051) +* fix(arrow/flight/flightsql): avoid double-retaining parameter readers by @fallintoplace in [#1066](https://github.com/apache/arrow-go/pull/1066) +* fix(flightsql): isolate example SQLite databases by @fallintoplace in [#1067](https://github.com/apache/arrow-go/pull/1067) +* fix(arrow): return an owned slice from Schema.FieldsByName by @fallintoplace in [#1069](https://github.com/apache/arrow-go/pull/1069) +* fix(arrow/ipc): return errors for malformed file record batches by @fallintoplace in [#1062](https://github.com/apache/arrow-go/pull/1062) +* fix(flight): return Unauthenticated for invalid unary auth tokens by @fallintoplace in [#1070](https://github.com/apache/arrow-go/pull/1070) +* fix(arrow/array): always release JSON reader builder by @fallintoplace in [#1065](https://github.com/apache/arrow-go/pull/1065) +* fix(flight): accept case-insensitive authorization schemes by @fallintoplace in [#1071](https://github.com/apache/arrow-go/pull/1071) +* fix(parquet): reject short page writes by @fallintoplace in [#1050](https://github.com/apache/arrow-go/pull/1050) +* fix(parquet/file): close file on OpenParquetFile errors by @fallintoplace in [#1053](https://github.com/apache/arrow-go/pull/1053) +* fix(parquet): reject truncated footer signatures by @fallintoplace in [#1074](https://github.com/apache/arrow-go/pull/1074) +* fix(arrow/ipc): reject non-dictionary initial messages by @fallintoplace in [#1054](https://github.com/apache/arrow-go/pull/1054) +* feat(flightsql): allow configuring the driver memory allocator by @fallintoplace in [#1076](https://github.com/apache/arrow-go/pull/1076) +* fix(arrow): avoid retains on invalid chunk construction by @fallintoplace in [#1078](https://github.com/apache/arrow-go/pull/1078) +* fix(parquet): row-align V1 pages with offset indexes by @fallintoplace in [#1075](https://github.com/apache/arrow-go/pull/1075) +* fix(arrow/array): clean up partial table construction by @fallintoplace in [#1077](https://github.com/apache/arrow-go/pull/1077) +* fix(parquet_reader): propagate batch read errors by @fallintoplace in [#1081](https://github.com/apache/arrow-go/pull/1081) +* fix(arrow): isolate schemas created by AddField by @fallintoplace in [#1079](https://github.com/apache/arrow-go/pull/1079) +* fix(arrow/util): avoid protobuf map schema goroutines by @fallintoplace in [#1082](https://github.com/apache/arrow-go/pull/1082) +* fix(array): format date arrays in String by @fallintoplace in [#1068](https://github.com/apache/arrow-go/pull/1068) +* fix(parquet): report initialized row-group columns correctly by @fallintoplace in [#1072](https://github.com/apache/arrow-go/pull/1072) +* fix(parquet): serialize column chunk file offsets by @fallintoplace in [#1073](https://github.com/apache/arrow-go/pull/1073) +* fix(arrow/util): release protobuf record builders by @fallintoplace in [#1084](https://github.com/apache/arrow-go/pull/1084) +* fix(arrow/util): release protobuf enum dictionaries by @fallintoplace in [#1083](https://github.com/apache/arrow-go/pull/1083) +* fix(arrow/avro): stop reader workers before reuse by @fallintoplace in [#1048](https://github.com/apache/arrow-go/pull/1048) +* fix(parquet): restore Arrow duration units from schema metadata by @fallintoplace in [#1095](https://github.com/apache/arrow-go/pull/1095) +* fix(compute): round negative HalfToOdd ties correctly by @fallintoplace in [#1094](https://github.com/apache/arrow-go/pull/1094) +* perf(arrow/bitutil): add SetBitSwap/ClearBitSwap, use in builder.SetNull by @tom-eon in [#1090](https://github.com/apache/arrow-go/pull/1090) +* fix(array): compare sparse union slices at relative offsets by @fallintoplace in [#1096](https://github.com/apache/arrow-go/pull/1096) +* fix(arrow/ipc): make writer failures terminal by @fallintoplace in [#1047](https://github.com/apache/arrow-go/pull/1047) +* feat(parquet): add WithDictionaryCostFallback writer property by @tom-eon in [#1036](https://github.com/apache/arrow-go/pull/1036) +* fix(parquet/pqarrow): release extension arrays after chunking by @fallintoplace in [#1060](https://github.com/apache/arrow-go/pull/1060) +* fix(parquet/encoding): read ahead in the streaming value buffer by @joechenrh in [#937](https://github.com/apache/arrow-go/pull/937) +* fix(parquet/variant): preserve metadata keys when cloning by @fallintoplace in [#1080](https://github.com/apache/arrow-go/pull/1080) +* test(arrow/util): cover zero-field protobuf records by @fallintoplace in [#1085](https://github.com/apache/arrow-go/pull/1085) +* fix(arrow/scalar): preserve metadata in FromScalar by @fallintoplace in [#1100](https://github.com/apache/arrow-go/pull/1100) +* fix(arrow/scalar): preserve binary cast target types by @fallintoplace in [#1101](https://github.com/apache/arrow-go/pull/1101) +* fix(arrow/scalar): support boolean identity casts by @fallintoplace in [#1102](https://github.com/apache/arrow-go/pull/1102) +* fix(arrow/util): handle invalid protobuf Any values by @fallintoplace in [#1086](https://github.com/apache/arrow-go/pull/1086) +* fix(arrow/array): avoid copying initialized extension refcounts by @fallintoplace in [#1089](https://github.com/apache/arrow-go/pull/1089) +* fix(parquet): preserve dictionary index types by @fallintoplace in [#1098](https://github.com/apache/arrow-go/pull/1098) +* fix(parquet_reader): propagate statistics decoding errors by @fallintoplace in [#1087](https://github.com/apache/arrow-go/pull/1087) +* fix(arrow/array): avoid panics cleaning partial tables by @fallintoplace in [#1103](https://github.com/apache/arrow-go/pull/1103) +* fix(arrow/ipc): release reader state on schema errors by @fallintoplace in [#1112](https://github.com/apache/arrow-go/pull/1112) +* fix(arrow/scalar): format extension values from storage scalars by @fallintoplace in [#1111](https://github.com/apache/arrow-go/pull/1111) +* fix(arrow/array): isolate table columns from caller slices by @fallintoplace in [#1105](https://github.com/apache/arrow-go/pull/1105) +* fix(arrow/scalar): preserve exact decimal string casts by @fallintoplace in [#1107](https://github.com/apache/arrow-go/pull/1107) +* fix(arrow/tensor): validate typed constructor data by @fallintoplace in [#1104](https://github.com/apache/arrow-go/pull/1104) +* fix(arrow/scalar): format interval scalars in String by @fallintoplace in [#1115](https://github.com/apache/arrow-go/pull/1115) +* fix(arrow/array): preserve rows for empty-schema JSON input by @fallintoplace in [#1117](https://github.com/apache/arrow-go/pull/1117) +* fix(arrow/util): return protobuf record conversion errors by @fallintoplace in [#1088](https://github.com/apache/arrow-go/pull/1088) +* fix(arrow/scalar): validate FromScalar destinations by @fallintoplace in [#1106](https://github.com/apache/arrow-go/pull/1106) +* fix(arrow/scalar): use wide offsets for large binary scalars by @fallintoplace in [#1108](https://github.com/apache/arrow-go/pull/1108) +* fix(arrow/scalar): apply timestamp zones when casting to dates by @fallintoplace in [#1110](https://github.com/apache/arrow-go/pull/1110) +* fix(arrow/ipc): validate file block framing and lengths by @fallintoplace in [#1097](https://github.com/apache/arrow-go/pull/1097) +* fix(arrow/scalar): release partial children after failed struct extraction by @fallintoplace in [#1118](https://github.com/apache/arrow-go/pull/1118) +* fix(arrow/ipc): return errors for invalid record indexes by @fallintoplace in [#1120](https://github.com/apache/arrow-go/pull/1120) +* fix(parquet): read null fixed-size-list parents by @fallintoplace in [#1099](https://github.com/apache/arrow-go/pull/1099) +* fix(arrow/scalar): release partial children after failed struct conversion by @fallintoplace in [#1114](https://github.com/apache/arrow-go/pull/1114) +* fix(parquet/metadata): return error for missing column metadata by @fallintoplace in [#1137](https://github.com/apache/arrow-go/pull/1137) +* fix(arrow/array): widen dictionary index bounds checks by @fallintoplace in [#1129](https://github.com/apache/arrow-go/pull/1129) +* fix(compute/exprs): avoid empty field reference panic by @fallintoplace in [#1130](https://github.com/apache/arrow-go/pull/1130) +* fix(compute): handle odd temporal multiples by @fallintoplace in [#1119](https://github.com/apache/arrow-go/pull/1119) +* fix(arrow/csv): preserve first row during headerless inference by @fallintoplace in [#1145](https://github.com/apache/arrow-go/pull/1145) +* fix(arrow/array): bound table reader batches by @fallintoplace in [#1146](https://github.com/apache/arrow-go/pull/1146) +* fix(arrow/extensions): compare JSON storage types semantically by @fallintoplace in [#1150](https://github.com/apache/arrow-go/pull/1150) +* fix(arrow/ipc): preserve minSpaceSavings in stream writer by @fallintoplace in [#1143](https://github.com/apache/arrow-go/pull/1143) +* fix(parquet/compress): honor LZ4 raw destination contract by @fallintoplace in [#1147](https://github.com/apache/arrow-go/pull/1147) +* fix(arrow): reject duplicate union type codes by @fallintoplace in [#1131](https://github.com/apache/arrow-go/pull/1131) +* fix(parquet/metadata): handle invalid row-group column indexes by @fallintoplace in [#1125](https://github.com/apache/arrow-go/pull/1125) +* fix(compute/exec): handle missing validity buffers by @fallintoplace in [#1136](https://github.com/apache/arrow-go/pull/1136) +* fix(compute/exec): report ListView buffer counts by @fallintoplace in [#1133](https://github.com/apache/arrow-go/pull/1133) +* fix(arrow): include KeysSorted in map equality by @fallintoplace in [#1134](https://github.com/apache/arrow-go/pull/1134) +* fix(compute): reject non-array-like filters by @fallintoplace in [#1135](https://github.com/apache/arrow-go/pull/1135) +* fix(memory): handle zero-length checked reallocations by @fallintoplace in [#1132](https://github.com/apache/arrow-go/pull/1132) +* fix(parquet/pqarrow): validate reader indexes by @fallintoplace in [#1124](https://github.com/apache/arrow-go/pull/1124) +* fix(arrow/csv): validate custom converter row counts by @fallintoplace in [#1148](https://github.com/apache/arrow-go/pull/1148) +* fix(parquet/encoding): honor BufferWriter offsets by @fallintoplace in [#1144](https://github.com/apache/arrow-go/pull/1144) +* fix(flightsql): stop row streaming after close by @fallintoplace in [#1151](https://github.com/apache/arrow-go/pull/1151) +* fix(flightsql): protect configured authorization metadata by @fallintoplace in [#1153](https://github.com/apache/arrow-go/pull/1153) +* perf(parquet/pqarrow): avoid reflection in integer transfers by @fallintoplace in [#1163](https://github.com/apache/arrow-go/pull/1163) +* fix(flightsql): reject unsupported transaction options by @fallintoplace in [#1152](https://github.com/apache/arrow-go/pull/1152) +* fix(arrow/scalar): handle zero-length scalar arrays by @fallintoplace in [#1116](https://github.com/apache/arrow-go/pull/1116) +* perf(arrow/array): pack validity values in batches by @fallintoplace in [#1185](https://github.com/apache/arrow-go/pull/1185) +* feat(arrow/array): roll back JSON builder state after failed rows by @fallintoplace in [#1113](https://github.com/apache/arrow-go/pull/1113) +* fix(arrow/ipc): avoid deadlock on compression errors by @fallintoplace in [#1142](https://github.com/apache/arrow-go/pull/1142) +* fix(arrow/csv): append null for later parse failures by @fallintoplace in [#1149](https://github.com/apache/arrow-go/pull/1149) +* feat(arrow/array): add ValueAsAny for native Go values by @dgvj-work in [#1170](https://github.com/apache/arrow-go/pull/1170) +* perf(arrow/array): bulk append fixed-size list nulls and empty values by @fallintoplace in [#1183](https://github.com/apache/arrow-go/pull/1183) +* perf(parquet): encode spaced booleans without compaction by @fallintoplace in [#1175](https://github.com/apache/arrow-go/pull/1175) +* fix(parquet/pqarrow): reject decimal overflow in integer columns by @fallintoplace in [#1122](https://github.com/apache/arrow-go/pull/1122) +* fix(arrow/array/arreflect): reject unrepresentable timestamps by @fallintoplace in [#1123](https://github.com/apache/arrow-go/pull/1123) +* feat(compute): add cumulative_sum and cumulative_sum_checked by @fallintoplace in [#1139](https://github.com/apache/arrow-go/pull/1139) +* fix(arrow/ipc): return compression errors from record encoder by @fallintoplace in [#1141](https://github.com/apache/arrow-go/pull/1141) +* perf(arrow/array): bulk append fixed-width empty values by @fallintoplace in [#1180](https://github.com/apache/arrow-go/pull/1180) +* perf(arrow/csv): reserve record builders for chunked reads by @fallintoplace in [#1198](https://github.com/apache/arrow-go/pull/1198) +* perf(arrow/array): compare list values by valid runs by @fallintoplace in [#1184](https://github.com/apache/arrow-go/pull/1184) +* perf(parquet): optimize numeric dictionary insertion by @fallintoplace in [#1178](https://github.com/apache/arrow-go/pull/1178) +* feat(arrow/flight/sql): Add is_update field to ActionCreatePreparedStatementResult by @ennuite in [#732](https://github.com/apache/arrow-go/pull/732) +* feat(compute): add dictionary_encode by @fallintoplace in [#1140](https://github.com/apache/arrow-go/pull/1140) +* perf(parquet): batch plain BYTE_ARRAY encoding by @fallintoplace in [#1193](https://github.com/apache/arrow-go/pull/1193) +* fix(arrow/csv): append null for invalid fixed-size binary values by @fallintoplace in [#1121](https://github.com/apache/arrow-go/pull/1121) +* fix(parquet/pqarrow): reject out-of-range INT96 timestamps by @fallintoplace in [#1128](https://github.com/apache/arrow-go/pull/1128) +* perf(parquet): batch delta binary packed decoding by @fallintoplace in [#1169](https://github.com/apache/arrow-go/pull/1169) +* perf(arrow/array): bulk append list nulls and empty values by @fallintoplace in [#1187](https://github.com/apache/arrow-go/pull/1187) +* perf(parquet/pqarrow): decode booleans directly to bitmaps by @fallintoplace in [#1182](https://github.com/apache/arrow-go/pull/1182) +* perf(parquet/pqarrow): build nullable fixed-size list children directly by @fallintoplace in [#1195](https://github.com/apache/arrow-go/pull/1195) +* perf(internal/hashing): build binary memo offsets in one pass by @fallintoplace in [#1196](https://github.com/apache/arrow-go/pull/1196) +* perf(arrow/compute): take chunked binary values without concatenation by @fallintoplace in [#1199](https://github.com/apache/arrow-go/pull/1199) +* perf(arrow/array): reuse TableReader scratch slices by @fallintoplace in [#1200](https://github.com/apache/arrow-go/pull/1200) +* fix(parquet/pqarrow): reject decimal overflow when writing integers by @fallintoplace in [#1161](https://github.com/apache/arrow-go/pull/1161) +* fix(arrow/array): preserve run-end encoded empty values by @fallintoplace in [#1156](https://github.com/apache/arrow-go/pull/1156) +* fix(compute): preserve caller context during execution by @fallintoplace in [#1154](https://github.com/apache/arrow-go/pull/1154) +* fix(compute): propagate nested expression serialization errors by @fallintoplace in [#1160](https://github.com/apache/arrow-go/pull/1160) +* fix(compute): report FuncMeta for meta functions by @fallintoplace in [#1155](https://github.com/apache/arrow-go/pull/1155) +* perf(arrow/array): optimize bulk appends for binary builders by @fallintoplace in [#1188](https://github.com/apache/arrow-go/pull/1188) +* perf(arrow/array): avoid temporary buffers when concatenating by @fallintoplace in [#1194](https://github.com/apache/arrow-go/pull/1194) +* perf(compute): batch contiguous filter take indices by @fallintoplace in [#1197](https://github.com/apache/arrow-go/pull/1197) +* perf(arrow/ipc): slice already-zero-based offset buffers by @fallintoplace in [#1191](https://github.com/apache/arrow-go/pull/1191) +* perf(parquet): encode RLE levels in batches by @fallintoplace in [#1177](https://github.com/apache/arrow-go/pull/1177) +* fix(arrow): include REE value nullability in type equality by @fallintoplace in [#1158](https://github.com/apache/arrow-go/pull/1158) +* fix(parquet): build bloom filters from dictionary entries by @fallintoplace in [#1164](https://github.com/apache/arrow-go/pull/1164) +* fix(arrow): validate precision in NewDecimalType by @fallintoplace in [#1162](https://github.com/apache/arrow-go/pull/1162) +* perf(arrow/csv): avoid materializing rows before writing by @fallintoplace in [#1190](https://github.com/apache/arrow-go/pull/1190) +* perf(arrow/array): batch run-end encoded appends by @fallintoplace in [#1214](https://github.com/apache/arrow-go/pull/1214) +* fix(arrow/array): validate union JSON type codes by @fallintoplace in [#1157](https://github.com/apache/arrow-go/pull/1157) +* fix(parquet/variant): validate compound value bounds by @fallintoplace in [#1126](https://github.com/apache/arrow-go/pull/1126) +* perf(arrow/array): validate dictionary indices by valid runs by @fallintoplace in [#1181](https://github.com/apache/arrow-go/pull/1181) +* perf(parquet): bound Bloom filter hash batches by @fallintoplace in [#1218](https://github.com/apache/arrow-go/pull/1218) +* perf(arrow/array): optimize bulk appends for struct builders by @fallintoplace in [#1189](https://github.com/apache/arrow-go/pull/1189) +* fix(arrow): return owned field index slices by @fallintoplace in [#1159](https://github.com/apache/arrow-go/pull/1159) +* perf(arrow/array): avoid temporary slices in Diff comparisons by @fallintoplace in [#1239](https://github.com/apache/arrow-go/pull/1239) +* perf(parquet/compress): reuse gzip writers by @fallintoplace in [#1238](https://github.com/apache/arrow-go/pull/1238) +* perf(parquet): reuse fixed-length zero scratch by @fallintoplace in [#1236](https://github.com/apache/arrow-go/pull/1236) +* perf(compute): share logical row maps across aligned sorts by @fallintoplace in [#1234](https://github.com/apache/arrow-go/pull/1234) +* perf(compute): fast-path single-span execution by @fallintoplace in [#1233](https://github.com/apache/arrow-go/pull/1233) +* perf(parquet): reuse boolean spaced scratch by @fallintoplace in [#1231](https://github.com/apache/arrow-go/pull/1231) +* perf(parquet): reuse delta PutSpaced scratch by @fallintoplace in [#1230](https://github.com/apache/arrow-go/pull/1230) +* perf(arrow/array): remove map validity slice by @fallintoplace in [#1228](https://github.com/apache/arrow-go/pull/1228) +* perf(arrow/array): copy same-dictionary indices directly by @fallintoplace in [#1225](https://github.com/apache/arrow-go/pull/1225) +* perf(parquet): reuse dictionary decoder scratch by @fallintoplace in [#1222](https://github.com/apache/arrow-go/pull/1222) +* perf(arrow/array): compare fixed-width values in bulk by @fallintoplace in [#1171](https://github.com/apache/arrow-go/pull/1171) +* perf(arrow/array): compare boolean values by valid runs by @fallintoplace in [#1186](https://github.com/apache/arrow-go/pull/1186) +* perf(arrow/array): avoid temporary buffers for binary concatenation by @fallintoplace in [#1215](https://github.com/apache/arrow-go/pull/1215) +* perf(arrow/array): batch union null and empty appends by @fallintoplace in [#1216](https://github.com/apache/arrow-go/pull/1216) +* perf(parquet): decode levels directly into int16 batches by @fallintoplace in [#1176](https://github.com/apache/arrow-go/pull/1176) +* perf(parquet): batch DELTA_BYTE_ARRAY encoding by @fallintoplace in [#1179](https://github.com/apache/arrow-go/pull/1179) +* perf(parquet/variant): skip metadata construction in Encode by @fallintoplace in [#1217](https://github.com/apache/arrow-go/pull/1217) +* perf(parquet): decode BSS FLBA into contiguous storage by @fallintoplace in [#1172](https://github.com/apache/arrow-go/pull/1172) +* perf(parquet): stop spaced expansion once aligned by @fallintoplace in [#1220](https://github.com/apache/arrow-go/pull/1220) +* perf(arrow/ipc): reuse record encoder scratch by @fallintoplace in [#1221](https://github.com/apache/arrow-go/pull/1221) +* perf(arrow/array): batch dictionary empty appends by @fallintoplace in [#1223](https://github.com/apache/arrow-go/pull/1223) +* perf(compute): reserve binary view cast payload by @fallintoplace in [#1227](https://github.com/apache/arrow-go/pull/1227) +* perf(compute): use stack scratch for null propagation by @fallintoplace in [#1229](https://github.com/apache/arrow-go/pull/1229) +* perf(arrow/array): bulk append BinaryView nulls by @fallintoplace in [#1232](https://github.com/apache/arrow-go/pull/1232) +* perf(arrow/array): use unsafe writes for binary bulk appends by @fallintoplace in [#1240](https://github.com/apache/arrow-go/pull/1240) +* perf(arrow/array): compare union values in runs by @fallintoplace in [#1244](https://github.com/apache/arrow-go/pull/1244) +* perf(arrow/array): batch-pack boolean values by @fallintoplace in [#1237](https://github.com/apache/arrow-go/pull/1237) +* fix(parquet/pqarrow): require Variant value field when writing by @dgvj-work in [#1242](https://github.com/apache/arrow-go/pull/1242) +* perf(arrow/compute): avoid goroutines for serial record filtering by @fallintoplace in [#1245](https://github.com/apache/arrow-go/pull/1245) +* perf(parquet): reuse dictionary RLE decoder across pages by @fallintoplace in [#1247](https://github.com/apache/arrow-go/pull/1247) +* perf(parquet/pqarrow): write record ranges directly by @fallintoplace in [#1248](https://github.com/apache/arrow-go/pull/1248) +* perf(parquet/pqarrow): avoid goroutines for serial struct reads by @fallintoplace in [#1250](https://github.com/apache/arrow-go/pull/1250) +* perf(parquet): decode RLE booleans into bitmaps by @fallintoplace in [#1246](https://github.com/apache/arrow-go/pull/1246) +* feat(compute): add list_element by @fallintoplace in [#1138](https://github.com/apache/arrow-go/pull/1138) +* perf(parquet): reuse DataPageV2 assembly buffer for eager writes by @fallintoplace in [#1192](https://github.com/apache/arrow-go/pull/1192) +* perf(arrow/array): append fixed-width nulls in bulk by @fallintoplace in [#1173](https://github.com/apache/arrow-go/pull/1173) +* perf(arrow/array): concatenate bitmaps directly by @fallintoplace in [#1224](https://github.com/apache/arrow-go/pull/1224) +* perf(arrow/compute): avoid goroutines for serial take by @fallintoplace in [#1226](https://github.com/apache/arrow-go/pull/1226) +* perf(compute): use typed memo insertion for numeric hashes by @fallintoplace in [#1251](https://github.com/apache/arrow-go/pull/1251) +* perf(parquet): decode DELTA_BYTE_ARRAY into contiguous storage by @fallintoplace in [#1252](https://github.com/apache/arrow-go/pull/1252) +* perf(parquet/pqarrow): use synchronous path for serial reads by @fallintoplace in [#1235](https://github.com/apache/arrow-go/pull/1235) +* perf(arrow/array): reuse full chunks in NewChunkedSlice by @fallintoplace in [#1257](https://github.com/apache/arrow-go/pull/1257) +* perf(parquet): reuse boolean Bloom filter hashes by @fallintoplace in [#1258](https://github.com/apache/arrow-go/pull/1258) +* perf(arrow/ipc): release dictionary payloads eagerly by @fallintoplace in [#1259](https://github.com/apache/arrow-go/pull/1259) +* perf(parquet): SIMD BYTE_STREAM_SPLIT encoding by @fallintoplace in [#1260](https://github.com/apache/arrow-go/pull/1260) +* perf(parquet): reuse DELTA byte-array scratch by @fallintoplace in [#1261](https://github.com/apache/arrow-go/pull/1261) +* perf(parquet/pqarrow): write FixedSizeBinary values directly by @fallintoplace in [#1263](https://github.com/apache/arrow-go/pull/1263) +* feat(extensions): add VariantGet for path extraction from variant arrays by @nssalian in [#1206](https://github.com/apache/arrow-go/pull/1206) +* perf(parquet): use typed memo insertion for byte-array dictionaries by @derekperkins in [#1272](https://github.com/apache/arrow-go/pull/1272) +* fix(arrow/array): clamp run ends when concatenating a sliced RunEndEncoded array by @winklemad in [#1219](https://github.com/apache/arrow-go/pull/1219) +* fix(parquet): guard byte-stream-split encoder release against never-flushed buffer by @tom-eon in [#1273](https://github.com/apache/arrow-go/pull/1273) +* perf(bitutil): add ARM64 NEON bitmap operations by @fallintoplace in [#1264](https://github.com/apache/arrow-go/pull/1264) +* perf(arrow/array): compare binary values by valid runs by @fallintoplace in [#1174](https://github.com/apache/arrow-go/pull/1174) +* perf(compute): add ARM64 NEON arithmetic kernels by @fallintoplace in [#1265](https://github.com/apache/arrow-go/pull/1265) +* perf(parquet/compress): reuse Brotli writers across encodes by @fallintoplace in [#1254](https://github.com/apache/arrow-go/pull/1254) +* perf(arrow/array): skip full-range SliceEqual slices by @fallintoplace in [#1249](https://github.com/apache/arrow-go/pull/1249) +* perf(parquet): vectorize fixed-width dictionary materialization by @fallintoplace in [#1269](https://github.com/apache/arrow-go/pull/1269) +* perf(compute): add ARM64 NEON comparison kernels by @fallintoplace in [#1266](https://github.com/apache/arrow-go/pull/1266) +* perf(parquet/pqarrow): write binary values from Arrow offsets by @fallintoplace in [#1262](https://github.com/apache/arrow-go/pull/1262) +* fix(arrow/extensions): reject Null Variant typed_value by @dgvj-work in [#1243](https://github.com/apache/arrow-go/pull/1243) +* fix(arrow/extensions): use canonical Variant extension name by @dgvj-work in [#1241](https://github.com/apache/arrow-go/pull/1241) +* chore: bump version number by @zeroshade in [#1276](https://github.com/apache/arrow-go/pull/1276) +* test(parquet): reduce macOS race test memory pressure by @zeroshade in [#1275](https://github.com/apache/arrow-go/pull/1275) +* fix(parquet): explicitly release bloom builders by @zeroshade in [#1280](https://github.com/apache/arrow-go/pull/1280) +* ci: retry and cache dependency downloads by @zeroshade in [#1279](https://github.com/apache/arrow-go/pull/1279) +* fix(compute): discard results after cancellation by @zeroshade in [#1278](https://github.com/apache/arrow-go/pull/1278) +* ci: use bundled GTest for conda integration by @zeroshade in [#1277](https://github.com/apache/arrow-go/pull/1277) +* fix(parquet): stop BYTE_STREAM_SPLIT FLBA spaced decode aliasing output by @zeroshade in [#1256](https://github.com/apache/arrow-go/pull/1256) +* perf(compute): vectorize mixed int32 filters with AVX2 by @fallintoplace in [#1284](https://github.com/apache/arrow-go/pull/1284) +* perf(compute): add ARM64 NEON constant multiplication by @fallintoplace in [#1268](https://github.com/apache/arrow-go/pull/1268) +* perf(compute): vectorize numeric to boolean casts by @fallintoplace in [#1282](https://github.com/apache/arrow-go/pull/1282) +* fix(arrow/scalar): align timestamp timezone parsing by @fallintoplace in [#1109](https://github.com/apache/arrow-go/pull/1109) +* perf(internal/utils): add ARM64 NEON min/max for 8- and 16-bit integers by @fallintoplace in [#1281](https://github.com/apache/arrow-go/pull/1281) +* perf(arrow/array): SIMD-pack boolean slices into bitmaps by @fallintoplace in [#1283](https://github.com/apache/arrow-go/pull/1283) +* chore: fix failing build tags by @zeroshade in [#1285](https://github.com/apache/arrow-go/pull/1285) +* perf(compute): SIMD-compact fragmented filters into take indices by @fallintoplace in [#1287](https://github.com/apache/arrow-go/pull/1287) +* perf(compute): add ARM64 NEON comparisons for 8- and 16-bit integers by @fallintoplace in [#1288](https://github.com/apache/arrow-go/pull/1288) +* perf(compute): add ARM64 NEON compression for null-free 32-bit fixed-width filters by @fallintoplace in [#1286](https://github.com/apache/arrow-go/pull/1286) + +### New Contributors +* @C-Loftus made their first contribution in [#960](https://github.com/apache/arrow-go/pull/960) +* @tom-eon made their first contribution in [#1027](https://github.com/apache/arrow-go/pull/1027) +* @prochac made their first contribution in [#830](https://github.com/apache/arrow-go/pull/830) +* @dgvj-work made their first contribution in [#1170](https://github.com/apache/arrow-go/pull/1170) +* @ennuite made their first contribution in [#732](https://github.com/apache/arrow-go/pull/732) +* @derekperkins made their first contribution in [#1272](https://github.com/apache/arrow-go/pull/1272) +* @winklemad made their first contribution in [#1219](https://github.com/apache/arrow-go/pull/1219) + +**Full Changelog**: https://github.com/apache/arrow-go/compare/v18.7.0...v18.8.0 From 3b04db8c0b6c538e90cf44babf965a829ee55909 Mon Sep 17 00:00:00 2001 From: Matt Topol Date: Mon, 16 Nov 2026 13:08:17 -0500 Subject: [PATCH 2/3] add highlights and correct changelog for Arrow Go 18.9.0 - replace the stale v18.8.0 changelog (gh release view picked the latest release) with v18.8.0..v18.9.0 changes - add Arrow and Parquet highlights with PR-reported benchmark figures - group selected changes by Arrow, Compute, Parquet and Performance - fix New Contributors list and Full Changelog compare range --- _posts/2026-11-16-arrow-go-18.9.0.md | 541 ++++++++++++--------------- 1 file changed, 247 insertions(+), 294 deletions(-) diff --git a/_posts/2026-11-16-arrow-go-18.9.0.md b/_posts/2026-11-16-arrow-go-18.9.0.md index 2d9cf3081718..27b9137e861d 100644 --- a/_posts/2026-11-16-arrow-go-18.9.0.md +++ b/_posts/2026-11-16-arrow-go-18.9.0.md @@ -59,301 +59,254 @@ $ git shortlog -sn v18.8.0..v18.9.0 1 ump45nose ``` +## Highlights + +- Go 1.26 is now the minimum supported Go version [#1326](https://github.com/apache/arrow-go/pull/1326) + +### Arrow + +#### Features + +- Added a scalar aggregate framework to the compute package, starting with the `count` and `sum` kernels [#1336](https://github.com/apache/arrow-go/pull/1336) +- Added the `starts_with` compute function, including `string_view`/`binary_view` inputs, `IgnoreCase`, and the Substrait `starts_with` mapping used by +iceberg-go prefix row filters [#1448](https://github.com/apache/arrow-go/pull/1448), [#1467](https://github.com/apache/arrow-go/pull/1467), [#1468](https://github.com/apache/arrow-go/pull/1468) +- Added `SortKey.FloatTotalOrder` to sort floats by IEEE 754 totalOrder, distinguishing `-0`/`+0` and signed NaNs [#1430](https://github.com/apache/arrow-go/pull/1430) +- Added `Decimal32`, `Decimal64`, `StringView` and `BinaryView` scalar types [#1501](https://github.com/apache/arrow-go/pull/1501), [#1490](https://github.com/apache/arrow-go/pull/1490) +- Added `ipc.WithZstdLevel` to choose the zstd compression level of IPC writers [#1376](https://github.com/apache/arrow-go/pull/1376) +- Added `Reader.ContinueFrom` to resume an IPC reader on a stream that arrives as several blobs, a pattern that stopped working in v18.8.0 [#1346](https://github.com/apache/arrow-go/pull/1346) +- The Flight SQL `database/sql` driver now accepts `grpc+tcp://`, `grpc+tls://` and `grpc+unix://` DSNs [#1420](https://github.com/apache/arrow-go/pull/1420) +- The JSON and CSV readers expose their input offset to help locate decoding errors [#1342](https://github.com/apache/arrow-go/pull/1342) + +#### Bug Fixes + +- `is_null`, `is_not_null` and `not` returned wrong masks for sliced arrays, dictionaries with null values, and null, union and run-end encoded arrays, so +filters could silently keep or drop the wrong rows. `is_null` now also honors `NullOptions.NanIsNull` [#1456](https://github.com/apache/arrow-go/pull/1456), +[#1458](https://github.com/apache/arrow-go/pull/1458), [#1461](https://github.com/apache/arrow-go/pull/1461), [#1465](https://github.com/apache/arrow-go/pull/1465), [#1488](https://github.com/apache/arrow-go/pull/1488) +- `NewDatumWithoutOwning` now returns a non-owning Datum; previously `Release()` freed the caller's memory [#1377](https://github.com/apache/arrow-go/pull/1377) +- `and_kleene` and `or_kleene` returned `false` instead of null when `ExecCtx.ChunkSize` was smaller than the input [#1306](https://github.com/apache/arrow-go/pull/1306) +- A cancelled `CallFunction` no longer returns a result with a nil error, and the default `NumParallel` follows the cgroup-aware `GOMAXPROCS` in +containers [#1410](https://github.com/apache/arrow-go/pull/1410), [#1384](https://github.com/apache/arrow-go/pull/1384) +- Fixed generated assembly that clobbered the frame pointer or Go's reserved registers, breaking profilers and frame-pointer unwinders, in the amd64 compute +kernels and the Parquet bit-unpacking and Bloom filter kernels [#1432](https://github.com/apache/arrow-go/pull/1432), [#1300](https://github.com/apache/arrow-go/pull/1300), [#1379](https://github.com/apache/arrow-go/pull/1379) +- Decimal rescaling no longer panics or returns wrong values, infinite decimal strings are rejected, and scales past `MaxScale` fail clearly [#1307](https://github.com/apache/arrow-go/pull/1307), +[#1409](https://github.com/apache/arrow-go/pull/1409), [#1513](https://github.com/apache/arrow-go/pull/1513), [#1514](https://github.com/apache/arrow-go/pull/1514), [#1515](https://github.com/apache/arrow-go/pull/1515) +- The C Data interface exports extension arrays as their storage and imports zero-length fixed-size lists, such as those returned by DataFusion [#1526](https://github.com/apache/arrow-go/pull/1526), +[#1308](https://github.com/apache/arrow-go/pull/1308) +- `arrow.TypeEqual` now compares `LargeListType` like `ListType`, so `array.Concatenate` no longer rejects identical large list types [#1440](https://github.com/apache/arrow-go/pull/1440) +- `NewMappedFileReader` no longer panics on dictionary fields, and Flight SQL `ExecuteIngest` sends the schema for empty inputs [#1365](https://github.com/apache/arrow-go/pull/1365), +[#1433](https://github.com/apache/arrow-go/pull/1433) +- The JSON record reader no longer swallows an unexpected EOF on truncated input [#1341](https://github.com/apache/arrow-go/pull/1341) + +#### Performance Improvements + +Performance remains a major theme: 51 of the merged pull requests are dedicated +optimizations. Unless noted otherwise, the figures below are the PR-reported +benchmarks on an Apple M1 Pro. + +- Bulk array comparisons: comparing an array without a validity bitmap against one with a materialized bitmap is up to 414x faster for booleans (8.8ms to 21.3us +over 1M values) and 15.3x for int64, and all-valid list equality over 65,536 rows drops from 17.8ms to 0.197ms [#1327](https://github.com/apache/arrow-go/pull/1327), +[#1330](https://github.com/apache/arrow-go/pull/1330), [#1347](https://github.com/apache/arrow-go/pull/1347) +- SIMD filtering now covers 8-, 16- and 64-bit types: 1M-element alternating int64 filters go from 3.97ms to 0.45ms (8.7x), int16 from 4.23ms to 225us, and +8-bit alternating masks run 22-24x faster. AVX2 take-index compaction adds up to 4.5x on AMD EPYC [#1311](https://github.com/apache/arrow-go/pull/1311), +[#1349](https://github.com/apache/arrow-go/pull/1349), [#1350](https://github.com/apache/arrow-go/pull/1350), [#1348](https://github.com/apache/arrow-go/pull/1348) +- `SortIndices` radix-sorts single-chunk, null-free integer columns: 65,536 random int64 values sort in 1.31ms instead of 12.3ms [#1391](https://github.com/apache/arrow-go/pull/1391) +- Dictionary-heavy workloads: `unique` over 256 chunks with changing dictionaries is 36.6x faster (60.0ms to 1.64ms) with 98.5% fewer bytes allocated, +concatenating chunks that share a dictionary drops from 82.4us to 3.7us, and builders reuse unchanged dictionaries, cutting the per-row cost of +small-batch dictionary reads through `pqarrow` by up to 93% [#1395](https://github.com/apache/arrow-go/pull/1395), [#1403](https://github.com/apache/arrow-go/pull/1403), [#1444](https://github.com/apache/arrow-go/pull/1444) +- Bitmap utilities: `AppendBools` takes 88% less time (geometric mean), unaligned bitmap operations are up to 11.4x faster, and misaligned bitmap equality and +unaligned transfers improve 6.5x and 5.4x (geometric means) [#1402](https://github.com/apache/arrow-go/pull/1402), [#1437](https://github.com/apache/arrow-go/pull/1437), +[#1414](https://github.com/apache/arrow-go/pull/1414), [#1416](https://github.com/apache/arrow-go/pull/1416) +- IPC zstd compression: pooled decoders cut decompression time by 69-78% for buffers up to 256KiB per column, and pooled encoders take a writer-per-record +workload from 246.4us to 4.5us and from about 2.2MiB to 7.1KiB allocated per op (Apple M2 Pro) [#1331](https://github.com/apache/arrow-go/pull/1331), [#1429](https://github.com/apache/arrow-go/pull/1429) +- Full-file CSV reads stream into builders: 1.69x faster with 71% fewer bytes allocated [#1393](https://github.com/apache/arrow-go/pull/1393) + +### Parquet + +#### Features + +- Added opt-in ALP (Adaptive Lossless floating-Point) encoding for FLOAT and DOUBLE columns, interoperable with Arrow C++. Writers enable it with +`WithALPEncoding`; reading needs no option [#1289](https://github.com/apache/arrow-go/pull/1289) +- Added an optional `parquet/encryption` package with KMS support, so files encrypted by pyarrow's `CryptoFactory` (PKMT1 key material, single or double +wrapping) can be read through `KmsKeyRetriever` [#1478](https://github.com/apache/arrow-go/pull/1478) +- Write geospatial statistics (bounding box and geometry types) for GEOMETRY column chunks, and `null_count` statistics for GEOMETRY and GEOGRAPHY [#1504](https://github.com/apache/arrow-go/pull/1504), +[#1533](https://github.com/apache/arrow-go/pull/1533) + +#### Bug Fixes + +- Encryption interoperability: read encrypted files written by parquet-mr (Spark, Hive, Iceberg on parquet-java), preserve `AES_GCM_CTR_V1` in plaintext +footers, and return an error instead of panicking on columns without a key [#1390](https://github.com/apache/arrow-go/pull/1390), [#1412](https://github.com/apache/arrow-go/pull/1412), [#1406](https://github.com/apache/arrow-go/pull/1406) +- `pqarrow` wrote wrong data for sliced `FixedSizeList` columns and for `FixedSizeList` columns spanning more than one row group [#1382](https://github.com/apache/arrow-go/pull/1382) +- Encoders no longer carry stale state across pages: RLE boolean columns repeated earlier values on later pages, plain boolean and delta encoders wrote +pre-`Reset` state, and all-null `DELTA_BINARY_PACKED` pages were padded with zeros [#1539](https://github.com/apache/arrow-go/pull/1539), [#1545](https://github.com/apache/arrow-go/pull/1545), +[#1546](https://github.com/apache/arrow-go/pull/1546), [#1400](https://github.com/apache/arrow-go/pull/1400) +- Statistics: V2 data pages no longer report zero nulls without page statistics, byte array statistics written from Arrow count empty strings, and +`UpdateSpaced` counts non-null values from the validity bitmap [#1543](https://github.com/apache/arrow-go/pull/1543), [#1527](https://github.com/apache/arrow-go/pull/1527), [#1367](https://github.com/apache/arrow-go/pull/1367) +- Fixed a memory leak in DELTA decoders, and dictionary-encoded boolean columns now return an error instead of panicking [#1394](https://github.com/apache/arrow-go/pull/1394), +[#1547](https://github.com/apache/arrow-go/pull/1547) + +#### Performance Improvements + +- Reading dictionary-encoded columns allocates the dictionary up front again, fixing an allocation regression introduced in v18.8.0 [#1473](https://github.com/apache/arrow-go/pull/1473) +- Bloom filter hashing pools its scratch buffers: hashing 64 values drops from 5.52us to 311ns, and dictionary Bloom filter writes take up to 32% less time +(Intel Core Ultra 7 155H) [#1378](https://github.com/apache/arrow-go/pull/1378) +- Level and boolean decoding: nullable boolean bitmap expansion takes 85% less time (geometric mean), 1-bit packed levels decode in 55-60% less time, all-true and +all-false RLE booleans decode 66-98% faster (amd64), and narrow RLE literal levels encode 2.25-2.61x faster [#1405](https://github.com/apache/arrow-go/pull/1405), +[#1404](https://github.com/apache/arrow-go/pull/1404), [#1417](https://github.com/apache/arrow-go/pull/1417), [#1419](https://github.com/apache/arrow-go/pull/1419) +- Dictionary encoding: indices are encoded in RLE batches (65,536 constant indices drop from 187.7us to 26.0us), and FLBA and Int96 dictionary insertion is +29-33% faster with up to 50% fewer allocations [#1328](https://github.com/apache/arrow-go/pull/1328), [#1312](https://github.com/apache/arrow-go/pull/1312) +- DELTA encodings: constant `DELTA_BINARY_PACKED` blocks decode 2.1x faster, `DELTA_LENGTH_BYTE_ARRAY` decodes 1.62-2.00x and discards 2.43-2.74x faster, +`DELTA_BYTE_ARRAY` prefix matching is up to 4.7x faster (AMD EPYC), and its discard path drops from 65,535 allocations to none [#1396](https://github.com/apache/arrow-go/pull/1396), +[#1415](https://github.com/apache/arrow-go/pull/1415), [#1388](https://github.com/apache/arrow-go/pull/1388), [#1321](https://github.com/apache/arrow-go/pull/1321) +- Reads: optional columns use dense decoding for batches without nulls (up to 45% faster per row on AMD EPYC), and serial reads build struct child readers +without goroutines, 37-43% faster (Apple M4 Max) [#1447](https://github.com/apache/arrow-go/pull/1447), [#1434](https://github.com/apache/arrow-go/pull/1434) + ## Changelog -### What's Changed -* feat(parquet/schema): Wire up logical type functionality for writing geo types by @C-Loftus in [#960](https://github.com/apache/arrow-go/pull/960) -* chore(dev/release): filter out non numeric go releases (rc's and beta's) when finding latest go by @raulcd in [#968](https://github.com/apache/arrow-go/pull/968) -* ci: fix release workflow discussions by @zeroshade in [#980](https://github.com/apache/arrow-go/pull/980) -* ci: fix lint on release workflow by @zeroshade in [#982](https://github.com/apache/arrow-go/pull/982) -* fix: remove superfluous struct field index compare by @Willem-J-an in [#970](https://github.com/apache/arrow-go/pull/970) -* fix(ipc): reject truncated message frames by @fallintoplace in [#991](https://github.com/apache/arrow-go/pull/991) -* fix(internal/utils): handle reader reset and short peeks by @fallintoplace in [#987](https://github.com/apache/arrow-go/pull/987) -* fix(arrow/ipc): propagate CLI stream errors by @fallintoplace in [#986](https://github.com/apache/arrow-go/pull/986) -* fix(arrow/flight): avoid blocking after stream cancellation by @fallintoplace in [#989](https://github.com/apache/arrow-go/pull/989) -* fix(parquet/variant): handle empty metadata keys by @fallintoplace in [#985](https://github.com/apache/arrow-go/pull/985) -* fix(flight/flightsql/example): propagate row iteration errors by @fallintoplace in [#990](https://github.com/apache/arrow-go/pull/990) -* fix(parquet): reject truncated scalar bit reads by @fallintoplace in [#944](https://github.com/apache/arrow-go/pull/944) -* fix: Fix arm64 assembly stack frame manipulation by @nsrip-dd in [#984](https://github.com/apache/arrow-go/pull/984) -* fix(arrow/array): support zero-column tables by @fallintoplace in [#988](https://github.com/apache/arrow-go/pull/988) -* docs(parquet): Add thrift generation contribution info by @C-Loftus in [#971](https://github.com/apache/arrow-go/pull/971) -* fix(array): reject mismatched known null counts during full validation by @fallintoplace in [#993](https://github.com/apache/arrow-go/pull/993) -* fix(arrow/array): `Resize(0)` on non-empty builder causes memory leak by @serramatutu in [#995](https://github.com/apache/arrow-go/pull/995) -* feat(parquet/pqarrow): Read support for extension types / geo by @C-Loftus in [#969](https://github.com/apache/arrow-go/pull/969) -* fix(array): validate run-end encoded invariants by @fallintoplace in [#992](https://github.com/apache/arrow-go/pull/992) -* fix(array): validate binary_view and string_view payloads by @fallintoplace in [#994](https://github.com/apache/arrow-go/pull/994) -* fix(parquet): wrap DELTA_BINARY_PACKED deltas at the physical type width by @tom-eon in [#1027](https://github.com/apache/arrow-go/pull/1027) -* fix(arrow/compute): use is_null/is_not_null in expressions by @fallintoplace in [#997](https://github.com/apache/arrow-go/pull/997) -* fix(arrow/ipc): reject NaN minSpaceSavings values by @fallintoplace in [#998](https://github.com/apache/arrow-go/pull/998) -* fix(arrow/tensor): default nil dimension names to empty strings by @fallintoplace in [#999](https://github.com/apache/arrow-go/pull/999) -* fix(arrow/scalar): allow null extension scalars to keep null storage by @fallintoplace in [#1000](https://github.com/apache/arrow-go/pull/1000) -* fix(arrow/scalar): handle nil child values in run-end scalars by @fallintoplace in [#1001](https://github.com/apache/arrow-go/pull/1001) -* fix(arrow/array): ignore zero-count union appends by @fallintoplace in [#1026](https://github.com/apache/arrow-go/pull/1026) -* fix(arrow/csv): preserve decimal precision by @fallintoplace in [#1022](https://github.com/apache/arrow-go/pull/1022) -* fix(parquet/metadata): compare bloom filter caps in bits by @fallintoplace in [#996](https://github.com/apache/arrow-go/pull/996) -* fix(parquet/pqarrow): reject timestamp coercion overflow by @fallintoplace in [#1008](https://github.com/apache/arrow-go/pull/1008) -* fix(parquet/file): reject records starting with nonzero repetition levels by @fallintoplace in [#1009](https://github.com/apache/arrow-go/pull/1009) -* fix(parquet/variant): reject truncated scalar values by @fallintoplace in [#1017](https://github.com/apache/arrow-go/pull/1017) -* fix(parquet/encoding): reject invalid delta byte prefixes by @fallintoplace in [#1014](https://github.com/apache/arrow-go/pull/1014) -* fix(parquet/metadata): preserve infinite floating-point statistics by @fallintoplace in [#1007](https://github.com/apache/arrow-go/pull/1007) -* fix(parquet/encoding): bound lazy dictionary growth by @fallintoplace in [#1011](https://github.com/apache/arrow-go/pull/1011) -* fix(parquet/encoding): respect declared RLE payload lengths by @fallintoplace in [#1015](https://github.com/apache/arrow-go/pull/1015) -* fix(parquet/encoding): validate delta binary packed headers by @fallintoplace in [#1012](https://github.com/apache/arrow-go/pull/1012) -* fix(parquet/encoding): reject invalid delta length payloads by @fallintoplace in [#1013](https://github.com/apache/arrow-go/pull/1013) -* fix(parquet/pqarrow): preserve INT96 timestamps across units by @fallintoplace in [#1006](https://github.com/apache/arrow-go/pull/1006) -* fix(parquet/pqarrow): Change interface name for arrow type on read for extension types by @C-Loftus in [#1028](https://github.com/apache/arrow-go/pull/1028) -* fix(arrow/csv): respect timestamp timezones by @fallintoplace in [#1025](https://github.com/apache/arrow-go/pull/1025) -* fix(parquet/compress): avoid closing caller streams for uncompressed wrappers by @fallintoplace in [#1003](https://github.com/apache/arrow-go/pull/1003) -* fix(parquet/compress): allocate uncompressed output for short destinations by @fallintoplace in [#1004](https://github.com/apache/arrow-go/pull/1004) -* fix(internal/utils): keep byteReader position after failed seeks by @fallintoplace in [#1005](https://github.com/apache/arrow-go/pull/1005) -* fix(parquet/file): count repeated skips in logical rows by @fallintoplace in [#1002](https://github.com/apache/arrow-go/pull/1002) -* fix(parquet/metadata): reject malformed fixed-width statistics by @fallintoplace in [#1010](https://github.com/apache/arrow-go/pull/1010) -* fix(arrow/array): skip empty table chunks by @fallintoplace in [#1023](https://github.com/apache/arrow-go/pull/1023) -* fix(arrow/scalar): reject negative array lengths by @fallintoplace in [#1024](https://github.com/apache/arrow-go/pull/1024) -* fix(arrow/array): validate fixed-size list child lengths by @fallintoplace in [#1019](https://github.com/apache/arrow-go/pull/1019) -* fix(parquet/pqarrow): calculate large decimal widths by @fallintoplace in [#1034](https://github.com/apache/arrow-go/pull/1034) -* fix(parquet/schema): reject invalid fixed lengths by @fallintoplace in [#1035](https://github.com/apache/arrow-go/pull/1035) -* fix(parquet/encoding): reject truncated plain boolean data by @fallintoplace in [#1031](https://github.com/apache/arrow-go/pull/1031) -* fix(parquet/metadata): return page index decode errors by @fallintoplace in [#1032](https://github.com/apache/arrow-go/pull/1032) -* fix(parquet/metadata): validate adaptive bloom options by @fallintoplace in [#1033](https://github.com/apache/arrow-go/pull/1033) -* fix(tmpl): report generated output write errors by @fallintoplace in [#1037](https://github.com/apache/arrow-go/pull/1037) -* fix(arrow/array): propagate embedded pointer nullability by @fallintoplace in [#1038](https://github.com/apache/arrow-go/pull/1038) -* fix(flight/flightsql/driver): convert Date32 values by @fallintoplace in [#1039](https://github.com/apache/arrow-go/pull/1039) -* fix(arrow/scalar): support same-width time casts by @fallintoplace in [#1040](https://github.com/apache/arrow-go/pull/1040) -* fix(arrow/scalar): use duration type for units by @fallintoplace in [#1041](https://github.com/apache/arrow-go/pull/1041) -* fix(arrow/array): validate union child upper bound by @fallintoplace in [#1043](https://github.com/apache/arrow-go/pull/1043) -* refactor(arrow/avro): migrate from hamba/avro to twmb/avro by @prochac in [#830](https://github.com/apache/arrow-go/pull/830) -* fix(arrow/array): validate struct child lengths by @fallintoplace in [#1021](https://github.com/apache/arrow-go/pull/1021) -* fix(arrow/cdata): validate imported schema topology by @fallintoplace in [#1046](https://github.com/apache/arrow-go/pull/1046) -* fix(parquet/metadata): validate row-group ordinals in PageIndexReader.WillNeed by @fallintoplace in [#1057](https://github.com/apache/arrow-go/pull/1057) -* fix(arrow/flight): limit descriptors to next payload by @fallintoplace in [#1042](https://github.com/apache/arrow-go/pull/1042) -* fix(parquet): propagate nonce generation failures by @fallintoplace in [#1045](https://github.com/apache/arrow-go/pull/1045) -* fix(parquet/metadata): reject negative row-group indices in Subset by @fallintoplace in [#1055](https://github.com/apache/arrow-go/pull/1055) -* fix(parquet/metadata): use default writer properties when props is nil by @fallintoplace in [#1058](https://github.com/apache/arrow-go/pull/1058) -* fix(parquet/metadata): return schema initialization errors by @fallintoplace in [#1056](https://github.com/apache/arrow-go/pull/1056) -* fix(parquet/pqarrow): propagate level builder errors by @fallintoplace in [#1049](https://github.com/apache/arrow-go/pull/1049) -* fix(arrow/flight/session): clone option values by @fallintoplace in [#1044](https://github.com/apache/arrow-go/pull/1044) -* fix(parquet/variant): validate metadata offset tables by @fallintoplace in [#1063](https://github.com/apache/arrow-go/pull/1063) -* fix(arrow/csv): release Reader record builders by @fallintoplace in [#1059](https://github.com/apache/arrow-go/pull/1059) -* fix(parquet/pqarrow): release partially constructed field readers by @fallintoplace in [#1061](https://github.com/apache/arrow-go/pull/1061) -* fix(arrow/array): make dictionary suffix null indexes relative by @fallintoplace in [#1064](https://github.com/apache/arrow-go/pull/1064) -* fix(parquet/pqarrow): prefer stored schema extension type on read by @twuebi in [#1051](https://github.com/apache/arrow-go/pull/1051) -* fix(arrow/flight/flightsql): avoid double-retaining parameter readers by @fallintoplace in [#1066](https://github.com/apache/arrow-go/pull/1066) -* fix(flightsql): isolate example SQLite databases by @fallintoplace in [#1067](https://github.com/apache/arrow-go/pull/1067) -* fix(arrow): return an owned slice from Schema.FieldsByName by @fallintoplace in [#1069](https://github.com/apache/arrow-go/pull/1069) -* fix(arrow/ipc): return errors for malformed file record batches by @fallintoplace in [#1062](https://github.com/apache/arrow-go/pull/1062) -* fix(flight): return Unauthenticated for invalid unary auth tokens by @fallintoplace in [#1070](https://github.com/apache/arrow-go/pull/1070) -* fix(arrow/array): always release JSON reader builder by @fallintoplace in [#1065](https://github.com/apache/arrow-go/pull/1065) -* fix(flight): accept case-insensitive authorization schemes by @fallintoplace in [#1071](https://github.com/apache/arrow-go/pull/1071) -* fix(parquet): reject short page writes by @fallintoplace in [#1050](https://github.com/apache/arrow-go/pull/1050) -* fix(parquet/file): close file on OpenParquetFile errors by @fallintoplace in [#1053](https://github.com/apache/arrow-go/pull/1053) -* fix(parquet): reject truncated footer signatures by @fallintoplace in [#1074](https://github.com/apache/arrow-go/pull/1074) -* fix(arrow/ipc): reject non-dictionary initial messages by @fallintoplace in [#1054](https://github.com/apache/arrow-go/pull/1054) -* feat(flightsql): allow configuring the driver memory allocator by @fallintoplace in [#1076](https://github.com/apache/arrow-go/pull/1076) -* fix(arrow): avoid retains on invalid chunk construction by @fallintoplace in [#1078](https://github.com/apache/arrow-go/pull/1078) -* fix(parquet): row-align V1 pages with offset indexes by @fallintoplace in [#1075](https://github.com/apache/arrow-go/pull/1075) -* fix(arrow/array): clean up partial table construction by @fallintoplace in [#1077](https://github.com/apache/arrow-go/pull/1077) -* fix(parquet_reader): propagate batch read errors by @fallintoplace in [#1081](https://github.com/apache/arrow-go/pull/1081) -* fix(arrow): isolate schemas created by AddField by @fallintoplace in [#1079](https://github.com/apache/arrow-go/pull/1079) -* fix(arrow/util): avoid protobuf map schema goroutines by @fallintoplace in [#1082](https://github.com/apache/arrow-go/pull/1082) -* fix(array): format date arrays in String by @fallintoplace in [#1068](https://github.com/apache/arrow-go/pull/1068) -* fix(parquet): report initialized row-group columns correctly by @fallintoplace in [#1072](https://github.com/apache/arrow-go/pull/1072) -* fix(parquet): serialize column chunk file offsets by @fallintoplace in [#1073](https://github.com/apache/arrow-go/pull/1073) -* fix(arrow/util): release protobuf record builders by @fallintoplace in [#1084](https://github.com/apache/arrow-go/pull/1084) -* fix(arrow/util): release protobuf enum dictionaries by @fallintoplace in [#1083](https://github.com/apache/arrow-go/pull/1083) -* fix(arrow/avro): stop reader workers before reuse by @fallintoplace in [#1048](https://github.com/apache/arrow-go/pull/1048) -* fix(parquet): restore Arrow duration units from schema metadata by @fallintoplace in [#1095](https://github.com/apache/arrow-go/pull/1095) -* fix(compute): round negative HalfToOdd ties correctly by @fallintoplace in [#1094](https://github.com/apache/arrow-go/pull/1094) -* perf(arrow/bitutil): add SetBitSwap/ClearBitSwap, use in builder.SetNull by @tom-eon in [#1090](https://github.com/apache/arrow-go/pull/1090) -* fix(array): compare sparse union slices at relative offsets by @fallintoplace in [#1096](https://github.com/apache/arrow-go/pull/1096) -* fix(arrow/ipc): make writer failures terminal by @fallintoplace in [#1047](https://github.com/apache/arrow-go/pull/1047) -* feat(parquet): add WithDictionaryCostFallback writer property by @tom-eon in [#1036](https://github.com/apache/arrow-go/pull/1036) -* fix(parquet/pqarrow): release extension arrays after chunking by @fallintoplace in [#1060](https://github.com/apache/arrow-go/pull/1060) -* fix(parquet/encoding): read ahead in the streaming value buffer by @joechenrh in [#937](https://github.com/apache/arrow-go/pull/937) -* fix(parquet/variant): preserve metadata keys when cloning by @fallintoplace in [#1080](https://github.com/apache/arrow-go/pull/1080) -* test(arrow/util): cover zero-field protobuf records by @fallintoplace in [#1085](https://github.com/apache/arrow-go/pull/1085) -* fix(arrow/scalar): preserve metadata in FromScalar by @fallintoplace in [#1100](https://github.com/apache/arrow-go/pull/1100) -* fix(arrow/scalar): preserve binary cast target types by @fallintoplace in [#1101](https://github.com/apache/arrow-go/pull/1101) -* fix(arrow/scalar): support boolean identity casts by @fallintoplace in [#1102](https://github.com/apache/arrow-go/pull/1102) -* fix(arrow/util): handle invalid protobuf Any values by @fallintoplace in [#1086](https://github.com/apache/arrow-go/pull/1086) -* fix(arrow/array): avoid copying initialized extension refcounts by @fallintoplace in [#1089](https://github.com/apache/arrow-go/pull/1089) -* fix(parquet): preserve dictionary index types by @fallintoplace in [#1098](https://github.com/apache/arrow-go/pull/1098) -* fix(parquet_reader): propagate statistics decoding errors by @fallintoplace in [#1087](https://github.com/apache/arrow-go/pull/1087) -* fix(arrow/array): avoid panics cleaning partial tables by @fallintoplace in [#1103](https://github.com/apache/arrow-go/pull/1103) -* fix(arrow/ipc): release reader state on schema errors by @fallintoplace in [#1112](https://github.com/apache/arrow-go/pull/1112) -* fix(arrow/scalar): format extension values from storage scalars by @fallintoplace in [#1111](https://github.com/apache/arrow-go/pull/1111) -* fix(arrow/array): isolate table columns from caller slices by @fallintoplace in [#1105](https://github.com/apache/arrow-go/pull/1105) -* fix(arrow/scalar): preserve exact decimal string casts by @fallintoplace in [#1107](https://github.com/apache/arrow-go/pull/1107) -* fix(arrow/tensor): validate typed constructor data by @fallintoplace in [#1104](https://github.com/apache/arrow-go/pull/1104) -* fix(arrow/scalar): format interval scalars in String by @fallintoplace in [#1115](https://github.com/apache/arrow-go/pull/1115) -* fix(arrow/array): preserve rows for empty-schema JSON input by @fallintoplace in [#1117](https://github.com/apache/arrow-go/pull/1117) -* fix(arrow/util): return protobuf record conversion errors by @fallintoplace in [#1088](https://github.com/apache/arrow-go/pull/1088) -* fix(arrow/scalar): validate FromScalar destinations by @fallintoplace in [#1106](https://github.com/apache/arrow-go/pull/1106) -* fix(arrow/scalar): use wide offsets for large binary scalars by @fallintoplace in [#1108](https://github.com/apache/arrow-go/pull/1108) -* fix(arrow/scalar): apply timestamp zones when casting to dates by @fallintoplace in [#1110](https://github.com/apache/arrow-go/pull/1110) -* fix(arrow/ipc): validate file block framing and lengths by @fallintoplace in [#1097](https://github.com/apache/arrow-go/pull/1097) -* fix(arrow/scalar): release partial children after failed struct extraction by @fallintoplace in [#1118](https://github.com/apache/arrow-go/pull/1118) -* fix(arrow/ipc): return errors for invalid record indexes by @fallintoplace in [#1120](https://github.com/apache/arrow-go/pull/1120) -* fix(parquet): read null fixed-size-list parents by @fallintoplace in [#1099](https://github.com/apache/arrow-go/pull/1099) -* fix(arrow/scalar): release partial children after failed struct conversion by @fallintoplace in [#1114](https://github.com/apache/arrow-go/pull/1114) -* fix(parquet/metadata): return error for missing column metadata by @fallintoplace in [#1137](https://github.com/apache/arrow-go/pull/1137) -* fix(arrow/array): widen dictionary index bounds checks by @fallintoplace in [#1129](https://github.com/apache/arrow-go/pull/1129) -* fix(compute/exprs): avoid empty field reference panic by @fallintoplace in [#1130](https://github.com/apache/arrow-go/pull/1130) -* fix(compute): handle odd temporal multiples by @fallintoplace in [#1119](https://github.com/apache/arrow-go/pull/1119) -* fix(arrow/csv): preserve first row during headerless inference by @fallintoplace in [#1145](https://github.com/apache/arrow-go/pull/1145) -* fix(arrow/array): bound table reader batches by @fallintoplace in [#1146](https://github.com/apache/arrow-go/pull/1146) -* fix(arrow/extensions): compare JSON storage types semantically by @fallintoplace in [#1150](https://github.com/apache/arrow-go/pull/1150) -* fix(arrow/ipc): preserve minSpaceSavings in stream writer by @fallintoplace in [#1143](https://github.com/apache/arrow-go/pull/1143) -* fix(parquet/compress): honor LZ4 raw destination contract by @fallintoplace in [#1147](https://github.com/apache/arrow-go/pull/1147) -* fix(arrow): reject duplicate union type codes by @fallintoplace in [#1131](https://github.com/apache/arrow-go/pull/1131) -* fix(parquet/metadata): handle invalid row-group column indexes by @fallintoplace in [#1125](https://github.com/apache/arrow-go/pull/1125) -* fix(compute/exec): handle missing validity buffers by @fallintoplace in [#1136](https://github.com/apache/arrow-go/pull/1136) -* fix(compute/exec): report ListView buffer counts by @fallintoplace in [#1133](https://github.com/apache/arrow-go/pull/1133) -* fix(arrow): include KeysSorted in map equality by @fallintoplace in [#1134](https://github.com/apache/arrow-go/pull/1134) -* fix(compute): reject non-array-like filters by @fallintoplace in [#1135](https://github.com/apache/arrow-go/pull/1135) -* fix(memory): handle zero-length checked reallocations by @fallintoplace in [#1132](https://github.com/apache/arrow-go/pull/1132) -* fix(parquet/pqarrow): validate reader indexes by @fallintoplace in [#1124](https://github.com/apache/arrow-go/pull/1124) -* fix(arrow/csv): validate custom converter row counts by @fallintoplace in [#1148](https://github.com/apache/arrow-go/pull/1148) -* fix(parquet/encoding): honor BufferWriter offsets by @fallintoplace in [#1144](https://github.com/apache/arrow-go/pull/1144) -* fix(flightsql): stop row streaming after close by @fallintoplace in [#1151](https://github.com/apache/arrow-go/pull/1151) -* fix(flightsql): protect configured authorization metadata by @fallintoplace in [#1153](https://github.com/apache/arrow-go/pull/1153) -* perf(parquet/pqarrow): avoid reflection in integer transfers by @fallintoplace in [#1163](https://github.com/apache/arrow-go/pull/1163) -* fix(flightsql): reject unsupported transaction options by @fallintoplace in [#1152](https://github.com/apache/arrow-go/pull/1152) -* fix(arrow/scalar): handle zero-length scalar arrays by @fallintoplace in [#1116](https://github.com/apache/arrow-go/pull/1116) -* perf(arrow/array): pack validity values in batches by @fallintoplace in [#1185](https://github.com/apache/arrow-go/pull/1185) -* feat(arrow/array): roll back JSON builder state after failed rows by @fallintoplace in [#1113](https://github.com/apache/arrow-go/pull/1113) -* fix(arrow/ipc): avoid deadlock on compression errors by @fallintoplace in [#1142](https://github.com/apache/arrow-go/pull/1142) -* fix(arrow/csv): append null for later parse failures by @fallintoplace in [#1149](https://github.com/apache/arrow-go/pull/1149) -* feat(arrow/array): add ValueAsAny for native Go values by @dgvj-work in [#1170](https://github.com/apache/arrow-go/pull/1170) -* perf(arrow/array): bulk append fixed-size list nulls and empty values by @fallintoplace in [#1183](https://github.com/apache/arrow-go/pull/1183) -* perf(parquet): encode spaced booleans without compaction by @fallintoplace in [#1175](https://github.com/apache/arrow-go/pull/1175) -* fix(parquet/pqarrow): reject decimal overflow in integer columns by @fallintoplace in [#1122](https://github.com/apache/arrow-go/pull/1122) -* fix(arrow/array/arreflect): reject unrepresentable timestamps by @fallintoplace in [#1123](https://github.com/apache/arrow-go/pull/1123) -* feat(compute): add cumulative_sum and cumulative_sum_checked by @fallintoplace in [#1139](https://github.com/apache/arrow-go/pull/1139) -* fix(arrow/ipc): return compression errors from record encoder by @fallintoplace in [#1141](https://github.com/apache/arrow-go/pull/1141) -* perf(arrow/array): bulk append fixed-width empty values by @fallintoplace in [#1180](https://github.com/apache/arrow-go/pull/1180) -* perf(arrow/csv): reserve record builders for chunked reads by @fallintoplace in [#1198](https://github.com/apache/arrow-go/pull/1198) -* perf(arrow/array): compare list values by valid runs by @fallintoplace in [#1184](https://github.com/apache/arrow-go/pull/1184) -* perf(parquet): optimize numeric dictionary insertion by @fallintoplace in [#1178](https://github.com/apache/arrow-go/pull/1178) -* feat(arrow/flight/sql): Add is_update field to ActionCreatePreparedStatementResult by @ennuite in [#732](https://github.com/apache/arrow-go/pull/732) -* feat(compute): add dictionary_encode by @fallintoplace in [#1140](https://github.com/apache/arrow-go/pull/1140) -* perf(parquet): batch plain BYTE_ARRAY encoding by @fallintoplace in [#1193](https://github.com/apache/arrow-go/pull/1193) -* fix(arrow/csv): append null for invalid fixed-size binary values by @fallintoplace in [#1121](https://github.com/apache/arrow-go/pull/1121) -* fix(parquet/pqarrow): reject out-of-range INT96 timestamps by @fallintoplace in [#1128](https://github.com/apache/arrow-go/pull/1128) -* perf(parquet): batch delta binary packed decoding by @fallintoplace in [#1169](https://github.com/apache/arrow-go/pull/1169) -* perf(arrow/array): bulk append list nulls and empty values by @fallintoplace in [#1187](https://github.com/apache/arrow-go/pull/1187) -* perf(parquet/pqarrow): decode booleans directly to bitmaps by @fallintoplace in [#1182](https://github.com/apache/arrow-go/pull/1182) -* perf(parquet/pqarrow): build nullable fixed-size list children directly by @fallintoplace in [#1195](https://github.com/apache/arrow-go/pull/1195) -* perf(internal/hashing): build binary memo offsets in one pass by @fallintoplace in [#1196](https://github.com/apache/arrow-go/pull/1196) -* perf(arrow/compute): take chunked binary values without concatenation by @fallintoplace in [#1199](https://github.com/apache/arrow-go/pull/1199) -* perf(arrow/array): reuse TableReader scratch slices by @fallintoplace in [#1200](https://github.com/apache/arrow-go/pull/1200) -* fix(parquet/pqarrow): reject decimal overflow when writing integers by @fallintoplace in [#1161](https://github.com/apache/arrow-go/pull/1161) -* fix(arrow/array): preserve run-end encoded empty values by @fallintoplace in [#1156](https://github.com/apache/arrow-go/pull/1156) -* fix(compute): preserve caller context during execution by @fallintoplace in [#1154](https://github.com/apache/arrow-go/pull/1154) -* fix(compute): propagate nested expression serialization errors by @fallintoplace in [#1160](https://github.com/apache/arrow-go/pull/1160) -* fix(compute): report FuncMeta for meta functions by @fallintoplace in [#1155](https://github.com/apache/arrow-go/pull/1155) -* perf(arrow/array): optimize bulk appends for binary builders by @fallintoplace in [#1188](https://github.com/apache/arrow-go/pull/1188) -* perf(arrow/array): avoid temporary buffers when concatenating by @fallintoplace in [#1194](https://github.com/apache/arrow-go/pull/1194) -* perf(compute): batch contiguous filter take indices by @fallintoplace in [#1197](https://github.com/apache/arrow-go/pull/1197) -* perf(arrow/ipc): slice already-zero-based offset buffers by @fallintoplace in [#1191](https://github.com/apache/arrow-go/pull/1191) -* perf(parquet): encode RLE levels in batches by @fallintoplace in [#1177](https://github.com/apache/arrow-go/pull/1177) -* fix(arrow): include REE value nullability in type equality by @fallintoplace in [#1158](https://github.com/apache/arrow-go/pull/1158) -* fix(parquet): build bloom filters from dictionary entries by @fallintoplace in [#1164](https://github.com/apache/arrow-go/pull/1164) -* fix(arrow): validate precision in NewDecimalType by @fallintoplace in [#1162](https://github.com/apache/arrow-go/pull/1162) -* perf(arrow/csv): avoid materializing rows before writing by @fallintoplace in [#1190](https://github.com/apache/arrow-go/pull/1190) -* perf(arrow/array): batch run-end encoded appends by @fallintoplace in [#1214](https://github.com/apache/arrow-go/pull/1214) -* fix(arrow/array): validate union JSON type codes by @fallintoplace in [#1157](https://github.com/apache/arrow-go/pull/1157) -* fix(parquet/variant): validate compound value bounds by @fallintoplace in [#1126](https://github.com/apache/arrow-go/pull/1126) -* perf(arrow/array): validate dictionary indices by valid runs by @fallintoplace in [#1181](https://github.com/apache/arrow-go/pull/1181) -* perf(parquet): bound Bloom filter hash batches by @fallintoplace in [#1218](https://github.com/apache/arrow-go/pull/1218) -* perf(arrow/array): optimize bulk appends for struct builders by @fallintoplace in [#1189](https://github.com/apache/arrow-go/pull/1189) -* fix(arrow): return owned field index slices by @fallintoplace in [#1159](https://github.com/apache/arrow-go/pull/1159) -* perf(arrow/array): avoid temporary slices in Diff comparisons by @fallintoplace in [#1239](https://github.com/apache/arrow-go/pull/1239) -* perf(parquet/compress): reuse gzip writers by @fallintoplace in [#1238](https://github.com/apache/arrow-go/pull/1238) -* perf(parquet): reuse fixed-length zero scratch by @fallintoplace in [#1236](https://github.com/apache/arrow-go/pull/1236) -* perf(compute): share logical row maps across aligned sorts by @fallintoplace in [#1234](https://github.com/apache/arrow-go/pull/1234) -* perf(compute): fast-path single-span execution by @fallintoplace in [#1233](https://github.com/apache/arrow-go/pull/1233) -* perf(parquet): reuse boolean spaced scratch by @fallintoplace in [#1231](https://github.com/apache/arrow-go/pull/1231) -* perf(parquet): reuse delta PutSpaced scratch by @fallintoplace in [#1230](https://github.com/apache/arrow-go/pull/1230) -* perf(arrow/array): remove map validity slice by @fallintoplace in [#1228](https://github.com/apache/arrow-go/pull/1228) -* perf(arrow/array): copy same-dictionary indices directly by @fallintoplace in [#1225](https://github.com/apache/arrow-go/pull/1225) -* perf(parquet): reuse dictionary decoder scratch by @fallintoplace in [#1222](https://github.com/apache/arrow-go/pull/1222) -* perf(arrow/array): compare fixed-width values in bulk by @fallintoplace in [#1171](https://github.com/apache/arrow-go/pull/1171) -* perf(arrow/array): compare boolean values by valid runs by @fallintoplace in [#1186](https://github.com/apache/arrow-go/pull/1186) -* perf(arrow/array): avoid temporary buffers for binary concatenation by @fallintoplace in [#1215](https://github.com/apache/arrow-go/pull/1215) -* perf(arrow/array): batch union null and empty appends by @fallintoplace in [#1216](https://github.com/apache/arrow-go/pull/1216) -* perf(parquet): decode levels directly into int16 batches by @fallintoplace in [#1176](https://github.com/apache/arrow-go/pull/1176) -* perf(parquet): batch DELTA_BYTE_ARRAY encoding by @fallintoplace in [#1179](https://github.com/apache/arrow-go/pull/1179) -* perf(parquet/variant): skip metadata construction in Encode by @fallintoplace in [#1217](https://github.com/apache/arrow-go/pull/1217) -* perf(parquet): decode BSS FLBA into contiguous storage by @fallintoplace in [#1172](https://github.com/apache/arrow-go/pull/1172) -* perf(parquet): stop spaced expansion once aligned by @fallintoplace in [#1220](https://github.com/apache/arrow-go/pull/1220) -* perf(arrow/ipc): reuse record encoder scratch by @fallintoplace in [#1221](https://github.com/apache/arrow-go/pull/1221) -* perf(arrow/array): batch dictionary empty appends by @fallintoplace in [#1223](https://github.com/apache/arrow-go/pull/1223) -* perf(compute): reserve binary view cast payload by @fallintoplace in [#1227](https://github.com/apache/arrow-go/pull/1227) -* perf(compute): use stack scratch for null propagation by @fallintoplace in [#1229](https://github.com/apache/arrow-go/pull/1229) -* perf(arrow/array): bulk append BinaryView nulls by @fallintoplace in [#1232](https://github.com/apache/arrow-go/pull/1232) -* perf(arrow/array): use unsafe writes for binary bulk appends by @fallintoplace in [#1240](https://github.com/apache/arrow-go/pull/1240) -* perf(arrow/array): compare union values in runs by @fallintoplace in [#1244](https://github.com/apache/arrow-go/pull/1244) -* perf(arrow/array): batch-pack boolean values by @fallintoplace in [#1237](https://github.com/apache/arrow-go/pull/1237) -* fix(parquet/pqarrow): require Variant value field when writing by @dgvj-work in [#1242](https://github.com/apache/arrow-go/pull/1242) -* perf(arrow/compute): avoid goroutines for serial record filtering by @fallintoplace in [#1245](https://github.com/apache/arrow-go/pull/1245) -* perf(parquet): reuse dictionary RLE decoder across pages by @fallintoplace in [#1247](https://github.com/apache/arrow-go/pull/1247) -* perf(parquet/pqarrow): write record ranges directly by @fallintoplace in [#1248](https://github.com/apache/arrow-go/pull/1248) -* perf(parquet/pqarrow): avoid goroutines for serial struct reads by @fallintoplace in [#1250](https://github.com/apache/arrow-go/pull/1250) -* perf(parquet): decode RLE booleans into bitmaps by @fallintoplace in [#1246](https://github.com/apache/arrow-go/pull/1246) -* feat(compute): add list_element by @fallintoplace in [#1138](https://github.com/apache/arrow-go/pull/1138) -* perf(parquet): reuse DataPageV2 assembly buffer for eager writes by @fallintoplace in [#1192](https://github.com/apache/arrow-go/pull/1192) -* perf(arrow/array): append fixed-width nulls in bulk by @fallintoplace in [#1173](https://github.com/apache/arrow-go/pull/1173) -* perf(arrow/array): concatenate bitmaps directly by @fallintoplace in [#1224](https://github.com/apache/arrow-go/pull/1224) -* perf(arrow/compute): avoid goroutines for serial take by @fallintoplace in [#1226](https://github.com/apache/arrow-go/pull/1226) -* perf(compute): use typed memo insertion for numeric hashes by @fallintoplace in [#1251](https://github.com/apache/arrow-go/pull/1251) -* perf(parquet): decode DELTA_BYTE_ARRAY into contiguous storage by @fallintoplace in [#1252](https://github.com/apache/arrow-go/pull/1252) -* perf(parquet/pqarrow): use synchronous path for serial reads by @fallintoplace in [#1235](https://github.com/apache/arrow-go/pull/1235) -* perf(arrow/array): reuse full chunks in NewChunkedSlice by @fallintoplace in [#1257](https://github.com/apache/arrow-go/pull/1257) -* perf(parquet): reuse boolean Bloom filter hashes by @fallintoplace in [#1258](https://github.com/apache/arrow-go/pull/1258) -* perf(arrow/ipc): release dictionary payloads eagerly by @fallintoplace in [#1259](https://github.com/apache/arrow-go/pull/1259) -* perf(parquet): SIMD BYTE_STREAM_SPLIT encoding by @fallintoplace in [#1260](https://github.com/apache/arrow-go/pull/1260) -* perf(parquet): reuse DELTA byte-array scratch by @fallintoplace in [#1261](https://github.com/apache/arrow-go/pull/1261) -* perf(parquet/pqarrow): write FixedSizeBinary values directly by @fallintoplace in [#1263](https://github.com/apache/arrow-go/pull/1263) -* feat(extensions): add VariantGet for path extraction from variant arrays by @nssalian in [#1206](https://github.com/apache/arrow-go/pull/1206) -* perf(parquet): use typed memo insertion for byte-array dictionaries by @derekperkins in [#1272](https://github.com/apache/arrow-go/pull/1272) -* fix(arrow/array): clamp run ends when concatenating a sliced RunEndEncoded array by @winklemad in [#1219](https://github.com/apache/arrow-go/pull/1219) -* fix(parquet): guard byte-stream-split encoder release against never-flushed buffer by @tom-eon in [#1273](https://github.com/apache/arrow-go/pull/1273) -* perf(bitutil): add ARM64 NEON bitmap operations by @fallintoplace in [#1264](https://github.com/apache/arrow-go/pull/1264) -* perf(arrow/array): compare binary values by valid runs by @fallintoplace in [#1174](https://github.com/apache/arrow-go/pull/1174) -* perf(compute): add ARM64 NEON arithmetic kernels by @fallintoplace in [#1265](https://github.com/apache/arrow-go/pull/1265) -* perf(parquet/compress): reuse Brotli writers across encodes by @fallintoplace in [#1254](https://github.com/apache/arrow-go/pull/1254) -* perf(arrow/array): skip full-range SliceEqual slices by @fallintoplace in [#1249](https://github.com/apache/arrow-go/pull/1249) -* perf(parquet): vectorize fixed-width dictionary materialization by @fallintoplace in [#1269](https://github.com/apache/arrow-go/pull/1269) -* perf(compute): add ARM64 NEON comparison kernels by @fallintoplace in [#1266](https://github.com/apache/arrow-go/pull/1266) -* perf(parquet/pqarrow): write binary values from Arrow offsets by @fallintoplace in [#1262](https://github.com/apache/arrow-go/pull/1262) -* fix(arrow/extensions): reject Null Variant typed_value by @dgvj-work in [#1243](https://github.com/apache/arrow-go/pull/1243) -* fix(arrow/extensions): use canonical Variant extension name by @dgvj-work in [#1241](https://github.com/apache/arrow-go/pull/1241) -* chore: bump version number by @zeroshade in [#1276](https://github.com/apache/arrow-go/pull/1276) -* test(parquet): reduce macOS race test memory pressure by @zeroshade in [#1275](https://github.com/apache/arrow-go/pull/1275) -* fix(parquet): explicitly release bloom builders by @zeroshade in [#1280](https://github.com/apache/arrow-go/pull/1280) -* ci: retry and cache dependency downloads by @zeroshade in [#1279](https://github.com/apache/arrow-go/pull/1279) -* fix(compute): discard results after cancellation by @zeroshade in [#1278](https://github.com/apache/arrow-go/pull/1278) -* ci: use bundled GTest for conda integration by @zeroshade in [#1277](https://github.com/apache/arrow-go/pull/1277) -* fix(parquet): stop BYTE_STREAM_SPLIT FLBA spaced decode aliasing output by @zeroshade in [#1256](https://github.com/apache/arrow-go/pull/1256) -* perf(compute): vectorize mixed int32 filters with AVX2 by @fallintoplace in [#1284](https://github.com/apache/arrow-go/pull/1284) -* perf(compute): add ARM64 NEON constant multiplication by @fallintoplace in [#1268](https://github.com/apache/arrow-go/pull/1268) -* perf(compute): vectorize numeric to boolean casts by @fallintoplace in [#1282](https://github.com/apache/arrow-go/pull/1282) -* fix(arrow/scalar): align timestamp timezone parsing by @fallintoplace in [#1109](https://github.com/apache/arrow-go/pull/1109) -* perf(internal/utils): add ARM64 NEON min/max for 8- and 16-bit integers by @fallintoplace in [#1281](https://github.com/apache/arrow-go/pull/1281) -* perf(arrow/array): SIMD-pack boolean slices into bitmaps by @fallintoplace in [#1283](https://github.com/apache/arrow-go/pull/1283) -* chore: fix failing build tags by @zeroshade in [#1285](https://github.com/apache/arrow-go/pull/1285) -* perf(compute): SIMD-compact fragmented filters into take indices by @fallintoplace in [#1287](https://github.com/apache/arrow-go/pull/1287) -* perf(compute): add ARM64 NEON comparisons for 8- and 16-bit integers by @fallintoplace in [#1288](https://github.com/apache/arrow-go/pull/1288) -* perf(compute): add ARM64 NEON compression for null-free 32-bit fixed-width filters by @fallintoplace in [#1286](https://github.com/apache/arrow-go/pull/1286) +### Selected Changes + +A selection of the user-facing changes in this release. See the full changelog link +below for all 153 merged pull requests, plus 36 automated dependency updates. + +#### Arrow + +* feat(arrow/ipc): add Reader.ContinueFrom for streams split across blobs by @zeroshade in [#1346](https://github.com/apache/arrow-go/pull/1346) +* feat(arrow/ipc): add WithZstdLevel option for IPC writer by @Asthenia0412 in [#1376](https://github.com/apache/arrow-go/pull/1376) +* feat(arrow/flight/flightsql/driver): accept grpc+tcp, grpc+tls and grpc+unix schemes by @nirajan1111 in [#1420](https://github.com/apache/arrow-go/pull/1420) +* fix: expose internal offsets in json/csv readers by @Willem-J-an in [#1342](https://github.com/apache/arrow-go/pull/1342) +* feat(arrow/scalar): add StringView and BinaryView scalars by @hcrosse in [#1490](https://github.com/apache/arrow-go/pull/1490) +* feat(arrow/scalar): add Decimal32 and Decimal64 scalars by @hcrosse in [#1501](https://github.com/apache/arrow-go/pull/1501) +* fix(arrow): support zero-length fixed-size lists by @jakthom in [#1308](https://github.com/apache/arrow-go/pull/1308) +* fix(array/record): bldr stop ignore unexpected EOF by @Willem-J-an in [#1341](https://github.com/apache/arrow-go/pull/1341) +* fix(arrow/ipc): open mapped files with dictionary fields by @hpaliwal8 in [#1365](https://github.com/apache/arrow-go/pull/1365) +* fix(arrow): handle dense union slices and empty validation by @skyon0522-ai in [#1385](https://github.com/apache/arrow-go/pull/1385) +* fix(arrow/array): validate bufferless run-end encoded arrays by @zeroshade in [#1421](https://github.com/apache/arrow-go/pull/1421) +* fix(arrow/flight/flightsql): send schema for empty ingest by @arnoldwakim in [#1433](https://github.com/apache/arrow-go/pull/1433) +* fix(arrow): compare LargeListType like ListType in TypeEqual by @paveon in [#1440](https://github.com/apache/arrow-go/pull/1440) +* fix(arrow/bitutil): preserve overlapping bitmap transfers by @fallintoplace in [#1389](https://github.com/apache/arrow-go/pull/1389) +* fix(arrow/decimal): widen Decimal64 rescale to 64 bits by @winklemad in [#1307](https://github.com/apache/arrow-go/pull/1307) +* fix(arrow/decimal): handle native word widths in big integer conversions by @zeroshade in [#1409](https://github.com/apache/arrow-go/pull/1409) +* fix(arrow/decimal): correct rescaling results and report cast errors by @hcrosse in [#1513](https://github.com/apache/arrow-go/pull/1513) +* fix(arrow/decimal): reject infinite decimal strings and bound extreme exponents by @hcrosse in [#1514](https://github.com/apache/arrow-go/pull/1514) +* fix(arrow/decimal): fail clearly for Decimal128 and Decimal256 scales past MaxScale by @hcrosse in [#1515](https://github.com/apache/arrow-go/pull/1515) +* fix(arrow/cdata): export extension arrays as their storage by @hcrosse in [#1526](https://github.com/apache/arrow-go/pull/1526) +* fix(arrow/cdata): export an empty view sizes buffer as the zero region by @hcrosse in [#1544](https://github.com/apache/arrow-go/pull/1544) +* fix(arrow/array): validate extension arrays by their storage type by @hcrosse in [#1487](https://github.com/apache/arrow-go/pull/1487) +* fix(arrow/array): keep the storage dictionary in NewExtensionArrayWithStorage by @hcrosse in [#1540](https://github.com/apache/arrow-go/pull/1540) +* fix(arrow/csv): preserve braces in list values by @peterolkhov in [#1499](https://github.com/apache/arrow-go/pull/1499) +* fix(arrow/csv): preserve lists holding a single empty string by @zeroshade in [#1505](https://github.com/apache/arrow-go/pull/1505) +* fix(arrow/csv): keep list items equal to the null value distinct from nulls by @zeroshade in [#1508](https://github.com/apache/arrow-go/pull/1508) + +#### Compute + +* feat(arrow/compute): scalar aggregate framework with the count and sum kernels by @singhpratech in [#1336](https://github.com/apache/arrow-go/pull/1336) +* feat(arrow/compute): add IEEE 754 total-order float sorting option by @zeroshade in [#1430](https://github.com/apache/arrow-go/pull/1430) +* feat(arrow/compute): add starts_with kernel by @hcrosse in [#1448](https://github.com/apache/arrow-go/pull/1448) +* feat(arrow/compute): support string_view and binary_view in starts_with by @hcrosse in [#1467](https://github.com/apache/arrow-go/pull/1467) +* feat(arrow/compute): support IgnoreCase in starts_with by @hcrosse in [#1468](https://github.com/apache/arrow-go/pull/1468) +* fix(compute): detect temporal rounding overflow by @fallintoplace in [#1127](https://github.com/apache/arrow-go/pull/1127) +* fix(compute/exec): do not carry a cached null count across ArraySpan.SetSlice by @singhpratech in [#1306](https://github.com/apache/arrow-go/pull/1306) +* fix(arrow/compute): allow casting nullable struct fields to non-nullable when no nulls by @sankalpsthakur in [#1363](https://github.com/apache/arrow-go/pull/1363) +* fix(arrow/compute): fix NewDatumWithoutOwning ownership semantics by @Asthenia0412 in [#1377](https://github.com/apache/arrow-go/pull/1377) +* fix(arrow/compute): respect GOMAXPROCS for default NumParallel by @samueleresca in [#1384](https://github.com/apache/arrow-go/pull/1384) +* fix(arrow/compute): do not return a result from a cancelled call by @zeroshade in [#1410](https://github.com/apache/arrow-go/pull/1410) +* fix(arrow/compute): correct amd64 kernel assembly ABI by @zeroshade in [#1432](https://github.com/apache/arrow-go/pull/1432) +* fix(arrow/compute): honor the input offset in is_not_null by @hcrosse in [#1456](https://github.com/apache/arrow-go/pull/1456) +* fix(arrow/compute): register is_null, is_not_null and is_nan as unary functions by @hcrosse in [#1457](https://github.com/apache/arrow-go/pull/1457) +* fix(arrow/compute): honor the input offset in not by @hcrosse in [#1458](https://github.com/apache/arrow-go/pull/1458) +* fix(arrow/compute): treat null dictionary values as null in is_null and is_not_null by @hcrosse in [#1461](https://github.com/apache/arrow-go/pull/1461) +* fix(arrow/compute): handle null, union and run-end encoded arrays in is_null and is_not_null by @hcrosse in [#1465](https://github.com/apache/arrow-go/pull/1465) +* fix(arrow/compute): honor NullOptions.NanIsNull in is_null by @hcrosse in [#1488](https://github.com/apache/arrow-go/pull/1488) +* fix(arrow/compute/exprs): accept string and binary dictionary columns in field references by @hcrosse in [#1449](https://github.com/apache/arrow-go/pull/1449) +* fix(arrow/compute/exprs): fix offsets, nulls and ownership of nested field references by @zeroshade in [#1474](https://github.com/apache/arrow-go/pull/1474) +* fix(arrow/compute): use int16 kernels for casts to int16 by @hcrosse in [#1503](https://github.com/apache/arrow-go/pull/1503) + +#### Parquet + +* feat(parquet): add ALP encoding for FLOAT and DOUBLE columns by @prtkgaur in [#1289](https://github.com/apache/arrow-go/pull/1289) +* feat(parquet/encryption): KMS support for reading PKMT1 key material by @EdwardShen125 in [#1478](https://github.com/apache/arrow-go/pull/1478) +* feat(parquet/file): Write geospatial parquet row group statistics for GEOMETRY column chunks by @C-Loftus in [#1504](https://github.com/apache/arrow-go/pull/1504) +* fix(parquet/file): write `null_count` statistics for GEOMETRY and GEOGRAPHY column chunks by @C-Loftus in [#1533](https://github.com/apache/arrow-go/pull/1533) +* fix(parquet/utils): make unpack32 NEON profiler-safe by @zeroshade in [#1300](https://github.com/apache/arrow-go/pull/1300) +* fix(parquet/metadata): stop bloom filter asm clobbering the frame pointer by @zeroshade in [#1379](https://github.com/apache/arrow-go/pull/1379) +* fix(parquet/variant): prevent metadata offset cursor overflow by @matanper in [#1290](https://github.com/apache/arrow-go/pull/1290) +* fix(parquet/variant): return "" for an empty long-form string by @CaptainAni187 in [#1353](https://github.com/apache/arrow-go/pull/1353) +* fix(parquet/metadata): count non-null values from the validity bitmap in UpdateSpaced by @CaptainAni187 in [#1367](https://github.com/apache/arrow-go/pull/1367) +* fix(parquet/pqarrow): honor offset and null slots when writing fixed-size lists by @zeroshade in [#1382](https://github.com/apache/arrow-go/pull/1382) +* fix(parquet/file): read encrypted page headers written by parquet-mr by @sankalpsthakur in [#1390](https://github.com/apache/arrow-go/pull/1390) +* fix(parquet/internal/encoding): do not leak delta bit width buffers from decoders by @zeroshade in [#1394](https://github.com/apache/arrow-go/pull/1394) +* fix(parquet/internal/encoding): reset delta bit pack encoder state after flush by @zeroshade in [#1400](https://github.com/apache/arrow-go/pull/1400) +* fix(parquet): return an error instead of panicking on a hidden encrypted column by @ump45nose in [#1406](https://github.com/apache/arrow-go/pull/1406) +* fix(parquet): preserve CTR algorithm in plaintext footers by @autoCAT6 in [#1412](https://github.com/apache/arrow-go/pull/1412) +* fix(parquet/pqarrow): release partial readers when building a field reader fails by @elsteveogrande in [#1435](https://github.com/apache/arrow-go/pull/1435) +* fix(parquet): release writer buffers when a column chunk flush fails by @twuebi in [#1506](https://github.com/apache/arrow-go/pull/1506) +* fix(parquet/metadata): count empty strings in byte array statistics from Arrow by @hcrosse in [#1527](https://github.com/apache/arrow-go/pull/1527) +* fix(parquet/compress): read to the end of a Brotli stream that fills dst by @hcrosse in [#1530](https://github.com/apache/arrow-go/pull/1530) +* fix(parquet/internal/encoding): clear buffered RLE booleans after each flush by @hcrosse in [#1539](https://github.com/apache/arrow-go/pull/1539) +* fix(parquet/file): count V2 page num_nulls from the definition levels by @hcrosse in [#1543](https://github.com/apache/arrow-go/pull/1543) +* fix(parquet/internal/encoding): clear buffered RLE booleans on Reset by @hcrosse in [#1545](https://github.com/apache/arrow-go/pull/1545) +* fix(parquet/internal/encoding): clear encoder state on Reset by @hcrosse in [#1546](https://github.com/apache/arrow-go/pull/1546) +* fix(parquet/file): return an error for dictionary encoded boolean columns by @hcrosse in [#1547](https://github.com/apache/arrow-go/pull/1547) + +#### Performance + +* perf(arrow/array): compare optional validity bitmaps in bulk by @fallintoplace in [#1327](https://github.com/apache/arrow-go/pull/1327) +* perf(arrow/array): compare approximate list values by valid runs by @fallintoplace in [#1330](https://github.com/apache/arrow-go/pull/1330) +* perf(arrow/array): compare fixed-size binary values in bulk by @fallintoplace in [#1347](https://github.com/apache/arrow-go/pull/1347) +* perf(arrow/array): copy view buffers directly by @fallintoplace in [#1324](https://github.com/apache/arrow-go/pull/1324) +* perf(arrow/array): concatenate list offsets directly by @fallintoplace in [#1323](https://github.com/apache/arrow-go/pull/1323) +* perf(arrow/array): write bulk binary offsets and payloads directly by @fallintoplace in [#1413](https://github.com/apache/arrow-go/pull/1413) +* perf(arrow/array): reuse unchanged dictionaries across builder arrays by @hcrosse in [#1444](https://github.com/apache/arrow-go/pull/1444) +* perf(arrow/array): reuse dictionary mappings and handle empty chunks in Concatenate by @fallintoplace in [#1403](https://github.com/apache/arrow-go/pull/1403) +* perf(compute): vectorize mixed 64-bit filters by @fallintoplace in [#1311](https://github.com/apache/arrow-go/pull/1311) +* perf(compute): vectorize mixed 16-bit filters by @fallintoplace in [#1349](https://github.com/apache/arrow-go/pull/1349) +* perf(compute): vectorize mixed 8-bit filters by @fallintoplace in [#1350](https://github.com/apache/arrow-go/pull/1350) +* perf(compute): add AVX2 take-index compaction by @fallintoplace in [#1348](https://github.com/apache/arrow-go/pull/1348) +* perf(compute): reuse amd64 SIMD comparisons for numeric-to-bool casts by @fallintoplace in [#1351](https://github.com/apache/arrow-go/pull/1351) +* perf(arrow/compute): read flat take values directly by @fallintoplace in [#1360](https://github.com/apache/arrow-go/pull/1360) +* perf(arrow/compute): avoid getter wrappers in chunked binary take by @fallintoplace in [#1359](https://github.com/apache/arrow-go/pull/1359) +* perf(compute): radix-sort null-free integer indices by @fallintoplace in [#1391](https://github.com/apache/arrow-go/pull/1391) +* perf(arrow/compute): unify changing dictionaries incrementally by @fallintoplace in [#1395](https://github.com/apache/arrow-go/pull/1395) +* perf(compute): use typed memo insertion for binary hashes by @fallintoplace in [#1315](https://github.com/apache/arrow-go/pull/1315) +* perf(bitutil): pack boolean bitmap writes by byte by @fallintoplace in [#1402](https://github.com/apache/arrow-go/pull/1402) +* perf(bitutil): accelerate misaligned bitmap equality by @fallintoplace in [#1414](https://github.com/apache/arrow-go/pull/1414) +* perf(bitutil): bulk-transfer unaligned bitmap ranges by @fallintoplace in [#1416](https://github.com/apache/arrow-go/pull/1416) +* perf(bitutil): process unaligned bitmap operations by word by @fallintoplace in [#1437](https://github.com/apache/arrow-go/pull/1437) +* perf(arrow/csv): stream full-file CSV reads into builders by @fallintoplace in [#1393](https://github.com/apache/arrow-go/pull/1393) +* perf: zstd implement Pooling + DecodeAll with no new reader. by @samueleresca in [#1331](https://github.com/apache/arrow-go/pull/1331) +* perf(arrow/ipc): reuse zstd encoders across writers by @vdpoora in [#1429](https://github.com/apache/arrow-go/pull/1429) +* perf(parquet/metadata): pool bloom filter hash scratch buffers by @zeroshade in [#1378](https://github.com/apache/arrow-go/pull/1378) +* perf(parquet): use typed memo insertion for fixed-width dictionaries by @fallintoplace in [#1312](https://github.com/apache/arrow-go/pull/1312) +* perf(parquet): encode dictionary indices in RLE batches by @fallintoplace in [#1328](https://github.com/apache/arrow-go/pull/1328) +* perf(parquet): reuse DELTA_BYTE_ARRAY discard storage by @fallintoplace in [#1321](https://github.com/apache/arrow-go/pull/1321) +* perf(parquet): reuse DELTA decoder scratch by @fallintoplace in [#1322](https://github.com/apache/arrow-go/pull/1322) +* perf(parquet): speed up delta byte array prefix matching by @fallintoplace in [#1388](https://github.com/apache/arrow-go/pull/1388) +* perf(parquet): fast-path constant delta bitpacked blocks by @fallintoplace in [#1396](https://github.com/apache/arrow-go/pull/1396) +* perf(parquet): batch DELTA_LENGTH byte-array decoding and discard by @fallintoplace in [#1415](https://github.com/apache/arrow-go/pull/1415) +* perf(parquet): decode narrow packed levels directly by @fallintoplace in [#1404](https://github.com/apache/arrow-go/pull/1404) +* perf(parquet): expand nullable boolean bitmaps by byte by @fallintoplace in [#1405](https://github.com/apache/arrow-go/pull/1405) +* perf(parquet): decode RLE booleans directly into bool slices by @fallintoplace in [#1417](https://github.com/apache/arrow-go/pull/1417) +* perf(parquet): pack narrow RLE literal levels eight at a time by @fallintoplace in [#1419](https://github.com/apache/arrow-go/pull/1419) +* perf(parquet/file): use dense decoding for batches without nulls by @hcrosse in [#1447](https://github.com/apache/arrow-go/pull/1447) +* perf(parquet/pqarrow): build struct child readers serially when Parallel is false by @elsteveogrande in [#1434](https://github.com/apache/arrow-go/pull/1434) +* perf(parquet/internal/encoding): preallocate dictionaries bounded by page data by @zeroshade in [#1473](https://github.com/apache/arrow-go/pull/1473) ### New Contributors -* @C-Loftus made their first contribution in [#960](https://github.com/apache/arrow-go/pull/960) -* @tom-eon made their first contribution in [#1027](https://github.com/apache/arrow-go/pull/1027) -* @prochac made their first contribution in [#830](https://github.com/apache/arrow-go/pull/830) -* @dgvj-work made their first contribution in [#1170](https://github.com/apache/arrow-go/pull/1170) -* @ennuite made their first contribution in [#732](https://github.com/apache/arrow-go/pull/732) -* @derekperkins made their first contribution in [#1272](https://github.com/apache/arrow-go/pull/1272) -* @winklemad made their first contribution in [#1219](https://github.com/apache/arrow-go/pull/1219) - -**Full Changelog**: https://github.com/apache/arrow-go/compare/v18.7.0...v18.8.0 +* @matanper made their first contribution in [#1290](https://github.com/apache/arrow-go/pull/1290) +* @singhpratech made their first contribution in [#1303](https://github.com/apache/arrow-go/pull/1303) +* @jakthom made their first contribution in [#1308](https://github.com/apache/arrow-go/pull/1308) +* @CaptainAni187 made their first contribution in [#1353](https://github.com/apache/arrow-go/pull/1353) +* @samueleresca made their first contribution in [#1331](https://github.com/apache/arrow-go/pull/1331) +* @hpaliwal8 made their first contribution in [#1365](https://github.com/apache/arrow-go/pull/1365) +* @sankalpsthakur made their first contribution in [#1363](https://github.com/apache/arrow-go/pull/1363) +* @Asthenia0412 made their first contribution in [#1376](https://github.com/apache/arrow-go/pull/1376) +* @skyon0522-ai made their first contribution in [#1385](https://github.com/apache/arrow-go/pull/1385) +* @nirajan1111 made their first contribution in [#1420](https://github.com/apache/arrow-go/pull/1420) +* @autoCAT6 made their first contribution in [#1412](https://github.com/apache/arrow-go/pull/1412) +* @vdpoora made their first contribution in [#1429](https://github.com/apache/arrow-go/pull/1429) +* @elsteveogrande made their first contribution in [#1435](https://github.com/apache/arrow-go/pull/1435) +* @EdwardShen125 made their first contribution in [#1439](https://github.com/apache/arrow-go/pull/1439) +* @AynSiddiqui made their first contribution in [#1477](https://github.com/apache/arrow-go/pull/1477) +* @peterolkhov made their first contribution in [#1499](https://github.com/apache/arrow-go/pull/1499) +* @prtkgaur made their first contribution in [#1289](https://github.com/apache/arrow-go/pull/1289) + +**Full Changelog**: https://github.com/apache/arrow-go/compare/v18.8.0...v18.9.0 From adca473a057f4dccf01419dabc2b07d0aa7c147b Mon Sep 17 00:00:00 2001 From: Matt Topol Date: Mon, 16 Nov 2026 13:09:26 -0500 Subject: [PATCH 3/3] drop conflicting PR count from Arrow Go 18.9.0 selected changes - the 153 PR / 36 dependabot figures read as 189 entries against the 152-commit summary (roborev job 13688) --- _posts/2026-11-16-arrow-go-18.9.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2026-11-16-arrow-go-18.9.0.md b/_posts/2026-11-16-arrow-go-18.9.0.md index 27b9137e861d..6535a7330a2b 100644 --- a/_posts/2026-11-16-arrow-go-18.9.0.md +++ b/_posts/2026-11-16-arrow-go-18.9.0.md @@ -165,7 +165,7 @@ without goroutines, 37-43% faster (Apple M4 Max) [#1447](https://github.com/apac ### Selected Changes A selection of the user-facing changes in this release. See the full changelog link -below for all 153 merged pull requests, plus 36 automated dependency updates. +below for every change in the release, including automated dependency updates. #### Arrow