From 92738391def3114a36953ff7c83709a878df68f6 Mon Sep 17 00:00:00 2001 From: Matt Topol Date: Tue, 8 Sep 2026 19:09:07 +0000 Subject: [PATCH 1/3] [Release] Add release notes for Arrow Go 18.8.0 --- _posts/2026-09-08-arrow-go-18.8.0.md | 348 +++++++++++++++++++++++++++ 1 file changed, 348 insertions(+) create mode 100644 _posts/2026-09-08-arrow-go-18.8.0.md diff --git a/_posts/2026-09-08-arrow-go-18.8.0.md b/_posts/2026-09-08-arrow-go-18.8.0.md new file mode 100644 index 000000000000..7f24518d9696 --- /dev/null +++ b/_posts/2026-09-08-arrow-go-18.8.0.md @@ -0,0 +1,348 @@ +--- +layout: post +title: "Apache Arrow Go 18.8.0 Release" +date: "2026-09-08 00:00:00" +author: pmc +categories: [release] +--- + + +The Apache Arrow team is pleased to announce the v18.8.0 release of Apache Arrow Go. +This minor release covers 283 commits from 16 distinct contributors. + +## Contributors +```console +$ git shortlog -sn v18.7.0..v18.8.0 + 251 Minh Vu + 10 Matt Topol + 4 Colton Loftus + 4 Digvijay + 4 Tom Frank + 1 Derek Perkins + 1 Lucas Valente + 1 Madan kumar + 1 Neelesh Salian + 1 Nick Ripley + 1 Pedro Matias + 1 Raúl Cumplido + 1 Ruihao Chen + 1 Tobias Pütz + 1 Tomáš Karela Procházka + 1 Willem Jan +``` + +## Changelog + +### What's Changed +* feat(parquet/schema): Wire up logical type functionality for writing geo types by @C-Loftus in [#960](https://github.com/apache/arrow-go/pull/960) +* chore(dev/release): filter out non numeric go releases (rc's and beta's) when finding latest go by @raulcd in [#968](https://github.com/apache/arrow-go/pull/968) +* ci: fix release workflow discussions by @zeroshade in [#980](https://github.com/apache/arrow-go/pull/980) +* ci: fix lint on release workflow by @zeroshade in [#982](https://github.com/apache/arrow-go/pull/982) +* fix: remove superfluous struct field index compare by @Willem-J-an in [#970](https://github.com/apache/arrow-go/pull/970) +* fix(ipc): reject truncated message frames by @fallintoplace in [#991](https://github.com/apache/arrow-go/pull/991) +* fix(internal/utils): handle reader reset and short peeks by @fallintoplace in [#987](https://github.com/apache/arrow-go/pull/987) +* fix(arrow/ipc): propagate CLI stream errors by @fallintoplace in [#986](https://github.com/apache/arrow-go/pull/986) +* fix(arrow/flight): avoid blocking after stream cancellation by @fallintoplace in [#989](https://github.com/apache/arrow-go/pull/989) +* fix(parquet/variant): handle empty metadata keys by @fallintoplace in [#985](https://github.com/apache/arrow-go/pull/985) +* fix(flight/flightsql/example): propagate row iteration errors by @fallintoplace in [#990](https://github.com/apache/arrow-go/pull/990) +* fix(parquet): reject truncated scalar bit reads by @fallintoplace in [#944](https://github.com/apache/arrow-go/pull/944) +* fix: Fix arm64 assembly stack frame manipulation by @nsrip-dd in [#984](https://github.com/apache/arrow-go/pull/984) +* fix(arrow/array): support zero-column tables by @fallintoplace in [#988](https://github.com/apache/arrow-go/pull/988) +* docs(parquet): Add thrift generation contribution info by @C-Loftus in [#971](https://github.com/apache/arrow-go/pull/971) +* fix(array): reject mismatched known null counts during full validation by @fallintoplace in [#993](https://github.com/apache/arrow-go/pull/993) +* fix(arrow/array): `Resize(0)` on non-empty builder causes memory leak by @serramatutu in [#995](https://github.com/apache/arrow-go/pull/995) +* feat(parquet/pqarrow): Read support for extension types / geo by @C-Loftus in [#969](https://github.com/apache/arrow-go/pull/969) +* fix(array): validate run-end encoded invariants by @fallintoplace in [#992](https://github.com/apache/arrow-go/pull/992) +* fix(array): validate binary_view and string_view payloads by @fallintoplace in [#994](https://github.com/apache/arrow-go/pull/994) +* fix(parquet): wrap DELTA_BINARY_PACKED deltas at the physical type width by @tom-eon in [#1027](https://github.com/apache/arrow-go/pull/1027) +* fix(arrow/compute): use is_null/is_not_null in expressions by @fallintoplace in [#997](https://github.com/apache/arrow-go/pull/997) +* fix(arrow/ipc): reject NaN minSpaceSavings values by @fallintoplace in [#998](https://github.com/apache/arrow-go/pull/998) +* fix(arrow/tensor): default nil dimension names to empty strings by @fallintoplace in [#999](https://github.com/apache/arrow-go/pull/999) +* fix(arrow/scalar): allow null extension scalars to keep null storage by @fallintoplace in [#1000](https://github.com/apache/arrow-go/pull/1000) +* fix(arrow/scalar): handle nil child values in run-end scalars by @fallintoplace in [#1001](https://github.com/apache/arrow-go/pull/1001) +* fix(arrow/array): ignore zero-count union appends by @fallintoplace in [#1026](https://github.com/apache/arrow-go/pull/1026) +* fix(arrow/csv): preserve decimal precision by @fallintoplace in [#1022](https://github.com/apache/arrow-go/pull/1022) +* fix(parquet/metadata): compare bloom filter caps in bits by @fallintoplace in [#996](https://github.com/apache/arrow-go/pull/996) +* fix(parquet/pqarrow): reject timestamp coercion overflow by @fallintoplace in [#1008](https://github.com/apache/arrow-go/pull/1008) +* fix(parquet/file): reject records starting with nonzero repetition levels by @fallintoplace in [#1009](https://github.com/apache/arrow-go/pull/1009) +* fix(parquet/variant): reject truncated scalar values by @fallintoplace in [#1017](https://github.com/apache/arrow-go/pull/1017) +* fix(parquet/encoding): reject invalid delta byte prefixes by @fallintoplace in [#1014](https://github.com/apache/arrow-go/pull/1014) +* fix(parquet/metadata): preserve infinite floating-point statistics by @fallintoplace in [#1007](https://github.com/apache/arrow-go/pull/1007) +* fix(parquet/encoding): bound lazy dictionary growth by @fallintoplace in [#1011](https://github.com/apache/arrow-go/pull/1011) +* fix(parquet/encoding): respect declared RLE payload lengths by @fallintoplace in [#1015](https://github.com/apache/arrow-go/pull/1015) +* fix(parquet/encoding): validate delta binary packed headers by @fallintoplace in [#1012](https://github.com/apache/arrow-go/pull/1012) +* fix(parquet/encoding): reject invalid delta length payloads by @fallintoplace in [#1013](https://github.com/apache/arrow-go/pull/1013) +* fix(parquet/pqarrow): preserve INT96 timestamps across units by @fallintoplace in [#1006](https://github.com/apache/arrow-go/pull/1006) +* fix(parquet/pqarrow): Change interface name for arrow type on read for extension types by @C-Loftus in [#1028](https://github.com/apache/arrow-go/pull/1028) +* fix(arrow/csv): respect timestamp timezones by @fallintoplace in [#1025](https://github.com/apache/arrow-go/pull/1025) +* fix(parquet/compress): avoid closing caller streams for uncompressed wrappers by @fallintoplace in [#1003](https://github.com/apache/arrow-go/pull/1003) +* fix(parquet/compress): allocate uncompressed output for short destinations by @fallintoplace in [#1004](https://github.com/apache/arrow-go/pull/1004) +* fix(internal/utils): keep byteReader position after failed seeks by @fallintoplace in [#1005](https://github.com/apache/arrow-go/pull/1005) +* fix(parquet/file): count repeated skips in logical rows by @fallintoplace in [#1002](https://github.com/apache/arrow-go/pull/1002) +* fix(parquet/metadata): reject malformed fixed-width statistics by @fallintoplace in [#1010](https://github.com/apache/arrow-go/pull/1010) +* fix(arrow/array): skip empty table chunks by @fallintoplace in [#1023](https://github.com/apache/arrow-go/pull/1023) +* fix(arrow/scalar): reject negative array lengths by @fallintoplace in [#1024](https://github.com/apache/arrow-go/pull/1024) +* fix(arrow/array): validate fixed-size list child lengths by @fallintoplace in [#1019](https://github.com/apache/arrow-go/pull/1019) +* fix(parquet/pqarrow): calculate large decimal widths by @fallintoplace in [#1034](https://github.com/apache/arrow-go/pull/1034) +* fix(parquet/schema): reject invalid fixed lengths by @fallintoplace in [#1035](https://github.com/apache/arrow-go/pull/1035) +* fix(parquet/encoding): reject truncated plain boolean data by @fallintoplace in [#1031](https://github.com/apache/arrow-go/pull/1031) +* fix(parquet/metadata): return page index decode errors by @fallintoplace in [#1032](https://github.com/apache/arrow-go/pull/1032) +* fix(parquet/metadata): validate adaptive bloom options by @fallintoplace in [#1033](https://github.com/apache/arrow-go/pull/1033) +* fix(tmpl): report generated output write errors by @fallintoplace in [#1037](https://github.com/apache/arrow-go/pull/1037) +* fix(arrow/array): propagate embedded pointer nullability by @fallintoplace in [#1038](https://github.com/apache/arrow-go/pull/1038) +* fix(flight/flightsql/driver): convert Date32 values by @fallintoplace in [#1039](https://github.com/apache/arrow-go/pull/1039) +* fix(arrow/scalar): support same-width time casts by @fallintoplace in [#1040](https://github.com/apache/arrow-go/pull/1040) +* fix(arrow/scalar): use duration type for units by @fallintoplace in [#1041](https://github.com/apache/arrow-go/pull/1041) +* fix(arrow/array): validate union child upper bound by @fallintoplace in [#1043](https://github.com/apache/arrow-go/pull/1043) +* refactor(arrow/avro): migrate from hamba/avro to twmb/avro by @prochac in [#830](https://github.com/apache/arrow-go/pull/830) +* fix(arrow/array): validate struct child lengths by @fallintoplace in [#1021](https://github.com/apache/arrow-go/pull/1021) +* fix(arrow/cdata): validate imported schema topology by @fallintoplace in [#1046](https://github.com/apache/arrow-go/pull/1046) +* fix(parquet/metadata): validate row-group ordinals in PageIndexReader.WillNeed by @fallintoplace in [#1057](https://github.com/apache/arrow-go/pull/1057) +* fix(arrow/flight): limit descriptors to next payload by @fallintoplace in [#1042](https://github.com/apache/arrow-go/pull/1042) +* fix(parquet): propagate nonce generation failures by @fallintoplace in [#1045](https://github.com/apache/arrow-go/pull/1045) +* fix(parquet/metadata): reject negative row-group indices in Subset by @fallintoplace in [#1055](https://github.com/apache/arrow-go/pull/1055) +* fix(parquet/metadata): use default writer properties when props is nil by @fallintoplace in [#1058](https://github.com/apache/arrow-go/pull/1058) +* fix(parquet/metadata): return schema initialization errors by @fallintoplace in [#1056](https://github.com/apache/arrow-go/pull/1056) +* fix(parquet/pqarrow): propagate level builder errors by @fallintoplace in [#1049](https://github.com/apache/arrow-go/pull/1049) +* fix(arrow/flight/session): clone option values by @fallintoplace in [#1044](https://github.com/apache/arrow-go/pull/1044) +* fix(parquet/variant): validate metadata offset tables by @fallintoplace in [#1063](https://github.com/apache/arrow-go/pull/1063) +* fix(arrow/csv): release Reader record builders by @fallintoplace in [#1059](https://github.com/apache/arrow-go/pull/1059) +* fix(parquet/pqarrow): release partially constructed field readers by @fallintoplace in [#1061](https://github.com/apache/arrow-go/pull/1061) +* fix(arrow/array): make dictionary suffix null indexes relative by @fallintoplace in [#1064](https://github.com/apache/arrow-go/pull/1064) +* fix(parquet/pqarrow): prefer stored schema extension type on read by @twuebi in [#1051](https://github.com/apache/arrow-go/pull/1051) +* fix(arrow/flight/flightsql): avoid double-retaining parameter readers by @fallintoplace in [#1066](https://github.com/apache/arrow-go/pull/1066) +* fix(flightsql): isolate example SQLite databases by @fallintoplace in [#1067](https://github.com/apache/arrow-go/pull/1067) +* fix(arrow): return an owned slice from Schema.FieldsByName by @fallintoplace in [#1069](https://github.com/apache/arrow-go/pull/1069) +* fix(arrow/ipc): return errors for malformed file record batches by @fallintoplace in [#1062](https://github.com/apache/arrow-go/pull/1062) +* fix(flight): return Unauthenticated for invalid unary auth tokens by @fallintoplace in [#1070](https://github.com/apache/arrow-go/pull/1070) +* fix(arrow/array): always release JSON reader builder by @fallintoplace in [#1065](https://github.com/apache/arrow-go/pull/1065) +* fix(flight): accept case-insensitive authorization schemes by @fallintoplace in [#1071](https://github.com/apache/arrow-go/pull/1071) +* fix(parquet): reject short page writes by @fallintoplace in [#1050](https://github.com/apache/arrow-go/pull/1050) +* fix(parquet/file): close file on OpenParquetFile errors by @fallintoplace in [#1053](https://github.com/apache/arrow-go/pull/1053) +* fix(parquet): reject truncated footer signatures by @fallintoplace in [#1074](https://github.com/apache/arrow-go/pull/1074) +* fix(arrow/ipc): reject non-dictionary initial messages by @fallintoplace in [#1054](https://github.com/apache/arrow-go/pull/1054) +* feat(flightsql): allow configuring the driver memory allocator by @fallintoplace in [#1076](https://github.com/apache/arrow-go/pull/1076) +* fix(arrow): avoid retains on invalid chunk construction by @fallintoplace in [#1078](https://github.com/apache/arrow-go/pull/1078) +* fix(parquet): row-align V1 pages with offset indexes by @fallintoplace in [#1075](https://github.com/apache/arrow-go/pull/1075) +* fix(arrow/array): clean up partial table construction by @fallintoplace in [#1077](https://github.com/apache/arrow-go/pull/1077) +* fix(parquet_reader): propagate batch read errors by @fallintoplace in [#1081](https://github.com/apache/arrow-go/pull/1081) +* fix(arrow): isolate schemas created by AddField by @fallintoplace in [#1079](https://github.com/apache/arrow-go/pull/1079) +* fix(arrow/util): avoid protobuf map schema goroutines by @fallintoplace in [#1082](https://github.com/apache/arrow-go/pull/1082) +* fix(array): format date arrays in String by @fallintoplace in [#1068](https://github.com/apache/arrow-go/pull/1068) +* fix(parquet): report initialized row-group columns correctly by @fallintoplace in [#1072](https://github.com/apache/arrow-go/pull/1072) +* fix(parquet): serialize column chunk file offsets by @fallintoplace in [#1073](https://github.com/apache/arrow-go/pull/1073) +* fix(arrow/util): release protobuf record builders by @fallintoplace in [#1084](https://github.com/apache/arrow-go/pull/1084) +* fix(arrow/util): release protobuf enum dictionaries by @fallintoplace in [#1083](https://github.com/apache/arrow-go/pull/1083) +* fix(arrow/avro): stop reader workers before reuse by @fallintoplace in [#1048](https://github.com/apache/arrow-go/pull/1048) +* fix(parquet): restore Arrow duration units from schema metadata by @fallintoplace in [#1095](https://github.com/apache/arrow-go/pull/1095) +* fix(compute): round negative HalfToOdd ties correctly by @fallintoplace in [#1094](https://github.com/apache/arrow-go/pull/1094) +* perf(arrow/bitutil): add SetBitSwap/ClearBitSwap, use in builder.SetNull by @tom-eon in [#1090](https://github.com/apache/arrow-go/pull/1090) +* fix(array): compare sparse union slices at relative offsets by @fallintoplace in [#1096](https://github.com/apache/arrow-go/pull/1096) +* fix(arrow/ipc): make writer failures terminal by @fallintoplace in [#1047](https://github.com/apache/arrow-go/pull/1047) +* feat(parquet): add WithDictionaryCostFallback writer property by @tom-eon in [#1036](https://github.com/apache/arrow-go/pull/1036) +* fix(parquet/pqarrow): release extension arrays after chunking by @fallintoplace in [#1060](https://github.com/apache/arrow-go/pull/1060) +* fix(parquet/encoding): read ahead in the streaming value buffer by @joechenrh in [#937](https://github.com/apache/arrow-go/pull/937) +* fix(parquet/variant): preserve metadata keys when cloning by @fallintoplace in [#1080](https://github.com/apache/arrow-go/pull/1080) +* test(arrow/util): cover zero-field protobuf records by @fallintoplace in [#1085](https://github.com/apache/arrow-go/pull/1085) +* fix(arrow/scalar): preserve metadata in FromScalar by @fallintoplace in [#1100](https://github.com/apache/arrow-go/pull/1100) +* fix(arrow/scalar): preserve binary cast target types by @fallintoplace in [#1101](https://github.com/apache/arrow-go/pull/1101) +* fix(arrow/scalar): support boolean identity casts by @fallintoplace in [#1102](https://github.com/apache/arrow-go/pull/1102) +* fix(arrow/util): handle invalid protobuf Any values by @fallintoplace in [#1086](https://github.com/apache/arrow-go/pull/1086) +* fix(arrow/array): avoid copying initialized extension refcounts by @fallintoplace in [#1089](https://github.com/apache/arrow-go/pull/1089) +* fix(parquet): preserve dictionary index types by @fallintoplace in [#1098](https://github.com/apache/arrow-go/pull/1098) +* fix(parquet_reader): propagate statistics decoding errors by @fallintoplace in [#1087](https://github.com/apache/arrow-go/pull/1087) +* fix(arrow/array): avoid panics cleaning partial tables by @fallintoplace in [#1103](https://github.com/apache/arrow-go/pull/1103) +* fix(arrow/ipc): release reader state on schema errors by @fallintoplace in [#1112](https://github.com/apache/arrow-go/pull/1112) +* fix(arrow/scalar): format extension values from storage scalars by @fallintoplace in [#1111](https://github.com/apache/arrow-go/pull/1111) +* fix(arrow/array): isolate table columns from caller slices by @fallintoplace in [#1105](https://github.com/apache/arrow-go/pull/1105) +* fix(arrow/scalar): preserve exact decimal string casts by @fallintoplace in [#1107](https://github.com/apache/arrow-go/pull/1107) +* fix(arrow/tensor): validate typed constructor data by @fallintoplace in [#1104](https://github.com/apache/arrow-go/pull/1104) +* fix(arrow/scalar): format interval scalars in String by @fallintoplace in [#1115](https://github.com/apache/arrow-go/pull/1115) +* fix(arrow/array): preserve rows for empty-schema JSON input by @fallintoplace in [#1117](https://github.com/apache/arrow-go/pull/1117) +* fix(arrow/util): return protobuf record conversion errors by @fallintoplace in [#1088](https://github.com/apache/arrow-go/pull/1088) +* fix(arrow/scalar): validate FromScalar destinations by @fallintoplace in [#1106](https://github.com/apache/arrow-go/pull/1106) +* fix(arrow/scalar): use wide offsets for large binary scalars by @fallintoplace in [#1108](https://github.com/apache/arrow-go/pull/1108) +* fix(arrow/scalar): apply timestamp zones when casting to dates by @fallintoplace in [#1110](https://github.com/apache/arrow-go/pull/1110) +* fix(arrow/ipc): validate file block framing and lengths by @fallintoplace in [#1097](https://github.com/apache/arrow-go/pull/1097) +* fix(arrow/scalar): release partial children after failed struct extraction by @fallintoplace in [#1118](https://github.com/apache/arrow-go/pull/1118) +* fix(arrow/ipc): return errors for invalid record indexes by @fallintoplace in [#1120](https://github.com/apache/arrow-go/pull/1120) +* fix(parquet): read null fixed-size-list parents by @fallintoplace in [#1099](https://github.com/apache/arrow-go/pull/1099) +* fix(arrow/scalar): release partial children after failed struct conversion by @fallintoplace in [#1114](https://github.com/apache/arrow-go/pull/1114) +* fix(parquet/metadata): return error for missing column metadata by @fallintoplace in [#1137](https://github.com/apache/arrow-go/pull/1137) +* fix(arrow/array): widen dictionary index bounds checks by @fallintoplace in [#1129](https://github.com/apache/arrow-go/pull/1129) +* fix(compute/exprs): avoid empty field reference panic by @fallintoplace in [#1130](https://github.com/apache/arrow-go/pull/1130) +* fix(compute): handle odd temporal multiples by @fallintoplace in [#1119](https://github.com/apache/arrow-go/pull/1119) +* fix(arrow/csv): preserve first row during headerless inference by @fallintoplace in [#1145](https://github.com/apache/arrow-go/pull/1145) +* fix(arrow/array): bound table reader batches by @fallintoplace in [#1146](https://github.com/apache/arrow-go/pull/1146) +* fix(arrow/extensions): compare JSON storage types semantically by @fallintoplace in [#1150](https://github.com/apache/arrow-go/pull/1150) +* fix(arrow/ipc): preserve minSpaceSavings in stream writer by @fallintoplace in [#1143](https://github.com/apache/arrow-go/pull/1143) +* fix(parquet/compress): honor LZ4 raw destination contract by @fallintoplace in [#1147](https://github.com/apache/arrow-go/pull/1147) +* fix(arrow): reject duplicate union type codes by @fallintoplace in [#1131](https://github.com/apache/arrow-go/pull/1131) +* fix(parquet/metadata): handle invalid row-group column indexes by @fallintoplace in [#1125](https://github.com/apache/arrow-go/pull/1125) +* fix(compute/exec): handle missing validity buffers by @fallintoplace in [#1136](https://github.com/apache/arrow-go/pull/1136) +* fix(compute/exec): report ListView buffer counts by @fallintoplace in [#1133](https://github.com/apache/arrow-go/pull/1133) +* fix(arrow): include KeysSorted in map equality by @fallintoplace in [#1134](https://github.com/apache/arrow-go/pull/1134) +* fix(compute): reject non-array-like filters by @fallintoplace in [#1135](https://github.com/apache/arrow-go/pull/1135) +* fix(memory): handle zero-length checked reallocations by @fallintoplace in [#1132](https://github.com/apache/arrow-go/pull/1132) +* fix(parquet/pqarrow): validate reader indexes by @fallintoplace in [#1124](https://github.com/apache/arrow-go/pull/1124) +* fix(arrow/csv): validate custom converter row counts by @fallintoplace in [#1148](https://github.com/apache/arrow-go/pull/1148) +* fix(parquet/encoding): honor BufferWriter offsets by @fallintoplace in [#1144](https://github.com/apache/arrow-go/pull/1144) +* fix(flightsql): stop row streaming after close by @fallintoplace in [#1151](https://github.com/apache/arrow-go/pull/1151) +* fix(flightsql): protect configured authorization metadata by @fallintoplace in [#1153](https://github.com/apache/arrow-go/pull/1153) +* perf(parquet/pqarrow): avoid reflection in integer transfers by @fallintoplace in [#1163](https://github.com/apache/arrow-go/pull/1163) +* fix(flightsql): reject unsupported transaction options by @fallintoplace in [#1152](https://github.com/apache/arrow-go/pull/1152) +* fix(arrow/scalar): handle zero-length scalar arrays by @fallintoplace in [#1116](https://github.com/apache/arrow-go/pull/1116) +* perf(arrow/array): pack validity values in batches by @fallintoplace in [#1185](https://github.com/apache/arrow-go/pull/1185) +* feat(arrow/array): roll back JSON builder state after failed rows by @fallintoplace in [#1113](https://github.com/apache/arrow-go/pull/1113) +* fix(arrow/ipc): avoid deadlock on compression errors by @fallintoplace in [#1142](https://github.com/apache/arrow-go/pull/1142) +* fix(arrow/csv): append null for later parse failures by @fallintoplace in [#1149](https://github.com/apache/arrow-go/pull/1149) +* feat(arrow/array): add ValueAsAny for native Go values by @dgvj-work in [#1170](https://github.com/apache/arrow-go/pull/1170) +* perf(arrow/array): bulk append fixed-size list nulls and empty values by @fallintoplace in [#1183](https://github.com/apache/arrow-go/pull/1183) +* perf(parquet): encode spaced booleans without compaction by @fallintoplace in [#1175](https://github.com/apache/arrow-go/pull/1175) +* fix(parquet/pqarrow): reject decimal overflow in integer columns by @fallintoplace in [#1122](https://github.com/apache/arrow-go/pull/1122) +* fix(arrow/array/arreflect): reject unrepresentable timestamps by @fallintoplace in [#1123](https://github.com/apache/arrow-go/pull/1123) +* feat(compute): add cumulative_sum and cumulative_sum_checked by @fallintoplace in [#1139](https://github.com/apache/arrow-go/pull/1139) +* fix(arrow/ipc): return compression errors from record encoder by @fallintoplace in [#1141](https://github.com/apache/arrow-go/pull/1141) +* perf(arrow/array): bulk append fixed-width empty values by @fallintoplace in [#1180](https://github.com/apache/arrow-go/pull/1180) +* perf(arrow/csv): reserve record builders for chunked reads by @fallintoplace in [#1198](https://github.com/apache/arrow-go/pull/1198) +* perf(arrow/array): compare list values by valid runs by @fallintoplace in [#1184](https://github.com/apache/arrow-go/pull/1184) +* perf(parquet): optimize numeric dictionary insertion by @fallintoplace in [#1178](https://github.com/apache/arrow-go/pull/1178) +* feat(arrow/flight/sql): Add is_update field to ActionCreatePreparedStatementResult by @ennuite in [#732](https://github.com/apache/arrow-go/pull/732) +* feat(compute): add dictionary_encode by @fallintoplace in [#1140](https://github.com/apache/arrow-go/pull/1140) +* perf(parquet): batch plain BYTE_ARRAY encoding by @fallintoplace in [#1193](https://github.com/apache/arrow-go/pull/1193) +* fix(arrow/csv): append null for invalid fixed-size binary values by @fallintoplace in [#1121](https://github.com/apache/arrow-go/pull/1121) +* fix(parquet/pqarrow): reject out-of-range INT96 timestamps by @fallintoplace in [#1128](https://github.com/apache/arrow-go/pull/1128) +* perf(parquet): batch delta binary packed decoding by @fallintoplace in [#1169](https://github.com/apache/arrow-go/pull/1169) +* perf(arrow/array): bulk append list nulls and empty values by @fallintoplace in [#1187](https://github.com/apache/arrow-go/pull/1187) +* perf(parquet/pqarrow): decode booleans directly to bitmaps by @fallintoplace in [#1182](https://github.com/apache/arrow-go/pull/1182) +* perf(parquet/pqarrow): build nullable fixed-size list children directly by @fallintoplace in [#1195](https://github.com/apache/arrow-go/pull/1195) +* perf(internal/hashing): build binary memo offsets in one pass by @fallintoplace in [#1196](https://github.com/apache/arrow-go/pull/1196) +* perf(arrow/compute): take chunked binary values without concatenation by @fallintoplace in [#1199](https://github.com/apache/arrow-go/pull/1199) +* perf(arrow/array): reuse TableReader scratch slices by @fallintoplace in [#1200](https://github.com/apache/arrow-go/pull/1200) +* fix(parquet/pqarrow): reject decimal overflow when writing integers by @fallintoplace in [#1161](https://github.com/apache/arrow-go/pull/1161) +* fix(arrow/array): preserve run-end encoded empty values by @fallintoplace in [#1156](https://github.com/apache/arrow-go/pull/1156) +* fix(compute): preserve caller context during execution by @fallintoplace in [#1154](https://github.com/apache/arrow-go/pull/1154) +* fix(compute): propagate nested expression serialization errors by @fallintoplace in [#1160](https://github.com/apache/arrow-go/pull/1160) +* fix(compute): report FuncMeta for meta functions by @fallintoplace in [#1155](https://github.com/apache/arrow-go/pull/1155) +* perf(arrow/array): optimize bulk appends for binary builders by @fallintoplace in [#1188](https://github.com/apache/arrow-go/pull/1188) +* perf(arrow/array): avoid temporary buffers when concatenating by @fallintoplace in [#1194](https://github.com/apache/arrow-go/pull/1194) +* perf(compute): batch contiguous filter take indices by @fallintoplace in [#1197](https://github.com/apache/arrow-go/pull/1197) +* perf(arrow/ipc): slice already-zero-based offset buffers by @fallintoplace in [#1191](https://github.com/apache/arrow-go/pull/1191) +* perf(parquet): encode RLE levels in batches by @fallintoplace in [#1177](https://github.com/apache/arrow-go/pull/1177) +* fix(arrow): include REE value nullability in type equality by @fallintoplace in [#1158](https://github.com/apache/arrow-go/pull/1158) +* fix(parquet): build bloom filters from dictionary entries by @fallintoplace in [#1164](https://github.com/apache/arrow-go/pull/1164) +* fix(arrow): validate precision in NewDecimalType by @fallintoplace in [#1162](https://github.com/apache/arrow-go/pull/1162) +* perf(arrow/csv): avoid materializing rows before writing by @fallintoplace in [#1190](https://github.com/apache/arrow-go/pull/1190) +* perf(arrow/array): batch run-end encoded appends by @fallintoplace in [#1214](https://github.com/apache/arrow-go/pull/1214) +* fix(arrow/array): validate union JSON type codes by @fallintoplace in [#1157](https://github.com/apache/arrow-go/pull/1157) +* fix(parquet/variant): validate compound value bounds by @fallintoplace in [#1126](https://github.com/apache/arrow-go/pull/1126) +* perf(arrow/array): validate dictionary indices by valid runs by @fallintoplace in [#1181](https://github.com/apache/arrow-go/pull/1181) +* perf(parquet): bound Bloom filter hash batches by @fallintoplace in [#1218](https://github.com/apache/arrow-go/pull/1218) +* perf(arrow/array): optimize bulk appends for struct builders by @fallintoplace in [#1189](https://github.com/apache/arrow-go/pull/1189) +* fix(arrow): return owned field index slices by @fallintoplace in [#1159](https://github.com/apache/arrow-go/pull/1159) +* perf(arrow/array): avoid temporary slices in Diff comparisons by @fallintoplace in [#1239](https://github.com/apache/arrow-go/pull/1239) +* perf(parquet/compress): reuse gzip writers by @fallintoplace in [#1238](https://github.com/apache/arrow-go/pull/1238) +* perf(parquet): reuse fixed-length zero scratch by @fallintoplace in [#1236](https://github.com/apache/arrow-go/pull/1236) +* perf(compute): share logical row maps across aligned sorts by @fallintoplace in [#1234](https://github.com/apache/arrow-go/pull/1234) +* perf(compute): fast-path single-span execution by @fallintoplace in [#1233](https://github.com/apache/arrow-go/pull/1233) +* perf(parquet): reuse boolean spaced scratch by @fallintoplace in [#1231](https://github.com/apache/arrow-go/pull/1231) +* perf(parquet): reuse delta PutSpaced scratch by @fallintoplace in [#1230](https://github.com/apache/arrow-go/pull/1230) +* perf(arrow/array): remove map validity slice by @fallintoplace in [#1228](https://github.com/apache/arrow-go/pull/1228) +* perf(arrow/array): copy same-dictionary indices directly by @fallintoplace in [#1225](https://github.com/apache/arrow-go/pull/1225) +* perf(parquet): reuse dictionary decoder scratch by @fallintoplace in [#1222](https://github.com/apache/arrow-go/pull/1222) +* perf(arrow/array): compare fixed-width values in bulk by @fallintoplace in [#1171](https://github.com/apache/arrow-go/pull/1171) +* perf(arrow/array): compare boolean values by valid runs by @fallintoplace in [#1186](https://github.com/apache/arrow-go/pull/1186) +* perf(arrow/array): avoid temporary buffers for binary concatenation by @fallintoplace in [#1215](https://github.com/apache/arrow-go/pull/1215) +* perf(arrow/array): batch union null and empty appends by @fallintoplace in [#1216](https://github.com/apache/arrow-go/pull/1216) +* perf(parquet): decode levels directly into int16 batches by @fallintoplace in [#1176](https://github.com/apache/arrow-go/pull/1176) +* perf(parquet): batch DELTA_BYTE_ARRAY encoding by @fallintoplace in [#1179](https://github.com/apache/arrow-go/pull/1179) +* perf(parquet/variant): skip metadata construction in Encode by @fallintoplace in [#1217](https://github.com/apache/arrow-go/pull/1217) +* perf(parquet): decode BSS FLBA into contiguous storage by @fallintoplace in [#1172](https://github.com/apache/arrow-go/pull/1172) +* perf(parquet): stop spaced expansion once aligned by @fallintoplace in [#1220](https://github.com/apache/arrow-go/pull/1220) +* perf(arrow/ipc): reuse record encoder scratch by @fallintoplace in [#1221](https://github.com/apache/arrow-go/pull/1221) +* perf(arrow/array): batch dictionary empty appends by @fallintoplace in [#1223](https://github.com/apache/arrow-go/pull/1223) +* perf(compute): reserve binary view cast payload by @fallintoplace in [#1227](https://github.com/apache/arrow-go/pull/1227) +* perf(compute): use stack scratch for null propagation by @fallintoplace in [#1229](https://github.com/apache/arrow-go/pull/1229) +* perf(arrow/array): bulk append BinaryView nulls by @fallintoplace in [#1232](https://github.com/apache/arrow-go/pull/1232) +* perf(arrow/array): use unsafe writes for binary bulk appends by @fallintoplace in [#1240](https://github.com/apache/arrow-go/pull/1240) +* perf(arrow/array): compare union values in runs by @fallintoplace in [#1244](https://github.com/apache/arrow-go/pull/1244) +* perf(arrow/array): batch-pack boolean values by @fallintoplace in [#1237](https://github.com/apache/arrow-go/pull/1237) +* fix(parquet/pqarrow): require Variant value field when writing by @dgvj-work in [#1242](https://github.com/apache/arrow-go/pull/1242) +* perf(arrow/compute): avoid goroutines for serial record filtering by @fallintoplace in [#1245](https://github.com/apache/arrow-go/pull/1245) +* perf(parquet): reuse dictionary RLE decoder across pages by @fallintoplace in [#1247](https://github.com/apache/arrow-go/pull/1247) +* perf(parquet/pqarrow): write record ranges directly by @fallintoplace in [#1248](https://github.com/apache/arrow-go/pull/1248) +* perf(parquet/pqarrow): avoid goroutines for serial struct reads by @fallintoplace in [#1250](https://github.com/apache/arrow-go/pull/1250) +* perf(parquet): decode RLE booleans into bitmaps by @fallintoplace in [#1246](https://github.com/apache/arrow-go/pull/1246) +* feat(compute): add list_element by @fallintoplace in [#1138](https://github.com/apache/arrow-go/pull/1138) +* perf(parquet): reuse DataPageV2 assembly buffer for eager writes by @fallintoplace in [#1192](https://github.com/apache/arrow-go/pull/1192) +* perf(arrow/array): append fixed-width nulls in bulk by @fallintoplace in [#1173](https://github.com/apache/arrow-go/pull/1173) +* perf(arrow/array): concatenate bitmaps directly by @fallintoplace in [#1224](https://github.com/apache/arrow-go/pull/1224) +* perf(arrow/compute): avoid goroutines for serial take by @fallintoplace in [#1226](https://github.com/apache/arrow-go/pull/1226) +* perf(compute): use typed memo insertion for numeric hashes by @fallintoplace in [#1251](https://github.com/apache/arrow-go/pull/1251) +* perf(parquet): decode DELTA_BYTE_ARRAY into contiguous storage by @fallintoplace in [#1252](https://github.com/apache/arrow-go/pull/1252) +* perf(parquet/pqarrow): use synchronous path for serial reads by @fallintoplace in [#1235](https://github.com/apache/arrow-go/pull/1235) +* perf(arrow/array): reuse full chunks in NewChunkedSlice by @fallintoplace in [#1257](https://github.com/apache/arrow-go/pull/1257) +* perf(parquet): reuse boolean Bloom filter hashes by @fallintoplace in [#1258](https://github.com/apache/arrow-go/pull/1258) +* perf(arrow/ipc): release dictionary payloads eagerly by @fallintoplace in [#1259](https://github.com/apache/arrow-go/pull/1259) +* perf(parquet): SIMD BYTE_STREAM_SPLIT encoding by @fallintoplace in [#1260](https://github.com/apache/arrow-go/pull/1260) +* perf(parquet): reuse DELTA byte-array scratch by @fallintoplace in [#1261](https://github.com/apache/arrow-go/pull/1261) +* perf(parquet/pqarrow): write FixedSizeBinary values directly by @fallintoplace in [#1263](https://github.com/apache/arrow-go/pull/1263) +* feat(extensions): add VariantGet for path extraction from variant arrays by @nssalian in [#1206](https://github.com/apache/arrow-go/pull/1206) +* perf(parquet): use typed memo insertion for byte-array dictionaries by @derekperkins in [#1272](https://github.com/apache/arrow-go/pull/1272) +* fix(arrow/array): clamp run ends when concatenating a sliced RunEndEncoded array by @winklemad in [#1219](https://github.com/apache/arrow-go/pull/1219) +* fix(parquet): guard byte-stream-split encoder release against never-flushed buffer by @tom-eon in [#1273](https://github.com/apache/arrow-go/pull/1273) +* perf(bitutil): add ARM64 NEON bitmap operations by @fallintoplace in [#1264](https://github.com/apache/arrow-go/pull/1264) +* perf(arrow/array): compare binary values by valid runs by @fallintoplace in [#1174](https://github.com/apache/arrow-go/pull/1174) +* perf(compute): add ARM64 NEON arithmetic kernels by @fallintoplace in [#1265](https://github.com/apache/arrow-go/pull/1265) +* perf(parquet/compress): reuse Brotli writers across encodes by @fallintoplace in [#1254](https://github.com/apache/arrow-go/pull/1254) +* perf(arrow/array): skip full-range SliceEqual slices by @fallintoplace in [#1249](https://github.com/apache/arrow-go/pull/1249) +* perf(parquet): vectorize fixed-width dictionary materialization by @fallintoplace in [#1269](https://github.com/apache/arrow-go/pull/1269) +* perf(compute): add ARM64 NEON comparison kernels by @fallintoplace in [#1266](https://github.com/apache/arrow-go/pull/1266) +* perf(parquet/pqarrow): write binary values from Arrow offsets by @fallintoplace in [#1262](https://github.com/apache/arrow-go/pull/1262) +* fix(arrow/extensions): reject Null Variant typed_value by @dgvj-work in [#1243](https://github.com/apache/arrow-go/pull/1243) +* fix(arrow/extensions): use canonical Variant extension name by @dgvj-work in [#1241](https://github.com/apache/arrow-go/pull/1241) +* chore: bump version number by @zeroshade in [#1276](https://github.com/apache/arrow-go/pull/1276) +* test(parquet): reduce macOS race test memory pressure by @zeroshade in [#1275](https://github.com/apache/arrow-go/pull/1275) +* fix(parquet): explicitly release bloom builders by @zeroshade in [#1280](https://github.com/apache/arrow-go/pull/1280) +* ci: retry and cache dependency downloads by @zeroshade in [#1279](https://github.com/apache/arrow-go/pull/1279) +* fix(compute): discard results after cancellation by @zeroshade in [#1278](https://github.com/apache/arrow-go/pull/1278) +* ci: use bundled GTest for conda integration by @zeroshade in [#1277](https://github.com/apache/arrow-go/pull/1277) +* fix(parquet): stop BYTE_STREAM_SPLIT FLBA spaced decode aliasing output by @zeroshade in [#1256](https://github.com/apache/arrow-go/pull/1256) +* perf(compute): vectorize mixed int32 filters with AVX2 by @fallintoplace in [#1284](https://github.com/apache/arrow-go/pull/1284) +* perf(compute): add ARM64 NEON constant multiplication by @fallintoplace in [#1268](https://github.com/apache/arrow-go/pull/1268) +* perf(compute): vectorize numeric to boolean casts by @fallintoplace in [#1282](https://github.com/apache/arrow-go/pull/1282) +* fix(arrow/scalar): align timestamp timezone parsing by @fallintoplace in [#1109](https://github.com/apache/arrow-go/pull/1109) +* perf(internal/utils): add ARM64 NEON min/max for 8- and 16-bit integers by @fallintoplace in [#1281](https://github.com/apache/arrow-go/pull/1281) +* perf(arrow/array): SIMD-pack boolean slices into bitmaps by @fallintoplace in [#1283](https://github.com/apache/arrow-go/pull/1283) +* chore: fix failing build tags by @zeroshade in [#1285](https://github.com/apache/arrow-go/pull/1285) +* perf(compute): SIMD-compact fragmented filters into take indices by @fallintoplace in [#1287](https://github.com/apache/arrow-go/pull/1287) +* perf(compute): add ARM64 NEON comparisons for 8- and 16-bit integers by @fallintoplace in [#1288](https://github.com/apache/arrow-go/pull/1288) +* perf(compute): add ARM64 NEON compression for null-free 32-bit fixed-width filters by @fallintoplace in [#1286](https://github.com/apache/arrow-go/pull/1286) + +### New Contributors +* @C-Loftus made their first contribution in [#960](https://github.com/apache/arrow-go/pull/960) +* @tom-eon made their first contribution in [#1027](https://github.com/apache/arrow-go/pull/1027) +* @prochac made their first contribution in [#830](https://github.com/apache/arrow-go/pull/830) +* @dgvj-work made their first contribution in [#1170](https://github.com/apache/arrow-go/pull/1170) +* @ennuite made their first contribution in [#732](https://github.com/apache/arrow-go/pull/732) +* @derekperkins made their first contribution in [#1272](https://github.com/apache/arrow-go/pull/1272) +* @winklemad made their first contribution in [#1219](https://github.com/apache/arrow-go/pull/1219) + +**Full Changelog**: https://github.com/apache/arrow-go/compare/v18.7.0...v18.8.0 From 48cb1e18dec8bb791098dcd2aef39b8b24d72d1c Mon Sep 17 00:00:00 2001 From: Matt Topol Date: Tue, 8 Sep 2026 15:40:49 -0400 Subject: [PATCH 2/3] add highlights and improve changelog --- _posts/2026-09-08-arrow-go-18.8.0.md | 383 +++++++++------------------ 1 file changed, 119 insertions(+), 264 deletions(-) diff --git a/_posts/2026-09-08-arrow-go-18.8.0.md b/_posts/2026-09-08-arrow-go-18.8.0.md index 7f24518d9696..af5d66ca5580 100644 --- a/_posts/2026-09-08-arrow-go-18.8.0.md +++ b/_posts/2026-09-08-arrow-go-18.8.0.md @@ -48,293 +48,148 @@ $ git shortlog -sn v18.7.0..v18.8.0 1 Willem Jan ``` +## Highlights + +### Arrow + +#### Features + +- Added `VariantGet` for efficient path extraction from variant arrays [#1206](https://github.com/apache/arrow-go/pull/1206) +- Added `list_element`, `cumulative_sum`, `cumulative_sum_checked`, and `dictionary_encode` compute functions [#1138](https://github.com/apache/arrow-go/pull/1138), +[#1139](https://github.com/apache/arrow-go/pull/1139), [#1140](https://github.com/apache/arrow-go/pull/1140) +- Added `ValueAsAny` for accessing Arrow array values as native Go values [#1170](https://github.com/apache/arrow-go/pull/1170) +- Strengthened IPC file validation for malformed block framing and lengths [#1097](https://github.com/apache/arrow-go/pull/1097) + +#### Bug Fixes +- Fixed a memory leak when calling `Resize(0)` on non-empty builders [#995](https://github.com/apache/arrow-go/pull/995) +- Preserved exact precision when casting strings to Decimal128 and Decimal256 scalars [#1107](https://github.com/apache/arrow-go/pull/1107) + +#### Performance Improvements + +- Expanded SIMD acceleration for compute arithmetic, comparisons, and filtering on ARM64 and AMD64 [#1265](https://github.com/apache/arrow-go/pull/1265), [#1266](https://github.com/apache/arrow-go/pull/1266), +[#1284](https://github.com/apache/arrow-go/pull/1284), [#1286](https://github.com/apache/arrow-go/pull/1286), [#1287](https://github.com/apache/arrow-go/pull/1287) +- Added SIMD-accelerated packing of boolean slices and validity data into Arrow bitmaps [#1283](https://github.com/apache/arrow-go/pull/1283) + +### Parquet + +#### Features + +- Added read and write support for GeoArrow extension types using Parquet geometry and geography logical types [#960](https://github.com/apache/arrow-go/pull/960), +[#969](https://github.com/apache/arrow-go/pull/969) +- Added codec-aware dictionary cost fallback and per-column writer controls [#1036](https://github.com/apache/arrow-go/pull/1036) + +#### Bug Fixes + +- Correctly populated Bloom filters from dictionary-encoded values [#1164](https://github.com/apache/arrow-go/pull/1164) +- Rejected decimal values that overflow Parquet INT32 or INT64 columns instead of silently narrowing them [#1122](https://github.com/apache/arrow-go/pull/1122), +[#1161](https://github.com/apache/arrow-go/pull/1161) +- Fixed reading fixed-size lists whose parent values are null [#1099](https://github.com/apache/arrow-go/pull/1099) + +#### Performance Improvements + +- Added SIMD BYTE_STREAM_SPLIT encoding on ARM64 and AMD64 [#1260](https://github.com/apache/arrow-go/pull/1260) +- Vectorized fixed-width dictionary materialization on AVX2 systems [#1269](https://github.com/apache/arrow-go/pull/1269) +- Reduced copying when writing binary and fixed-size binary Arrow values to Parquet [#1262](https://github.com/apache/arrow-go/pull/1262), [#1263](https://github.com/apache/arrow-go/pull/1263) + ## Changelog -### What's Changed -* feat(parquet/schema): Wire up logical type functionality for writing geo types by @C-Loftus in [#960](https://github.com/apache/arrow-go/pull/960) -* chore(dev/release): filter out non numeric go releases (rc's and beta's) when finding latest go by @raulcd in [#968](https://github.com/apache/arrow-go/pull/968) -* ci: fix release workflow discussions by @zeroshade in [#980](https://github.com/apache/arrow-go/pull/980) -* ci: fix lint on release workflow by @zeroshade in [#982](https://github.com/apache/arrow-go/pull/982) +### Selected Changes + +A selection of the user-facing changes in this release. See the full changelog link +below for all 285 merged pull requests. + +#### Arrow + * fix: remove superfluous struct field index compare by @Willem-J-an in [#970](https://github.com/apache/arrow-go/pull/970) * fix(ipc): reject truncated message frames by @fallintoplace in [#991](https://github.com/apache/arrow-go/pull/991) -* fix(internal/utils): handle reader reset and short peeks by @fallintoplace in [#987](https://github.com/apache/arrow-go/pull/987) -* fix(arrow/ipc): propagate CLI stream errors by @fallintoplace in [#986](https://github.com/apache/arrow-go/pull/986) -* fix(arrow/flight): avoid blocking after stream cancellation by @fallintoplace in [#989](https://github.com/apache/arrow-go/pull/989) -* fix(parquet/variant): handle empty metadata keys by @fallintoplace in [#985](https://github.com/apache/arrow-go/pull/985) -* fix(flight/flightsql/example): propagate row iteration errors by @fallintoplace in [#990](https://github.com/apache/arrow-go/pull/990) -* fix(parquet): reject truncated scalar bit reads by @fallintoplace in [#944](https://github.com/apache/arrow-go/pull/944) * fix: Fix arm64 assembly stack frame manipulation by @nsrip-dd in [#984](https://github.com/apache/arrow-go/pull/984) -* fix(arrow/array): support zero-column tables by @fallintoplace in [#988](https://github.com/apache/arrow-go/pull/988) -* docs(parquet): Add thrift generation contribution info by @C-Loftus in [#971](https://github.com/apache/arrow-go/pull/971) -* fix(array): reject mismatched known null counts during full validation by @fallintoplace in [#993](https://github.com/apache/arrow-go/pull/993) * fix(arrow/array): `Resize(0)` on non-empty builder causes memory leak by @serramatutu in [#995](https://github.com/apache/arrow-go/pull/995) -* feat(parquet/pqarrow): Read support for extension types / geo by @C-Loftus in [#969](https://github.com/apache/arrow-go/pull/969) -* fix(array): validate run-end encoded invariants by @fallintoplace in [#992](https://github.com/apache/arrow-go/pull/992) -* fix(array): validate binary_view and string_view payloads by @fallintoplace in [#994](https://github.com/apache/arrow-go/pull/994) -* fix(parquet): wrap DELTA_BINARY_PACKED deltas at the physical type width by @tom-eon in [#1027](https://github.com/apache/arrow-go/pull/1027) -* fix(arrow/compute): use is_null/is_not_null in expressions by @fallintoplace in [#997](https://github.com/apache/arrow-go/pull/997) -* fix(arrow/ipc): reject NaN minSpaceSavings values by @fallintoplace in [#998](https://github.com/apache/arrow-go/pull/998) -* fix(arrow/tensor): default nil dimension names to empty strings by @fallintoplace in [#999](https://github.com/apache/arrow-go/pull/999) -* fix(arrow/scalar): allow null extension scalars to keep null storage by @fallintoplace in [#1000](https://github.com/apache/arrow-go/pull/1000) -* fix(arrow/scalar): handle nil child values in run-end scalars by @fallintoplace in [#1001](https://github.com/apache/arrow-go/pull/1001) -* fix(arrow/array): ignore zero-count union appends by @fallintoplace in [#1026](https://github.com/apache/arrow-go/pull/1026) * fix(arrow/csv): preserve decimal precision by @fallintoplace in [#1022](https://github.com/apache/arrow-go/pull/1022) -* fix(parquet/metadata): compare bloom filter caps in bits by @fallintoplace in [#996](https://github.com/apache/arrow-go/pull/996) -* fix(parquet/pqarrow): reject timestamp coercion overflow by @fallintoplace in [#1008](https://github.com/apache/arrow-go/pull/1008) -* fix(parquet/file): reject records starting with nonzero repetition levels by @fallintoplace in [#1009](https://github.com/apache/arrow-go/pull/1009) -* fix(parquet/variant): reject truncated scalar values by @fallintoplace in [#1017](https://github.com/apache/arrow-go/pull/1017) -* fix(parquet/encoding): reject invalid delta byte prefixes by @fallintoplace in [#1014](https://github.com/apache/arrow-go/pull/1014) -* fix(parquet/metadata): preserve infinite floating-point statistics by @fallintoplace in [#1007](https://github.com/apache/arrow-go/pull/1007) -* fix(parquet/encoding): bound lazy dictionary growth by @fallintoplace in [#1011](https://github.com/apache/arrow-go/pull/1011) -* fix(parquet/encoding): respect declared RLE payload lengths by @fallintoplace in [#1015](https://github.com/apache/arrow-go/pull/1015) -* fix(parquet/encoding): validate delta binary packed headers by @fallintoplace in [#1012](https://github.com/apache/arrow-go/pull/1012) -* fix(parquet/encoding): reject invalid delta length payloads by @fallintoplace in [#1013](https://github.com/apache/arrow-go/pull/1013) -* fix(parquet/pqarrow): preserve INT96 timestamps across units by @fallintoplace in [#1006](https://github.com/apache/arrow-go/pull/1006) -* fix(parquet/pqarrow): Change interface name for arrow type on read for extension types by @C-Loftus in [#1028](https://github.com/apache/arrow-go/pull/1028) -* fix(arrow/csv): respect timestamp timezones by @fallintoplace in [#1025](https://github.com/apache/arrow-go/pull/1025) -* fix(parquet/compress): avoid closing caller streams for uncompressed wrappers by @fallintoplace in [#1003](https://github.com/apache/arrow-go/pull/1003) -* fix(parquet/compress): allocate uncompressed output for short destinations by @fallintoplace in [#1004](https://github.com/apache/arrow-go/pull/1004) -* fix(internal/utils): keep byteReader position after failed seeks by @fallintoplace in [#1005](https://github.com/apache/arrow-go/pull/1005) -* fix(parquet/file): count repeated skips in logical rows by @fallintoplace in [#1002](https://github.com/apache/arrow-go/pull/1002) -* fix(parquet/metadata): reject malformed fixed-width statistics by @fallintoplace in [#1010](https://github.com/apache/arrow-go/pull/1010) -* fix(arrow/array): skip empty table chunks by @fallintoplace in [#1023](https://github.com/apache/arrow-go/pull/1023) -* fix(arrow/scalar): reject negative array lengths by @fallintoplace in [#1024](https://github.com/apache/arrow-go/pull/1024) -* fix(arrow/array): validate fixed-size list child lengths by @fallintoplace in [#1019](https://github.com/apache/arrow-go/pull/1019) -* fix(parquet/pqarrow): calculate large decimal widths by @fallintoplace in [#1034](https://github.com/apache/arrow-go/pull/1034) -* fix(parquet/schema): reject invalid fixed lengths by @fallintoplace in [#1035](https://github.com/apache/arrow-go/pull/1035) -* fix(parquet/encoding): reject truncated plain boolean data by @fallintoplace in [#1031](https://github.com/apache/arrow-go/pull/1031) -* fix(parquet/metadata): return page index decode errors by @fallintoplace in [#1032](https://github.com/apache/arrow-go/pull/1032) -* fix(parquet/metadata): validate adaptive bloom options by @fallintoplace in [#1033](https://github.com/apache/arrow-go/pull/1033) -* fix(tmpl): report generated output write errors by @fallintoplace in [#1037](https://github.com/apache/arrow-go/pull/1037) -* fix(arrow/array): propagate embedded pointer nullability by @fallintoplace in [#1038](https://github.com/apache/arrow-go/pull/1038) -* fix(flight/flightsql/driver): convert Date32 values by @fallintoplace in [#1039](https://github.com/apache/arrow-go/pull/1039) -* fix(arrow/scalar): support same-width time casts by @fallintoplace in [#1040](https://github.com/apache/arrow-go/pull/1040) -* fix(arrow/scalar): use duration type for units by @fallintoplace in [#1041](https://github.com/apache/arrow-go/pull/1041) -* fix(arrow/array): validate union child upper bound by @fallintoplace in [#1043](https://github.com/apache/arrow-go/pull/1043) * refactor(arrow/avro): migrate from hamba/avro to twmb/avro by @prochac in [#830](https://github.com/apache/arrow-go/pull/830) -* fix(arrow/array): validate struct child lengths by @fallintoplace in [#1021](https://github.com/apache/arrow-go/pull/1021) -* fix(arrow/cdata): validate imported schema topology by @fallintoplace in [#1046](https://github.com/apache/arrow-go/pull/1046) -* fix(parquet/metadata): validate row-group ordinals in PageIndexReader.WillNeed by @fallintoplace in [#1057](https://github.com/apache/arrow-go/pull/1057) -* fix(arrow/flight): limit descriptors to next payload by @fallintoplace in [#1042](https://github.com/apache/arrow-go/pull/1042) -* fix(parquet): propagate nonce generation failures by @fallintoplace in [#1045](https://github.com/apache/arrow-go/pull/1045) -* fix(parquet/metadata): reject negative row-group indices in Subset by @fallintoplace in [#1055](https://github.com/apache/arrow-go/pull/1055) -* fix(parquet/metadata): use default writer properties when props is nil by @fallintoplace in [#1058](https://github.com/apache/arrow-go/pull/1058) -* fix(parquet/metadata): return schema initialization errors by @fallintoplace in [#1056](https://github.com/apache/arrow-go/pull/1056) -* fix(parquet/pqarrow): propagate level builder errors by @fallintoplace in [#1049](https://github.com/apache/arrow-go/pull/1049) -* fix(arrow/flight/session): clone option values by @fallintoplace in [#1044](https://github.com/apache/arrow-go/pull/1044) -* fix(parquet/variant): validate metadata offset tables by @fallintoplace in [#1063](https://github.com/apache/arrow-go/pull/1063) -* fix(arrow/csv): release Reader record builders by @fallintoplace in [#1059](https://github.com/apache/arrow-go/pull/1059) -* fix(parquet/pqarrow): release partially constructed field readers by @fallintoplace in [#1061](https://github.com/apache/arrow-go/pull/1061) -* fix(arrow/array): make dictionary suffix null indexes relative by @fallintoplace in [#1064](https://github.com/apache/arrow-go/pull/1064) -* fix(parquet/pqarrow): prefer stored schema extension type on read by @twuebi in [#1051](https://github.com/apache/arrow-go/pull/1051) -* fix(arrow/flight/flightsql): avoid double-retaining parameter readers by @fallintoplace in [#1066](https://github.com/apache/arrow-go/pull/1066) -* fix(flightsql): isolate example SQLite databases by @fallintoplace in [#1067](https://github.com/apache/arrow-go/pull/1067) -* fix(arrow): return an owned slice from Schema.FieldsByName by @fallintoplace in [#1069](https://github.com/apache/arrow-go/pull/1069) -* fix(arrow/ipc): return errors for malformed file record batches by @fallintoplace in [#1062](https://github.com/apache/arrow-go/pull/1062) -* fix(flight): return Unauthenticated for invalid unary auth tokens by @fallintoplace in [#1070](https://github.com/apache/arrow-go/pull/1070) -* fix(arrow/array): always release JSON reader builder by @fallintoplace in [#1065](https://github.com/apache/arrow-go/pull/1065) -* fix(flight): accept case-insensitive authorization schemes by @fallintoplace in [#1071](https://github.com/apache/arrow-go/pull/1071) -* fix(parquet): reject short page writes by @fallintoplace in [#1050](https://github.com/apache/arrow-go/pull/1050) -* fix(parquet/file): close file on OpenParquetFile errors by @fallintoplace in [#1053](https://github.com/apache/arrow-go/pull/1053) -* fix(parquet): reject truncated footer signatures by @fallintoplace in [#1074](https://github.com/apache/arrow-go/pull/1074) -* fix(arrow/ipc): reject non-dictionary initial messages by @fallintoplace in [#1054](https://github.com/apache/arrow-go/pull/1054) * feat(flightsql): allow configuring the driver memory allocator by @fallintoplace in [#1076](https://github.com/apache/arrow-go/pull/1076) -* fix(arrow): avoid retains on invalid chunk construction by @fallintoplace in [#1078](https://github.com/apache/arrow-go/pull/1078) -* fix(parquet): row-align V1 pages with offset indexes by @fallintoplace in [#1075](https://github.com/apache/arrow-go/pull/1075) -* fix(arrow/array): clean up partial table construction by @fallintoplace in [#1077](https://github.com/apache/arrow-go/pull/1077) -* fix(parquet_reader): propagate batch read errors by @fallintoplace in [#1081](https://github.com/apache/arrow-go/pull/1081) -* fix(arrow): isolate schemas created by AddField by @fallintoplace in [#1079](https://github.com/apache/arrow-go/pull/1079) -* fix(arrow/util): avoid protobuf map schema goroutines by @fallintoplace in [#1082](https://github.com/apache/arrow-go/pull/1082) -* fix(array): format date arrays in String by @fallintoplace in [#1068](https://github.com/apache/arrow-go/pull/1068) -* fix(parquet): report initialized row-group columns correctly by @fallintoplace in [#1072](https://github.com/apache/arrow-go/pull/1072) -* fix(parquet): serialize column chunk file offsets by @fallintoplace in [#1073](https://github.com/apache/arrow-go/pull/1073) -* fix(arrow/util): release protobuf record builders by @fallintoplace in [#1084](https://github.com/apache/arrow-go/pull/1084) -* fix(arrow/util): release protobuf enum dictionaries by @fallintoplace in [#1083](https://github.com/apache/arrow-go/pull/1083) -* fix(arrow/avro): stop reader workers before reuse by @fallintoplace in [#1048](https://github.com/apache/arrow-go/pull/1048) -* fix(parquet): restore Arrow duration units from schema metadata by @fallintoplace in [#1095](https://github.com/apache/arrow-go/pull/1095) -* fix(compute): round negative HalfToOdd ties correctly by @fallintoplace in [#1094](https://github.com/apache/arrow-go/pull/1094) -* perf(arrow/bitutil): add SetBitSwap/ClearBitSwap, use in builder.SetNull by @tom-eon in [#1090](https://github.com/apache/arrow-go/pull/1090) -* fix(array): compare sparse union slices at relative offsets by @fallintoplace in [#1096](https://github.com/apache/arrow-go/pull/1096) -* fix(arrow/ipc): make writer failures terminal by @fallintoplace in [#1047](https://github.com/apache/arrow-go/pull/1047) -* feat(parquet): add WithDictionaryCostFallback writer property by @tom-eon in [#1036](https://github.com/apache/arrow-go/pull/1036) -* fix(parquet/pqarrow): release extension arrays after chunking by @fallintoplace in [#1060](https://github.com/apache/arrow-go/pull/1060) -* fix(parquet/encoding): read ahead in the streaming value buffer by @joechenrh in [#937](https://github.com/apache/arrow-go/pull/937) -* fix(parquet/variant): preserve metadata keys when cloning by @fallintoplace in [#1080](https://github.com/apache/arrow-go/pull/1080) -* test(arrow/util): cover zero-field protobuf records by @fallintoplace in [#1085](https://github.com/apache/arrow-go/pull/1085) -* fix(arrow/scalar): preserve metadata in FromScalar by @fallintoplace in [#1100](https://github.com/apache/arrow-go/pull/1100) -* fix(arrow/scalar): preserve binary cast target types by @fallintoplace in [#1101](https://github.com/apache/arrow-go/pull/1101) -* fix(arrow/scalar): support boolean identity casts by @fallintoplace in [#1102](https://github.com/apache/arrow-go/pull/1102) -* fix(arrow/util): handle invalid protobuf Any values by @fallintoplace in [#1086](https://github.com/apache/arrow-go/pull/1086) -* fix(arrow/array): avoid copying initialized extension refcounts by @fallintoplace in [#1089](https://github.com/apache/arrow-go/pull/1089) -* fix(parquet): preserve dictionary index types by @fallintoplace in [#1098](https://github.com/apache/arrow-go/pull/1098) -* fix(parquet_reader): propagate statistics decoding errors by @fallintoplace in [#1087](https://github.com/apache/arrow-go/pull/1087) -* fix(arrow/array): avoid panics cleaning partial tables by @fallintoplace in [#1103](https://github.com/apache/arrow-go/pull/1103) -* fix(arrow/ipc): release reader state on schema errors by @fallintoplace in [#1112](https://github.com/apache/arrow-go/pull/1112) -* fix(arrow/scalar): format extension values from storage scalars by @fallintoplace in [#1111](https://github.com/apache/arrow-go/pull/1111) -* fix(arrow/array): isolate table columns from caller slices by @fallintoplace in [#1105](https://github.com/apache/arrow-go/pull/1105) -* fix(arrow/scalar): preserve exact decimal string casts by @fallintoplace in [#1107](https://github.com/apache/arrow-go/pull/1107) -* fix(arrow/tensor): validate typed constructor data by @fallintoplace in [#1104](https://github.com/apache/arrow-go/pull/1104) -* fix(arrow/scalar): format interval scalars in String by @fallintoplace in [#1115](https://github.com/apache/arrow-go/pull/1115) -* fix(arrow/array): preserve rows for empty-schema JSON input by @fallintoplace in [#1117](https://github.com/apache/arrow-go/pull/1117) -* fix(arrow/util): return protobuf record conversion errors by @fallintoplace in [#1088](https://github.com/apache/arrow-go/pull/1088) -* fix(arrow/scalar): validate FromScalar destinations by @fallintoplace in [#1106](https://github.com/apache/arrow-go/pull/1106) -* fix(arrow/scalar): use wide offsets for large binary scalars by @fallintoplace in [#1108](https://github.com/apache/arrow-go/pull/1108) -* fix(arrow/scalar): apply timestamp zones when casting to dates by @fallintoplace in [#1110](https://github.com/apache/arrow-go/pull/1110) -* fix(arrow/ipc): validate file block framing and lengths by @fallintoplace in [#1097](https://github.com/apache/arrow-go/pull/1097) -* fix(arrow/scalar): release partial children after failed struct extraction by @fallintoplace in [#1118](https://github.com/apache/arrow-go/pull/1118) -* fix(arrow/ipc): return errors for invalid record indexes by @fallintoplace in [#1120](https://github.com/apache/arrow-go/pull/1120) -* fix(parquet): read null fixed-size-list parents by @fallintoplace in [#1099](https://github.com/apache/arrow-go/pull/1099) -* fix(arrow/scalar): release partial children after failed struct conversion by @fallintoplace in [#1114](https://github.com/apache/arrow-go/pull/1114) -* fix(parquet/metadata): return error for missing column metadata by @fallintoplace in [#1137](https://github.com/apache/arrow-go/pull/1137) -* fix(arrow/array): widen dictionary index bounds checks by @fallintoplace in [#1129](https://github.com/apache/arrow-go/pull/1129) -* fix(compute/exprs): avoid empty field reference panic by @fallintoplace in [#1130](https://github.com/apache/arrow-go/pull/1130) -* fix(compute): handle odd temporal multiples by @fallintoplace in [#1119](https://github.com/apache/arrow-go/pull/1119) -* fix(arrow/csv): preserve first row during headerless inference by @fallintoplace in [#1145](https://github.com/apache/arrow-go/pull/1145) -* fix(arrow/array): bound table reader batches by @fallintoplace in [#1146](https://github.com/apache/arrow-go/pull/1146) -* fix(arrow/extensions): compare JSON storage types semantically by @fallintoplace in [#1150](https://github.com/apache/arrow-go/pull/1150) -* fix(arrow/ipc): preserve minSpaceSavings in stream writer by @fallintoplace in [#1143](https://github.com/apache/arrow-go/pull/1143) -* fix(parquet/compress): honor LZ4 raw destination contract by @fallintoplace in [#1147](https://github.com/apache/arrow-go/pull/1147) -* fix(arrow): reject duplicate union type codes by @fallintoplace in [#1131](https://github.com/apache/arrow-go/pull/1131) -* fix(parquet/metadata): handle invalid row-group column indexes by @fallintoplace in [#1125](https://github.com/apache/arrow-go/pull/1125) -* fix(compute/exec): handle missing validity buffers by @fallintoplace in [#1136](https://github.com/apache/arrow-go/pull/1136) -* fix(compute/exec): report ListView buffer counts by @fallintoplace in [#1133](https://github.com/apache/arrow-go/pull/1133) -* fix(arrow): include KeysSorted in map equality by @fallintoplace in [#1134](https://github.com/apache/arrow-go/pull/1134) -* fix(compute): reject non-array-like filters by @fallintoplace in [#1135](https://github.com/apache/arrow-go/pull/1135) -* fix(memory): handle zero-length checked reallocations by @fallintoplace in [#1132](https://github.com/apache/arrow-go/pull/1132) -* fix(parquet/pqarrow): validate reader indexes by @fallintoplace in [#1124](https://github.com/apache/arrow-go/pull/1124) -* fix(arrow/csv): validate custom converter row counts by @fallintoplace in [#1148](https://github.com/apache/arrow-go/pull/1148) -* fix(parquet/encoding): honor BufferWriter offsets by @fallintoplace in [#1144](https://github.com/apache/arrow-go/pull/1144) -* fix(flightsql): stop row streaming after close by @fallintoplace in [#1151](https://github.com/apache/arrow-go/pull/1151) -* fix(flightsql): protect configured authorization metadata by @fallintoplace in [#1153](https://github.com/apache/arrow-go/pull/1153) -* perf(parquet/pqarrow): avoid reflection in integer transfers by @fallintoplace in [#1163](https://github.com/apache/arrow-go/pull/1163) -* fix(flightsql): reject unsupported transaction options by @fallintoplace in [#1152](https://github.com/apache/arrow-go/pull/1152) -* fix(arrow/scalar): handle zero-length scalar arrays by @fallintoplace in [#1116](https://github.com/apache/arrow-go/pull/1116) -* perf(arrow/array): pack validity values in batches by @fallintoplace in [#1185](https://github.com/apache/arrow-go/pull/1185) +* feat(arrow/flight/sql): Add is_update field to ActionCreatePreparedStatementResult by @ennuite in [#732](https://github.com/apache/arrow-go/pull/732) * feat(arrow/array): roll back JSON builder state after failed rows by @fallintoplace in [#1113](https://github.com/apache/arrow-go/pull/1113) +* feat(arrow/array): add ValueAsAny for native Go values by @dgvj-work in [#1170](https://github.com/apache/arrow-go/pull/1170) +* feat(extensions): add VariantGet for path extraction from variant arrays by @nssalian in [#1206](https://github.com/apache/arrow-go/pull/1206) +* fix(arrow/extensions): use canonical Variant extension name by @dgvj-work in [#1241](https://github.com/apache/arrow-go/pull/1241) +* fix(arrow/extensions): reject Null Variant typed_value by @dgvj-work in [#1243](https://github.com/apache/arrow-go/pull/1243) +* fix(arrow/ipc): validate file block framing and lengths by @fallintoplace in [#1097](https://github.com/apache/arrow-go/pull/1097) * fix(arrow/ipc): avoid deadlock on compression errors by @fallintoplace in [#1142](https://github.com/apache/arrow-go/pull/1142) +* fix(arrow/scalar): preserve exact decimal string casts by @fallintoplace in [#1107](https://github.com/apache/arrow-go/pull/1107) +* fix(arrow/scalar): align timestamp timezone parsing by @fallintoplace in [#1109](https://github.com/apache/arrow-go/pull/1109) +* fix(arrow): validate precision in NewDecimalType by @fallintoplace in [#1162](https://github.com/apache/arrow-go/pull/1162) +* fix(arrow): reject duplicate union type codes by @fallintoplace in [#1131](https://github.com/apache/arrow-go/pull/1131) +* fix(arrow/csv): preserve first row during headerless inference by @fallintoplace in [#1145](https://github.com/apache/arrow-go/pull/1145) * fix(arrow/csv): append null for later parse failures by @fallintoplace in [#1149](https://github.com/apache/arrow-go/pull/1149) -* feat(arrow/array): add ValueAsAny for native Go values by @dgvj-work in [#1170](https://github.com/apache/arrow-go/pull/1170) -* perf(arrow/array): bulk append fixed-size list nulls and empty values by @fallintoplace in [#1183](https://github.com/apache/arrow-go/pull/1183) -* perf(parquet): encode spaced booleans without compaction by @fallintoplace in [#1175](https://github.com/apache/arrow-go/pull/1175) -* fix(parquet/pqarrow): reject decimal overflow in integer columns by @fallintoplace in [#1122](https://github.com/apache/arrow-go/pull/1122) -* fix(arrow/array/arreflect): reject unrepresentable timestamps by @fallintoplace in [#1123](https://github.com/apache/arrow-go/pull/1123) +* fix(arrow/array): clamp run ends when concatenating a sliced RunEndEncoded array by @winklemad in [#1219](https://github.com/apache/arrow-go/pull/1219) +* fix(arrow/cdata): validate imported schema topology by @fallintoplace in [#1046](https://github.com/apache/arrow-go/pull/1046) + +#### Compute + +* feat(compute): add list_element by @fallintoplace in [#1138](https://github.com/apache/arrow-go/pull/1138) * feat(compute): add cumulative_sum and cumulative_sum_checked by @fallintoplace in [#1139](https://github.com/apache/arrow-go/pull/1139) -* fix(arrow/ipc): return compression errors from record encoder by @fallintoplace in [#1141](https://github.com/apache/arrow-go/pull/1141) -* perf(arrow/array): bulk append fixed-width empty values by @fallintoplace in [#1180](https://github.com/apache/arrow-go/pull/1180) -* perf(arrow/csv): reserve record builders for chunked reads by @fallintoplace in [#1198](https://github.com/apache/arrow-go/pull/1198) -* perf(arrow/array): compare list values by valid runs by @fallintoplace in [#1184](https://github.com/apache/arrow-go/pull/1184) -* perf(parquet): optimize numeric dictionary insertion by @fallintoplace in [#1178](https://github.com/apache/arrow-go/pull/1178) -* feat(arrow/flight/sql): Add is_update field to ActionCreatePreparedStatementResult by @ennuite in [#732](https://github.com/apache/arrow-go/pull/732) * feat(compute): add dictionary_encode by @fallintoplace in [#1140](https://github.com/apache/arrow-go/pull/1140) -* perf(parquet): batch plain BYTE_ARRAY encoding by @fallintoplace in [#1193](https://github.com/apache/arrow-go/pull/1193) -* fix(arrow/csv): append null for invalid fixed-size binary values by @fallintoplace in [#1121](https://github.com/apache/arrow-go/pull/1121) -* fix(parquet/pqarrow): reject out-of-range INT96 timestamps by @fallintoplace in [#1128](https://github.com/apache/arrow-go/pull/1128) -* perf(parquet): batch delta binary packed decoding by @fallintoplace in [#1169](https://github.com/apache/arrow-go/pull/1169) -* perf(arrow/array): bulk append list nulls and empty values by @fallintoplace in [#1187](https://github.com/apache/arrow-go/pull/1187) -* perf(parquet/pqarrow): decode booleans directly to bitmaps by @fallintoplace in [#1182](https://github.com/apache/arrow-go/pull/1182) -* perf(parquet/pqarrow): build nullable fixed-size list children directly by @fallintoplace in [#1195](https://github.com/apache/arrow-go/pull/1195) -* perf(internal/hashing): build binary memo offsets in one pass by @fallintoplace in [#1196](https://github.com/apache/arrow-go/pull/1196) -* perf(arrow/compute): take chunked binary values without concatenation by @fallintoplace in [#1199](https://github.com/apache/arrow-go/pull/1199) -* perf(arrow/array): reuse TableReader scratch slices by @fallintoplace in [#1200](https://github.com/apache/arrow-go/pull/1200) -* fix(parquet/pqarrow): reject decimal overflow when writing integers by @fallintoplace in [#1161](https://github.com/apache/arrow-go/pull/1161) -* fix(arrow/array): preserve run-end encoded empty values by @fallintoplace in [#1156](https://github.com/apache/arrow-go/pull/1156) +* fix(compute): round negative HalfToOdd ties correctly by @fallintoplace in [#1094](https://github.com/apache/arrow-go/pull/1094) +* fix(compute): handle odd temporal multiples by @fallintoplace in [#1119](https://github.com/apache/arrow-go/pull/1119) * fix(compute): preserve caller context during execution by @fallintoplace in [#1154](https://github.com/apache/arrow-go/pull/1154) -* fix(compute): propagate nested expression serialization errors by @fallintoplace in [#1160](https://github.com/apache/arrow-go/pull/1160) -* fix(compute): report FuncMeta for meta functions by @fallintoplace in [#1155](https://github.com/apache/arrow-go/pull/1155) -* perf(arrow/array): optimize bulk appends for binary builders by @fallintoplace in [#1188](https://github.com/apache/arrow-go/pull/1188) -* perf(arrow/array): avoid temporary buffers when concatenating by @fallintoplace in [#1194](https://github.com/apache/arrow-go/pull/1194) -* perf(compute): batch contiguous filter take indices by @fallintoplace in [#1197](https://github.com/apache/arrow-go/pull/1197) -* perf(arrow/ipc): slice already-zero-based offset buffers by @fallintoplace in [#1191](https://github.com/apache/arrow-go/pull/1191) -* perf(parquet): encode RLE levels in batches by @fallintoplace in [#1177](https://github.com/apache/arrow-go/pull/1177) -* fix(arrow): include REE value nullability in type equality by @fallintoplace in [#1158](https://github.com/apache/arrow-go/pull/1158) +* fix(compute): discard results after cancellation by @zeroshade in [#1278](https://github.com/apache/arrow-go/pull/1278) +* fix(compute/exprs): avoid empty field reference panic by @fallintoplace in [#1130](https://github.com/apache/arrow-go/pull/1130) + +#### Parquet + +* feat(parquet/schema): Wire up logical type functionality for writing geo types by @C-Loftus in [#960](https://github.com/apache/arrow-go/pull/960) +* feat(parquet/pqarrow): Read support for extension types / geo by @C-Loftus in [#969](https://github.com/apache/arrow-go/pull/969) +* feat(parquet): add WithDictionaryCostFallback writer property by @tom-eon in [#1036](https://github.com/apache/arrow-go/pull/1036) +* fix(parquet): wrap DELTA_BINARY_PACKED deltas at the physical type width by @tom-eon in [#1027](https://github.com/apache/arrow-go/pull/1027) +* fix(parquet/encoding): read ahead in the streaming value buffer by @joechenrh in [#937](https://github.com/apache/arrow-go/pull/937) +* fix(parquet/pqarrow): prefer stored schema extension type on read by @twuebi in [#1051](https://github.com/apache/arrow-go/pull/1051) +* fix(parquet/pqarrow): require Variant value field when writing by @dgvj-work in [#1242](https://github.com/apache/arrow-go/pull/1242) +* fix(parquet): guard byte-stream-split encoder release against never-flushed buffer by @tom-eon in [#1273](https://github.com/apache/arrow-go/pull/1273) +* fix(parquet): stop BYTE_STREAM_SPLIT FLBA spaced decode aliasing output by @zeroshade in [#1256](https://github.com/apache/arrow-go/pull/1256) +* fix(parquet): read null fixed-size-list parents by @fallintoplace in [#1099](https://github.com/apache/arrow-go/pull/1099) +* fix(parquet/pqarrow): reject decimal overflow in integer columns by @fallintoplace in [#1122](https://github.com/apache/arrow-go/pull/1122) +* fix(parquet/pqarrow): reject decimal overflow when writing integers by @fallintoplace in [#1161](https://github.com/apache/arrow-go/pull/1161) * fix(parquet): build bloom filters from dictionary entries by @fallintoplace in [#1164](https://github.com/apache/arrow-go/pull/1164) -* fix(arrow): validate precision in NewDecimalType by @fallintoplace in [#1162](https://github.com/apache/arrow-go/pull/1162) -* perf(arrow/csv): avoid materializing rows before writing by @fallintoplace in [#1190](https://github.com/apache/arrow-go/pull/1190) +* fix(parquet): preserve dictionary index types by @fallintoplace in [#1098](https://github.com/apache/arrow-go/pull/1098) +* fix(parquet): restore Arrow duration units from schema metadata by @fallintoplace in [#1095](https://github.com/apache/arrow-go/pull/1095) +* fix(parquet/compress): honor LZ4 raw destination contract by @fallintoplace in [#1147](https://github.com/apache/arrow-go/pull/1147) +* fix(parquet): row-align V1 pages with offset indexes by @fallintoplace in [#1075](https://github.com/apache/arrow-go/pull/1075) +* fix(parquet/variant): validate metadata offset tables by @fallintoplace in [#1063](https://github.com/apache/arrow-go/pull/1063) + +#### Performance + +* perf(arrow/bitutil): add SetBitSwap/ClearBitSwap, use in builder.SetNull by @tom-eon in [#1090](https://github.com/apache/arrow-go/pull/1090) +* perf(bitutil): add ARM64 NEON bitmap operations by @fallintoplace in [#1264](https://github.com/apache/arrow-go/pull/1264) +* perf(arrow/array): pack validity values in batches by @fallintoplace in [#1185](https://github.com/apache/arrow-go/pull/1185) +* perf(arrow/array): SIMD-pack boolean slices into bitmaps by @fallintoplace in [#1283](https://github.com/apache/arrow-go/pull/1283) * perf(arrow/array): batch run-end encoded appends by @fallintoplace in [#1214](https://github.com/apache/arrow-go/pull/1214) -* fix(arrow/array): validate union JSON type codes by @fallintoplace in [#1157](https://github.com/apache/arrow-go/pull/1157) -* fix(parquet/variant): validate compound value bounds by @fallintoplace in [#1126](https://github.com/apache/arrow-go/pull/1126) -* perf(arrow/array): validate dictionary indices by valid runs by @fallintoplace in [#1181](https://github.com/apache/arrow-go/pull/1181) -* perf(parquet): bound Bloom filter hash batches by @fallintoplace in [#1218](https://github.com/apache/arrow-go/pull/1218) -* perf(arrow/array): optimize bulk appends for struct builders by @fallintoplace in [#1189](https://github.com/apache/arrow-go/pull/1189) -* fix(arrow): return owned field index slices by @fallintoplace in [#1159](https://github.com/apache/arrow-go/pull/1159) -* perf(arrow/array): avoid temporary slices in Diff comparisons by @fallintoplace in [#1239](https://github.com/apache/arrow-go/pull/1239) -* perf(parquet/compress): reuse gzip writers by @fallintoplace in [#1238](https://github.com/apache/arrow-go/pull/1238) -* perf(parquet): reuse fixed-length zero scratch by @fallintoplace in [#1236](https://github.com/apache/arrow-go/pull/1236) -* perf(compute): share logical row maps across aligned sorts by @fallintoplace in [#1234](https://github.com/apache/arrow-go/pull/1234) +* perf(arrow/array): avoid temporary buffers when concatenating by @fallintoplace in [#1194](https://github.com/apache/arrow-go/pull/1194) +* perf(arrow/array): concatenate bitmaps directly by @fallintoplace in [#1224](https://github.com/apache/arrow-go/pull/1224) +* perf(compute): add ARM64 NEON arithmetic kernels by @fallintoplace in [#1265](https://github.com/apache/arrow-go/pull/1265) +* perf(compute): add ARM64 NEON comparison kernels by @fallintoplace in [#1266](https://github.com/apache/arrow-go/pull/1266) +* perf(compute): add ARM64 NEON comparisons for 8- and 16-bit integers by @fallintoplace in [#1288](https://github.com/apache/arrow-go/pull/1288) +* perf(compute): add ARM64 NEON compression for null-free 32-bit fixed-width filters by @fallintoplace in [#1286](https://github.com/apache/arrow-go/pull/1286) +* perf(compute): vectorize mixed int32 filters with AVX2 by @fallintoplace in [#1284](https://github.com/apache/arrow-go/pull/1284) +* perf(compute): SIMD-compact fragmented filters into take indices by @fallintoplace in [#1287](https://github.com/apache/arrow-go/pull/1287) * perf(compute): fast-path single-span execution by @fallintoplace in [#1233](https://github.com/apache/arrow-go/pull/1233) -* perf(parquet): reuse boolean spaced scratch by @fallintoplace in [#1231](https://github.com/apache/arrow-go/pull/1231) -* perf(parquet): reuse delta PutSpaced scratch by @fallintoplace in [#1230](https://github.com/apache/arrow-go/pull/1230) -* perf(arrow/array): remove map validity slice by @fallintoplace in [#1228](https://github.com/apache/arrow-go/pull/1228) -* perf(arrow/array): copy same-dictionary indices directly by @fallintoplace in [#1225](https://github.com/apache/arrow-go/pull/1225) -* perf(parquet): reuse dictionary decoder scratch by @fallintoplace in [#1222](https://github.com/apache/arrow-go/pull/1222) -* perf(arrow/array): compare fixed-width values in bulk by @fallintoplace in [#1171](https://github.com/apache/arrow-go/pull/1171) -* perf(arrow/array): compare boolean values by valid runs by @fallintoplace in [#1186](https://github.com/apache/arrow-go/pull/1186) -* perf(arrow/array): avoid temporary buffers for binary concatenation by @fallintoplace in [#1215](https://github.com/apache/arrow-go/pull/1215) -* perf(arrow/array): batch union null and empty appends by @fallintoplace in [#1216](https://github.com/apache/arrow-go/pull/1216) -* perf(parquet): decode levels directly into int16 batches by @fallintoplace in [#1176](https://github.com/apache/arrow-go/pull/1176) -* perf(parquet): batch DELTA_BYTE_ARRAY encoding by @fallintoplace in [#1179](https://github.com/apache/arrow-go/pull/1179) -* perf(parquet/variant): skip metadata construction in Encode by @fallintoplace in [#1217](https://github.com/apache/arrow-go/pull/1217) -* perf(parquet): decode BSS FLBA into contiguous storage by @fallintoplace in [#1172](https://github.com/apache/arrow-go/pull/1172) -* perf(parquet): stop spaced expansion once aligned by @fallintoplace in [#1220](https://github.com/apache/arrow-go/pull/1220) -* perf(arrow/ipc): reuse record encoder scratch by @fallintoplace in [#1221](https://github.com/apache/arrow-go/pull/1221) -* perf(arrow/array): batch dictionary empty appends by @fallintoplace in [#1223](https://github.com/apache/arrow-go/pull/1223) -* perf(compute): reserve binary view cast payload by @fallintoplace in [#1227](https://github.com/apache/arrow-go/pull/1227) -* perf(compute): use stack scratch for null propagation by @fallintoplace in [#1229](https://github.com/apache/arrow-go/pull/1229) -* perf(arrow/array): bulk append BinaryView nulls by @fallintoplace in [#1232](https://github.com/apache/arrow-go/pull/1232) -* perf(arrow/array): use unsafe writes for binary bulk appends by @fallintoplace in [#1240](https://github.com/apache/arrow-go/pull/1240) -* perf(arrow/array): compare union values in runs by @fallintoplace in [#1244](https://github.com/apache/arrow-go/pull/1244) -* perf(arrow/array): batch-pack boolean values by @fallintoplace in [#1237](https://github.com/apache/arrow-go/pull/1237) -* fix(parquet/pqarrow): require Variant value field when writing by @dgvj-work in [#1242](https://github.com/apache/arrow-go/pull/1242) * perf(arrow/compute): avoid goroutines for serial record filtering by @fallintoplace in [#1245](https://github.com/apache/arrow-go/pull/1245) -* perf(parquet): reuse dictionary RLE decoder across pages by @fallintoplace in [#1247](https://github.com/apache/arrow-go/pull/1247) -* perf(parquet/pqarrow): write record ranges directly by @fallintoplace in [#1248](https://github.com/apache/arrow-go/pull/1248) -* perf(parquet/pqarrow): avoid goroutines for serial struct reads by @fallintoplace in [#1250](https://github.com/apache/arrow-go/pull/1250) -* perf(parquet): decode RLE booleans into bitmaps by @fallintoplace in [#1246](https://github.com/apache/arrow-go/pull/1246) -* feat(compute): add list_element by @fallintoplace in [#1138](https://github.com/apache/arrow-go/pull/1138) -* perf(parquet): reuse DataPageV2 assembly buffer for eager writes by @fallintoplace in [#1192](https://github.com/apache/arrow-go/pull/1192) -* perf(arrow/array): append fixed-width nulls in bulk by @fallintoplace in [#1173](https://github.com/apache/arrow-go/pull/1173) -* perf(arrow/array): concatenate bitmaps directly by @fallintoplace in [#1224](https://github.com/apache/arrow-go/pull/1224) * perf(arrow/compute): avoid goroutines for serial take by @fallintoplace in [#1226](https://github.com/apache/arrow-go/pull/1226) -* perf(compute): use typed memo insertion for numeric hashes by @fallintoplace in [#1251](https://github.com/apache/arrow-go/pull/1251) -* perf(parquet): decode DELTA_BYTE_ARRAY into contiguous storage by @fallintoplace in [#1252](https://github.com/apache/arrow-go/pull/1252) -* perf(parquet/pqarrow): use synchronous path for serial reads by @fallintoplace in [#1235](https://github.com/apache/arrow-go/pull/1235) -* perf(arrow/array): reuse full chunks in NewChunkedSlice by @fallintoplace in [#1257](https://github.com/apache/arrow-go/pull/1257) -* perf(parquet): reuse boolean Bloom filter hashes by @fallintoplace in [#1258](https://github.com/apache/arrow-go/pull/1258) -* perf(arrow/ipc): release dictionary payloads eagerly by @fallintoplace in [#1259](https://github.com/apache/arrow-go/pull/1259) * perf(parquet): SIMD BYTE_STREAM_SPLIT encoding by @fallintoplace in [#1260](https://github.com/apache/arrow-go/pull/1260) -* perf(parquet): reuse DELTA byte-array scratch by @fallintoplace in [#1261](https://github.com/apache/arrow-go/pull/1261) -* perf(parquet/pqarrow): write FixedSizeBinary values directly by @fallintoplace in [#1263](https://github.com/apache/arrow-go/pull/1263) -* feat(extensions): add VariantGet for path extraction from variant arrays by @nssalian in [#1206](https://github.com/apache/arrow-go/pull/1206) -* perf(parquet): use typed memo insertion for byte-array dictionaries by @derekperkins in [#1272](https://github.com/apache/arrow-go/pull/1272) -* fix(arrow/array): clamp run ends when concatenating a sliced RunEndEncoded array by @winklemad in [#1219](https://github.com/apache/arrow-go/pull/1219) -* fix(parquet): guard byte-stream-split encoder release against never-flushed buffer by @tom-eon in [#1273](https://github.com/apache/arrow-go/pull/1273) -* perf(bitutil): add ARM64 NEON bitmap operations by @fallintoplace in [#1264](https://github.com/apache/arrow-go/pull/1264) -* perf(arrow/array): compare binary values by valid runs by @fallintoplace in [#1174](https://github.com/apache/arrow-go/pull/1174) -* perf(compute): add ARM64 NEON arithmetic kernels by @fallintoplace in [#1265](https://github.com/apache/arrow-go/pull/1265) -* perf(parquet/compress): reuse Brotli writers across encodes by @fallintoplace in [#1254](https://github.com/apache/arrow-go/pull/1254) -* perf(arrow/array): skip full-range SliceEqual slices by @fallintoplace in [#1249](https://github.com/apache/arrow-go/pull/1249) * perf(parquet): vectorize fixed-width dictionary materialization by @fallintoplace in [#1269](https://github.com/apache/arrow-go/pull/1269) -* perf(compute): add ARM64 NEON comparison kernels by @fallintoplace in [#1266](https://github.com/apache/arrow-go/pull/1266) +* perf(parquet): optimize numeric dictionary insertion by @fallintoplace in [#1178](https://github.com/apache/arrow-go/pull/1178) +* perf(parquet): use typed memo insertion for byte-array dictionaries by @derekperkins in [#1272](https://github.com/apache/arrow-go/pull/1272) +* perf(parquet): batch plain BYTE_ARRAY encoding by @fallintoplace in [#1193](https://github.com/apache/arrow-go/pull/1193) +* perf(parquet): encode RLE levels in batches by @fallintoplace in [#1177](https://github.com/apache/arrow-go/pull/1177) +* perf(parquet): batch delta binary packed decoding by @fallintoplace in [#1169](https://github.com/apache/arrow-go/pull/1169) +* perf(parquet): decode RLE booleans into bitmaps by @fallintoplace in [#1246](https://github.com/apache/arrow-go/pull/1246) +* perf(parquet/pqarrow): decode booleans directly to bitmaps by @fallintoplace in [#1182](https://github.com/apache/arrow-go/pull/1182) * perf(parquet/pqarrow): write binary values from Arrow offsets by @fallintoplace in [#1262](https://github.com/apache/arrow-go/pull/1262) -* fix(arrow/extensions): reject Null Variant typed_value by @dgvj-work in [#1243](https://github.com/apache/arrow-go/pull/1243) -* fix(arrow/extensions): use canonical Variant extension name by @dgvj-work in [#1241](https://github.com/apache/arrow-go/pull/1241) -* chore: bump version number by @zeroshade in [#1276](https://github.com/apache/arrow-go/pull/1276) -* test(parquet): reduce macOS race test memory pressure by @zeroshade in [#1275](https://github.com/apache/arrow-go/pull/1275) -* fix(parquet): explicitly release bloom builders by @zeroshade in [#1280](https://github.com/apache/arrow-go/pull/1280) -* ci: retry and cache dependency downloads by @zeroshade in [#1279](https://github.com/apache/arrow-go/pull/1279) -* fix(compute): discard results after cancellation by @zeroshade in [#1278](https://github.com/apache/arrow-go/pull/1278) -* ci: use bundled GTest for conda integration by @zeroshade in [#1277](https://github.com/apache/arrow-go/pull/1277) -* fix(parquet): stop BYTE_STREAM_SPLIT FLBA spaced decode aliasing output by @zeroshade in [#1256](https://github.com/apache/arrow-go/pull/1256) -* perf(compute): vectorize mixed int32 filters with AVX2 by @fallintoplace in [#1284](https://github.com/apache/arrow-go/pull/1284) -* perf(compute): add ARM64 NEON constant multiplication by @fallintoplace in [#1268](https://github.com/apache/arrow-go/pull/1268) -* perf(compute): vectorize numeric to boolean casts by @fallintoplace in [#1282](https://github.com/apache/arrow-go/pull/1282) -* fix(arrow/scalar): align timestamp timezone parsing by @fallintoplace in [#1109](https://github.com/apache/arrow-go/pull/1109) -* perf(internal/utils): add ARM64 NEON min/max for 8- and 16-bit integers by @fallintoplace in [#1281](https://github.com/apache/arrow-go/pull/1281) -* perf(arrow/array): SIMD-pack boolean slices into bitmaps by @fallintoplace in [#1283](https://github.com/apache/arrow-go/pull/1283) -* chore: fix failing build tags by @zeroshade in [#1285](https://github.com/apache/arrow-go/pull/1285) -* perf(compute): SIMD-compact fragmented filters into take indices by @fallintoplace in [#1287](https://github.com/apache/arrow-go/pull/1287) -* perf(compute): add ARM64 NEON comparisons for 8- and 16-bit integers by @fallintoplace in [#1288](https://github.com/apache/arrow-go/pull/1288) -* perf(compute): add ARM64 NEON compression for null-free 32-bit fixed-width filters by @fallintoplace in [#1286](https://github.com/apache/arrow-go/pull/1286) +* perf(parquet/pqarrow): write FixedSizeBinary values directly by @fallintoplace in [#1263](https://github.com/apache/arrow-go/pull/1263) +* perf(parquet/pqarrow): use synchronous path for serial reads by @fallintoplace in [#1235](https://github.com/apache/arrow-go/pull/1235) +* perf(parquet/compress): reuse gzip writers by @fallintoplace in [#1238](https://github.com/apache/arrow-go/pull/1238) +* perf(parquet/compress): reuse Brotli writers across encodes by @fallintoplace in [#1254](https://github.com/apache/arrow-go/pull/1254) +* perf(arrow/csv): avoid materializing rows before writing by @fallintoplace in [#1190](https://github.com/apache/arrow-go/pull/1190) +* perf(arrow/ipc): reuse record encoder scratch by @fallintoplace in [#1221](https://github.com/apache/arrow-go/pull/1221) ### New Contributors * @C-Loftus made their first contribution in [#960](https://github.com/apache/arrow-go/pull/960) From 4c150ee4b69ccdafcc3ea6bf21bd0f575ec4f3ab Mon Sep 17 00:00:00 2001 From: Matt Topol Date: Tue, 8 Sep 2026 15:51:40 -0400 Subject: [PATCH 3/3] highlight the performance improvements --- _posts/2026-09-08-arrow-go-18.8.0.md | 41 ++++++++++++++++++++++++---- 1 file changed, 35 insertions(+), 6 deletions(-) diff --git a/_posts/2026-09-08-arrow-go-18.8.0.md b/_posts/2026-09-08-arrow-go-18.8.0.md index af5d66ca5580..86582e113b3e 100644 --- a/_posts/2026-09-08-arrow-go-18.8.0.md +++ b/_posts/2026-09-08-arrow-go-18.8.0.md @@ -66,9 +66,28 @@ $ git shortlog -sn v18.7.0..v18.8.0 #### Performance Improvements -- Expanded SIMD acceleration for compute arithmetic, comparisons, and filtering on ARM64 and AMD64 [#1265](https://github.com/apache/arrow-go/pull/1265), [#1266](https://github.com/apache/arrow-go/pull/1266), -[#1284](https://github.com/apache/arrow-go/pull/1284), [#1286](https://github.com/apache/arrow-go/pull/1286), [#1287](https://github.com/apache/arrow-go/pull/1287) -- Added SIMD-accelerated packing of boolean slices and validity data into Arrow bitmaps [#1283](https://github.com/apache/arrow-go/pull/1283) +This is a performance-heavy release: 32 of the merged pull requests are dedicated +optimizations. Unless noted otherwise, the figures below are the PR-reported +benchmarks on an Apple M1 Pro, comparing the new path against the scalar +baseline it replaces. + +- Expanded SIMD acceleration for compute arithmetic, comparisons, and filtering on ARM64 and AMD64. Arithmetic kernels run 4.1-4.4x faster on int64 and float64 +adds, int64 array/array comparisons drop from 313.6us to 29.3us per op (10.7x), narrow 8- and 16-bit integer comparisons reach 12.8-15.1x, and null-free +32-bit fixed-width filters compress up to 11.8x faster on 1M-element inputs [#1265](https://github.com/apache/arrow-go/pull/1265), [#1266](https://github.com/apache/arrow-go/pull/1266), +[#1284](https://github.com/apache/arrow-go/pull/1284), [#1286](https://github.com/apache/arrow-go/pull/1286), [#1287](https://github.com/apache/arrow-go/pull/1287), [#1288](https://github.com/apache/arrow-go/pull/1288) +- Building take indices from fragmented selection vectors is no longer quadratic. For a 1M-element alternating filter it drops from 4.31s to 515us, and a +serial `FilterRecordBatch` over the same filter falls from 2.06s to 831us [#1287](https://github.com/apache/arrow-go/pull/1287) +- Added SIMD-accelerated packing of boolean slices and validity data into Arrow bitmaps: `BooleanBuilder` packing of 65,536 values improves roughly 4-5x +(23.4us to 3.9us), batched validity packing is up to 4.4x faster, and ARM64 NEON bitmap operations gain 2.2-2.9x [#1283](https://github.com/apache/arrow-go/pull/1283), +[#1185](https://github.com/apache/arrow-go/pull/1185), [#1264](https://github.com/apache/arrow-go/pull/1264), [#1090](https://github.com/apache/arrow-go/pull/1090) +- Batched run-end encoded appends turn a millisecond-scale operation into a microsecond-scale one for null and empty-value runs, cutting 1.16MB across 55 +allocations per op down to 1.7KB across 19 [#1214](https://github.com/apache/arrow-go/pull/1214) +- Concatenation avoids temporary buffers and copies bitmaps directly: 65,536 int64 values spread over 8,192 chunks go from 571.7us to 214.5us (-62%) with +allocations falling from 8,199 to 6 per op [#1194](https://github.com/apache/arrow-go/pull/1194), [#1224](https://github.com/apache/arrow-go/pull/1224) +- Serial and small-batch execution skips goroutine and span overhead: single-span kernel dispatch nearly halves latency at length 1 (342.0ns to 189.0ns), +serial record filtering over 128 columns goes from 709.8us to 420.9us, and serial `Take` plus IPC encoder scratch reuse cut per-op allocations across the +board [#1233](https://github.com/apache/arrow-go/pull/1233), [#1245](https://github.com/apache/arrow-go/pull/1245), [#1226](https://github.com/apache/arrow-go/pull/1226), [#1221](https://github.com/apache/arrow-go/pull/1221) +- CSV writing no longer materializes rows first, cutting a 1,000-row by 16-column write from 4.78ms to 0.84ms (-82%) and allocated bytes by 81% [#1190](https://github.com/apache/arrow-go/pull/1190) ### Parquet @@ -87,9 +106,19 @@ $ git shortlog -sn v18.7.0..v18.8.0 #### Performance Improvements -- Added SIMD BYTE_STREAM_SPLIT encoding on ARM64 and AMD64 [#1260](https://github.com/apache/arrow-go/pull/1260) -- Vectorized fixed-width dictionary materialization on AVX2 systems [#1269](https://github.com/apache/arrow-go/pull/1269) -- Reduced copying when writing binary and fixed-size binary Arrow values to Parquet [#1262](https://github.com/apache/arrow-go/pull/1262), [#1263](https://github.com/apache/arrow-go/pull/1263) +- Added SIMD `BYTE_STREAM_SPLIT` encoding on ARM64 and AMD64: 16.9x faster for 4-byte values (113,877ns to 6,745ns per op) and 7.6x for 8-byte values [#1260](https://github.com/apache/arrow-go/pull/1260) +- Boolean decoding now targets Arrow bitmaps directly. All-true RLE boolean pages of 1M values decode in 2.45us instead of 2.65ms, and dense boolean column +reads through pqarrow drop 90% in time and 65% in heap use [#1246](https://github.com/apache/arrow-go/pull/1246), [#1182](https://github.com/apache/arrow-go/pull/1182) +- Batched encoding and decoding across the core codecs: plain `BYTE_ARRAY` encoding is 45-62% faster, RLE level encoding is 79% faster for all-defined +levels, and `DELTA_BINARY_PACKED` decoding is 34-66% faster [#1193](https://github.com/apache/arrow-go/pull/1193), [#1177](https://github.com/apache/arrow-go/pull/1177), [#1169](https://github.com/apache/arrow-go/pull/1169) +- Dictionary encoding allocates far less: numeric insertion of 65,535 float32 values goes from 64,905 allocations per op to 13, and typed byte-array memo +insertion cuts time 24% and allocations 50% (Apple M3 Max). Fixed-width dictionary materialization is also vectorized on AVX2 systems [#1178](https://github.com/apache/arrow-go/pull/1178), +[#1272](https://github.com/apache/arrow-go/pull/1272), [#1269](https://github.com/apache/arrow-go/pull/1269) +- Reduced copying when writing binary and fixed-size binary Arrow values to Parquet: 16-byte `FixedSizeBinary` writes are 72% faster for required columns +with statistics disabled, and 64K String writes go from 1.798ms to 1.260ms [#1262](https://github.com/apache/arrow-go/pull/1262), [#1263](https://github.com/apache/arrow-go/pull/1263) +- Compressors are reused across pages: random 64KiB gzip page encoding is 6.1x faster (94.7us to 15.6us) with allocations down from 15 to 1 per op, and +Brotli encoding drops from about 22 allocations per op to 1 [#1238](https://github.com/apache/arrow-go/pull/1238), [#1254](https://github.com/apache/arrow-go/pull/1254) +- Serial reads skip goroutine orchestration, trimming 18% from single-column row-group reads [#1235](https://github.com/apache/arrow-go/pull/1235) ## Changelog