[jbonofre]

What's Changed

Found while reviewing #1422. This change is independent of it and based on main.

New option: ArrowReader.setValidateBatches(boolean), off by default. When it is on, every record batch and dictionary batch is checked with ValueVectorUtility.validateFull once it is loaded. A batch that fails the check is cleared, and the load throws ValidateUtil.ValidateException. A delta dictionary is checked before it is appended, so an invalid delta leaves the dictionary unchanged. The option is meant for reading untrusted data: without it, a malformed batch, such as a Utf8 column with an offset past the end of its data buffer, loads fine and is then read out of bounds (without bounds checks when arrow.enable_unsafe_memory_access is set). The option is off by default because the check reads every value.

validateFull now works on every type a batch can hold. Turning the option on in TestRoundTrip showed that it did not:

• ListViewVector / LargeListViewVector: added to the type, buffer and data validators (they threw UnsupportedOperationException). The data check rejects a non-empty list view outside the inner vector.
• RunEndEncodedVector: added to the data validator. Run ends must be strictly increasing, and each run must have a value.
• FixedSizeListVector: the data validator no longer reads an offset buffer the vector does not have (UnsupportedOperationException for any non-empty vector).
• MapVector: the type validator accepted only ArrowType.List, so it rejected every map.
• View vectors: each non-inline view is checked against the data buffers before validateScalars reads it. TestVariableWidthViewVector.testValidateInvalidOffsets now expects a ValidateException instead of the IndexOutOfBoundsException the out-of-bounds read used to throw.
• Dense unions: negative offsets and type ids without a sub-vector are rejected.

Tests

• TestArrowReaderValidation: a valid batch with string views, list views, large list views and run-end encoding passes. A batch with an invalid interior offset, and a dictionary batch with one, are rejected only when the option is on.
• TestValidateVectorFull: a case for each validator change.
• TestRoundTrip reads every batch with the option on, which covers struct, union, dictionary, fixed-size list, map and the other round-tripped types.
• ValueVectorDataPopulator.setVector(LargeListViewVector, ...) wrote 4-byte sizes at the wrong stride. It now writes 64-bit offsets and sizes (it was not used by any other test).

Without the validator changes, the new tests and the fixed-size list, map, list-as-map and union V5 round trips fail. The full vector test suite passes.

Not covered: C Data imports carry no buffer sizes (the importer sizes buffers from the producer's length), and Flight's FlightStream loads batches with its own VectorLoader.

Closes #1466.