[Java] Add an option to fully validate IPC batches when reading them #1466

Describe the enhancement requested

Found while reviewing #1422. IPC readers (ArrowStreamReader, ArrowFileReader) trust the offsets, views and run ends of the batches they load. A malformed batch, for example a Utf8 column with an offset past the end of its data buffer, loads without error, and reading it then reads past the end of the buffer. With bounds checking this throws IndexOutOfBoundsException; with arrow.enable_unsafe_memory_access=true it reads arbitrary memory.

Fixed-width columns with a length larger than their buffers are not affected: VectorLoader sets the row count after loading, which re-allocates them.

ValueVectorUtility.validateFull finds these problems, but readers never call it, and it fails or is incomplete on several vector types:

• ListViewVector and LargeListViewVector: no validator supports them (UnsupportedOperationException).
• RunEndEncodedVector: not supported by the data validator.
• FixedSizeListVector: the data validator reads its offset buffer, which throws UnsupportedOperationException for any non-empty vector.
• MapVector: the type validator expects type List, so validate() rejects every map.
• View vectors: views are not checked against the data buffers, and validateScalars then reads them out of bounds.
• Dense unions: negative offsets and type ids without a sub-vector are not checked.

Proposal: add an opt-in reader option that runs validateFull on every loaded record batch and dictionary batch, and fix the gaps above so it works on every type a batch can hold. The option is off by default, since the check reads every value.

Component(s)

Java