Skip to content

Render Arrow record batches progressively - #21

Draft
e-kotov wants to merge 1 commit into
r-spatial:mainfrom
e-kotov:feat/progressive-arrow-batches
Draft

e-kotov wants to merge 1 commit into
r-spatial:mainfrom
e-kotov:feat/progressive-arrow-batches

Conversation

@e-kotov

@e-kotov e-kotov commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Arrow.tableFromIPC(response) waits for the complete HTTP response before exposing any record batches, so smaller server-side batches currently create more layers without showing pixels earlier.

This change reads Arrow IPC with RecordBatchReader.from(response.body) and appends one stable Deck.gl layer as each batch arrives. The shared loader keeps the current parse2ArrowTable() path for GeoParquet and FlatGeobuf, including the new upstream FlatGeobuf support.

Verification

  • node tests-js/progressive-arrow-batches.test.mjs
  • JavaScript syntax checks for every file in inst/htmlwidgets/
  • R CMD build .
  • R CMD check --no-manual deckglgeoarrow_0.1.0.9001.tar.gz

The JavaScript regression test verifies that the first layer is installed before the second Arrow batch is read and that the non-Arrow parser path remains available. The package check completes with the warnings already present on current main for the removed point-cloud methods and the development geoarrowWidget FlatGeobuf helper.

@e-kotov
e-kotov marked this pull request as draft September 17, 2026 14:11
@e-kotov

e-kotov commented Sep 17, 2026

Copy link
Copy Markdown
Contributor Author

@tim-salabim this is in relation to the batching question, but ignore for now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant