Skip to content

feat: Add the C Async Device Stream Interface - #503

Draft
CurtHagenlocher wants to merge 7 commits into
apache:mainfrom
CurtHagenlocher:AsyncApi
Draft

CurtHagenlocher wants to merge 7 commits into
apache:mainfrom
CurtHagenlocher:AsyncApi

Conversation

@CurtHagenlocher

@CurtHagenlocher CurtHagenlocher commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

What's Changed

Adds producer and consumer support for the experimental C Async Device Stream Interface (ArrowAsyncDeviceStreamHandler, ArrowAsyncProducer, ArrowAsyncTask). The new API is marked [Experimental("ArrowAsyncDeviceStreamApi")].

Producer: CArrowAsyncDeviceStreamProducer.Produce(IArrowArrayStream, handler*)

  • A single produce loop, driven by request(n), so handler callbacks are never concurrent and the source stream is never read from two threads. A request made inside on_schema doesn't start delivery until on_schema has returned.
  • Follows the spec for errors and shutdown:
    • request(n <= 0) gets on_error(EINVAL) and then release.
    • request after cancel does nothing.
    • cancel ends with release and no on_error.
    • A failed extract_data is followed by on_error.
    • Nothing is called on the handler after release.
  • Frees its ArrowAsyncProducer after release, as the spec allows. extract_data works even after that, since some consumers extract later.
  • Like the C stream exporters (fix: Dispose record batches exported through the C stream interface #483), it takes ownership of each batch it exports.

Consumer: CArrowAsyncDeviceStreamConsumer.CreateStream(out handler*, queueSize = 16)

  • Returns an IArrowArrayStream. It requests queueSize batches up front, then asks for more in chunks as the reader drains them.
  • The consumer allocates the handler and frees it in its own release callback, so callers never have to work out when it's safe to free.
  • Disposing the stream cancels the producer. Batches that arrive afterwards are extracted and dropped.

Testing

  • In-process (20 tests, net8.0/net462/net472): round trips, ordering at queue sizes 1/3/64 (this checks that the source is never read concurrently), a cancelled read not losing a batch, schema export and import failures, and request(n <= 0). Cancellation is covered in several forms: disposing before a producer attaches, interrupting a source read that's waiting for input, and a producer that calls release synchronously from cancel. Some tests use a test consumer or producer written directly against the C callbacks with their own delegate types, so on .NET Framework they also cover Call C Data Interface callbacks through any delegate type on .NET Framework #491.
  • Interop with arrow-go v18.8.0 (11 tests, net8.0): run through a small c-shared Go shim in test/arrow-go-interop, in both directions: round trips, empty streams, errors, cancellation from either side, and arrow-go rejecting a schema. Data enters and leaves the shim through the synchronous C stream interface, so the only async step is the one being tested. A proxy in the shim works around two arrow-go consumer bugs (see proxy.h):

CI

  • New go-interop job on Linux, macOS and Windows. It's separate from the integration job, which is already long. It builds the shim and runs only the interop tests, with ARROW_GO_INTEROP_REQUIRED=1 so a missing shim fails instead of skipping.
  • Elsewhere, the interop tests skip, so the main test job doesn't need Go.

This is a draft partly to find out whether windows-latest has a gcc that cgo can use. If it doesn't, the Windows leg of go-interop will fail at the shim build, and its toolchain step prints go env CC and the compiler version.

🤖 Generated with Claude Code

CurtHagenlocher and others added 2 commits November 15, 2026 05:02
Add producer and consumer support for the experimental Arrow C Async
Device Stream Interface (ArrowAsyncDeviceStreamHandler, ArrowAsyncProducer
and ArrowAsyncTask), marked [Experimental("ArrowAsyncDeviceStreamApi")].

- CArrowAsyncDeviceStreamProducer.Produce pushes an IArrowArrayStream
  through a handler. A single produce loop driven by request(n) keeps
  handler callbacks serialized; request(n <= 0) is reported as EINVAL via
  on_error; cancel ends with release and no on_error; the producer struct
  is freed after release, as the spec allows. As with the C stream
  exporters, exported batches are owned by the exported array.
- CArrowAsyncDeviceStreamConsumer.CreateStream(out handler, queueSize)
  returns an IArrowArrayStream with bounded request()-based backpressure.
  The consumer owns the handler and frees it in its release callback.
  Disposing the stream cancels the producer.

Interop tests run against arrow-go v18.8.0 through a small c-shared Go
shim (test/arrow-go-interop), in both directions, including errors and
cancellation. The shim works around apache/arrow-go#1572 (request() after
release) with a proxy producer. They run in a new go-interop CI job and
are skipped elsewhere unless ARROW_GO_INTEROP_REQUIRED=1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Release completion is published before callbacks finish accessing native state, allowing concurrent cleanup to cause use-after-free in the interop proxy.

9 open issues
What changed in this PR

Adds experimental C Async Device Stream support to Apache Arrow .NET, connecting managed IArrowArrayStream implementations with native producers and consumers.

Changes:

  • Adds async interface structures, demand-driven production, and buffered consumption with cancellation and ownership handling.
  • Adds in-process tests and bidirectional interoperability tests against arrow-go.
  • Adds a dedicated cross-platform interop CI job and supporting build scripts.
File Description
test/​arrow-go-interop/​proxy.h Declares the arrow-go compatibility proxy.
test/​arrow-go-interop/​proxy.c Implements proxy forwarding and lifetime handling.
test/​arrow-go-interop/​main.go Exposes Go producer and consumer adapters for testing.
test/​arrow-go-interop/​go.sum Records Go dependency checksums.
test/​arrow-go-interop/​go.mod Defines the interop module and dependencies.
test/​Apache.Arrow.Tests/​CAsyncDeviceStreamTests.cs Tests round trips, ordering, errors, and cancellation.
test/​Apache.Arrow.Tests/​CAsyncDeviceStreamGoInteropTests.cs Tests async interoperability in both directions.
test/​Apache.Arrow.Tests/​ArrowGoInteropFixture.cs Loads and invokes the native Go test library.
src/​Apache.Arrow/​C/​CArrowAsyncTask.cs Defines the native async task structure.
src/​Apache.Arrow/​C/​CArrowAsyncProducer.cs Defines producer flow-control callbacks and allocation helpers.
src/​Apache.Arrow/​C/​CArrowAsyncDeviceStreamProducer.cs Implements demand-driven production from managed streams.
src/​Apache.Arrow/​C/​CArrowAsyncDeviceStreamHandler.cs Defines consumer callbacks and the native handler structure.
src/​Apache.Arrow/​C/​CArrowAsyncDeviceStreamConsumer.cs Adapts native async delivery to a managed stream.
dev/​release/​rat_exclude_files.txt Excludes generated Go checksums from license-header checks.
ci/​scripts/​test_go_interop.sh Runs required interop tests and reports Go crashes.
ci/​scripts/​build_go_interop.sh Builds the Go shared library.
.github/​workflows/​test.yaml Adds Linux, macOS, and Windows interop CI coverage.

Review effort: Balanced

Comment thread test/arrow-go-interop/proxy.c Resolved
Comment thread test/arrow-go-interop/proxy.c Resolved
Producer:
- Recheck pending errors and cancellation after each source read, and stop
  accepting request/cancel/errors once the stream has committed to its
  final callbacks, so a request(n <= 0) can't be lost to end-of-stream.
- Pass a producer-owned cancellation token to source reads and cancel it
  from cancel(), so a read waiting for input can't block release.

Consumer:
- If the stream is disposed before a producer attaches, send the cancel
  from on_schema instead of leaving the producer with no demand.
- Report schema import failures to the reader instead of ending the
  stream as if it were empty.
- Dispose batches that arrive after the stream is disposed, and fail
  reads still waiting when it's disposed.

Go interop harness:
- Only report the proxy and outer handler as released after the last
  access to them, and read the handler's release field with matching
  ordering in the tests.
- Fail the .NET-to-Go tests if the producer never releases the handler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The change still has active moderate and critical concurrency issues that can cause reads to hang indefinitely and cancellation to deadlock during synchronous release.

1 open issue
1 previously missed issue

In code that hasn't changed since last review

Medium severity Recheck disposal under the lock before enqueueing waiters

src/​Apache.Arrow/​C/​CArrowAsyncDeviceStreamConsumer.cs:545

Dispose sets _disposed before taking _gate, but this method only checks _disposed before entering the lock. A read can pass that check, be delayed while Dispose drains the waiters, and then enqueue a new waiter in the else branch after disposal; no producer callback remains to complete it, so the read can hang forever. Recheck _disposed while holding _gate before enqueueing the waiter (and before the other queue-state branches).

9 resolved since last review

Review effort: Lite

ReadNextRecordBatchAsync checked for disposal only before taking the
queue lock, so a read racing Dispose could add a waiter after Dispose had
drained them, and wait forever. Check again under the lock.

Also document why the consumer holds its producer lock across request and
cancel, and test that a producer calling release synchronously from cancel
doesn't deadlock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@CurtHagenlocher

Copy link
Copy Markdown
Contributor Author

On the "previously missed" item in the latest review (recheck disposal under the lock in ReadNextRecordBatchAsync): fixed in b44fe7c. The read now checks _disposed again while holding _gate and fails with ObjectDisposedException, so it can no longer add a waiter after Dispose has drained them. It's a race between threads with no reliable way to test it, so this one is fixed by inspection.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

A critical active finding shows that failed schema imports can leave an installed release callback pointing to freed data, allowing the producer cleanup path to crash the interop process.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

High severity Avoid releasing the freed inner handler after schema import fails

test/​arrow-go-interop/​proxy.c:116

In arrow-go v18.8.0, a schema/device import failure in asyncStreamOnSchema frees and deletes the inner handler's private_data before returning a nonzero status, but leaves inner.release installed. The .NET producer then correctly calls outer_release after the failed on_schema, and this unconditional call to inner.release makes arrow-go dereference that freed handle. A schema incompatibility can therefore crash the interop process; track the failed-schema state and suppress or safely handle the inner release callback on that path.

1 resolved since last review

Review effort: Lite

When arrow-go's async consumer rejects a schema, its on_schema frees the
handler state but leaves release installed, and release then panics on
the freed handle (apache/arrow-go#1573). The spec has the producer call
release after a failed on_schema, so the interop proxy no longer forwards
release to arrow-go in that case. A new test sends arrow-go a schema it
can't import and checks that release completes cleanly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@CurtHagenlocher
CurtHagenlocher requested a lite review from Copilot November 15, 2026 14:32
@CurtHagenlocher

Copy link
Copy Markdown
Contributor Author

On the "previously missed" item in the latest review (the inner handler's release after a failed schema import, proxy.c): that one is real, and it's a bug in arrow-go v18.8.0, now filed as apache/arrow-go#1573. When asyncStreamOnSchema fails, it frees its handler state but leaves release installed. asyncStreamRelease then panics with runtime/cgo: misuse of an invalid Handle.

Fixed in 4848e65:

  • The proxy now records when arrow-go's on_schema failed, and in that case doesn't forward release to arrow-go, since arrow-go has already cleaned up.
  • A new test, GoConsumerRejectingSchemaReleasesCleanly, acts as the producer, sends a schema arrow-go can't import, then calls release as the spec requires. Before the fix it crashed the test host every time; with the fix, all of the async tests passed 20 of 20 runs.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

A moderate error-path interaction in the async producer may suppress the documented on_error notification after export failure, so human review is needed to validate terminal-state handling.

No open issues 🎉

Review effort: Lite

The consumer dropped a batch whose extract_data failed and relied on the
producer to follow up with on_error. A producer can't always do that (for
example when the consumer extracts after it has finished), so the stream
could end normally with the batch silently missing. The consumer now
records the failure itself and tells the producer to stop.

Also explain in the producer why a late extract failure isn't reported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@CurtHagenlocher

Copy link
Copy Markdown
Contributor Author

On the latest review's summary (an export failure possibly suppressing on_error): no change to the producer, but it pointed at a real gap in the consumer, fixed in 684d1df.

Producer, unchanged. When exporting a batch fails inside extract_data, the producer returns the error and reports it through on_error if it still can. It drops the report once the stream is finishing or released, because nothing may follow release, and an on_error after end-of-stream would contradict the end of the stream. Consumers may extract that late (arrow-go does), so on_error can't be guaranteed for a failed extract. That comes from the spec. There's now a comment explaining this in ReportError.

Consumer, fixed. On a failed extract_data, the consumer used to drop the batch and wait for the producer's on_error. If none came, the stream ended normally with that batch missing. It now records the failure as a stream error, with the error code, and returns non-zero so the producer stops. The new FailedExtractIsReportedWithoutOnError test uses a fake producer that fails extract_data and releases without on_error. It fails with the old behavior and passes with the fix.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because the proxy’s cancellation path can deadlock when a producer synchronously invokes release while the proxy lock is held.

No open issues 🎉

Review effort: Lite

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@CurtHagenlocher

Copy link
Copy Markdown
Contributor Author

On the latest review's summary (the proxy's cancel path deadlocking if a producer calls release synchronously): no behavior change, now documented in 2693c2e.

It's true of the proxy in general. proxy_request and proxy_cancel hold a non-reentrant spinlock while calling the real producer, and outer_release takes the same lock, so a producer that called release from inside request or cancel would spin forever.

It can't happen with this proxy, though. The proxy is test-only, and the only producer it forwards to is Apache.Arrow's. That producer's request and cancel only update state: they start the produce loop with Task.Run, and cancellation of a pending source read also runs on the thread pool. So release always comes from another thread, and outer_release just waits briefly for the forwarded call to return. A comment in proxy.c now states that assumption.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because the change contains moderate hidden issues in async error propagation and partial schema-export cleanup, making its behavior and resource safety too complex to approve confidently.

No open issues 🎉

Review effort: Lite

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants