You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat: Add the C Async Device Stream Interface - #503
Adds producer and consumer support for the experimental C Async Device Stream Interface (ArrowAsyncDeviceStreamHandler, ArrowAsyncProducer, ArrowAsyncTask). The new API is marked [Experimental("ArrowAsyncDeviceStreamApi")].
A single produce loop, driven by request(n), so handler callbacks are never concurrent and the source stream is never read from two threads. A request made inside on_schema doesn't start delivery until on_schema has returned.
Follows the spec for errors and shutdown:
request(n <= 0) gets on_error(EINVAL) and then release.
request after cancel does nothing.
cancel ends with release and no on_error.
A failed extract_data is followed by on_error.
Nothing is called on the handler after release.
Frees its ArrowAsyncProducer after release, as the spec allows. extract_data works even after that, since some consumers extract later.
Returns an IArrowArrayStream. It requests queueSize batches up front, then asks for more in chunks as the reader drains them.
The consumer allocates the handler and frees it in its own release callback, so callers never have to work out when it's safe to free.
Disposing the stream cancels the producer. Batches that arrive afterwards are extracted and dropped.
Testing
In-process (20 tests, net8.0/net462/net472): round trips, ordering at queue sizes 1/3/64 (this checks that the source is never read concurrently), a cancelled read not losing a batch, schema export and import failures, and request(n <= 0). Cancellation is covered in several forms: disposing before a producer attaches, interrupting a source read that's waiting for input, and a producer that calls release synchronously from cancel. Some tests use a test consumer or producer written directly against the C callbacks with their own delegate types, so on .NET Framework they also cover Call C Data Interface callbacks through any delegate type on .NET Framework #491.
Interop with arrow-go v18.8.0 (11 tests, net8.0): run through a small c-shared Go shim in test/arrow-go-interop, in both directions: round trips, empty streams, errors, cancellation from either side, and arrow-go rejecting a schema. Data enters and leaves the shim through the synchronous C stream interface, so the only async step is the one being tested. A proxy in the shim works around two arrow-go consumer bugs (see proxy.h):
New go-interop job on Linux, macOS and Windows. It's separate from the integration job, which is already long. It builds the shim and runs only the interop tests, with ARROW_GO_INTEROP_REQUIRED=1 so a missing shim fails instead of skipping.
Elsewhere, the interop tests skip, so the main test job doesn't need Go.
This is a draft partly to find out whether windows-latest has a gcc that cgo can use. If it doesn't, the Windows leg of go-interop will fail at the shim build, and its toolchain step prints go env CC and the compiler version.
Add producer and consumer support for the experimental Arrow C Async
Device Stream Interface (ArrowAsyncDeviceStreamHandler, ArrowAsyncProducer
and ArrowAsyncTask), marked [Experimental("ArrowAsyncDeviceStreamApi")].
- CArrowAsyncDeviceStreamProducer.Produce pushes an IArrowArrayStream
through a handler. A single produce loop driven by request(n) keeps
handler callbacks serialized; request(n <= 0) is reported as EINVAL via
on_error; cancel ends with release and no on_error; the producer struct
is freed after release, as the spec allows. As with the C stream
exporters, exported batches are owned by the exported array.
- CArrowAsyncDeviceStreamConsumer.CreateStream(out handler, queueSize)
returns an IArrowArrayStream with bounded request()-based backpressure.
The consumer owns the handler and frees it in its release callback.
Disposing the stream cancels the producer.
Interop tests run against arrow-go v18.8.0 through a small c-shared Go
shim (test/arrow-go-interop), in both directions, including errors and
cancellation. The shim works around apache/arrow-go#1572 (request() after
release) with a proxy producer. They run in a new go-interop CI job and
are skipped elsewhere unless ARROW_GO_INTEROP_REQUIRED=1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Release completion is published before callbacks finish accessing native state, allowing concurrent cleanup to cause use-after-free in the interop proxy.
Adds experimental C Async Device Stream support to Apache Arrow .NET, connecting managed IArrowArrayStream implementations with native producers and consumers.
Changes:
Adds async interface structures, demand-driven production, and buffered consumption with cancellation and ownership handling.
Adds in-process tests and bidirectional interoperability tests against arrow-go.
Adds a dedicated cross-platform interop CI job and supporting build scripts.
File
Description
test/arrow-go-interop/proxy.h
Declares the arrow-go compatibility proxy.
test/arrow-go-interop/proxy.c
Implements proxy forwarding and lifetime handling.
test/arrow-go-interop/main.go
Exposes Go producer and consumer adapters for testing.
Producer:
- Recheck pending errors and cancellation after each source read, and stop
accepting request/cancel/errors once the stream has committed to its
final callbacks, so a request(n <= 0) can't be lost to end-of-stream.
- Pass a producer-owned cancellation token to source reads and cancel it
from cancel(), so a read waiting for input can't block release.
Consumer:
- If the stream is disposed before a producer attaches, send the cancel
from on_schema instead of leaving the producer with no demand.
- Report schema import failures to the reader instead of ending the
stream as if it were empty.
- Dispose batches that arrive after the stream is disposed, and fail
reads still waiting when it's disposed.
Go interop harness:
- Only report the proxy and outer handler as released after the last
access to them, and read the handler's release field with matching
ordering in the tests.
- Fail the .NET-to-Go tests if the producer never releases the handler.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The change still has active moderate and critical concurrency issues that can cause reads to hang indefinitely and cancellation to deadlock during synchronous release.
Dispose sets _disposed before taking _gate, but this method only checks _disposed before entering the lock. A read can pass that check, be delayed while Dispose drains the waiters, and then enqueue a new waiter in the else branch after disposal; no producer callback remains to complete it, so the read can hang forever. Recheck _disposed while holding _gate before enqueueing the waiter (and before the other queue-state branches).
ReadNextRecordBatchAsync checked for disposal only before taking the
queue lock, so a read racing Dispose could add a waiter after Dispose had
drained them, and wait forever. Check again under the lock.
Also document why the consumer holds its producer lock across request and
cancel, and test that a producer calling release synchronously from cancel
doesn't deadlock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the "previously missed" item in the latest review (recheck disposal under the lock in ReadNextRecordBatchAsync): fixed in b44fe7c. The read now checks _disposed again while holding _gate and fails with ObjectDisposedException, so it can no longer add a waiter after Dispose has drained them. It's a race between threads with no reliable way to test it, so this one is fixed by inspection.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
A critical active finding shows that failed schema imports can leave an installed release callback pointing to freed data, allowing the producer cleanup path to crash the interop process.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Avoid releasing the freed inner handler after schema import fails
test/arrow-go-interop/proxy.c:116
In arrow-go v18.8.0, a schema/device import failure in asyncStreamOnSchema frees and deletes the inner handler's private_data before returning a nonzero status, but leaves inner.release installed. The .NET producer then correctly calls outer_release after the failed on_schema, and this unconditional call to inner.release makes arrow-go dereference that freed handle. A schema incompatibility can therefore crash the interop process; track the failed-schema state and suppress or safely handle the inner release callback on that path.
When arrow-go's async consumer rejects a schema, its on_schema frees the
handler state but leaves release installed, and release then panics on
the freed handle (apache/arrow-go#1573). The spec has the producer call
release after a failed on_schema, so the interop proxy no longer forwards
release to arrow-go in that case. A new test sends arrow-go a schema it
can't import and checks that release completes cleanly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the "previously missed" item in the latest review (the inner handler's release after a failed schema import, proxy.c): that one is real, and it's a bug in arrow-go v18.8.0, now filed as apache/arrow-go#1573. When asyncStreamOnSchema fails, it frees its handler state but leaves release installed. asyncStreamRelease then panics with runtime/cgo: misuse of an invalid Handle.
The proxy now records when arrow-go's on_schema failed, and in that case doesn't forward release to arrow-go, since arrow-go has already cleaned up.
A new test, GoConsumerRejectingSchemaReleasesCleanly, acts as the producer, sends a schema arrow-go can't import, then calls release as the spec requires. Before the fix it crashed the test host every time; with the fix, all of the async tests passed 20 of 20 runs.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
A moderate error-path interaction in the async producer may suppress the documented on_error notification after export failure, so human review is needed to validate terminal-state handling.
The consumer dropped a batch whose extract_data failed and relied on the
producer to follow up with on_error. A producer can't always do that (for
example when the consumer extracts after it has finished), so the stream
could end normally with the batch silently missing. The consumer now
records the failure itself and tells the producer to stop.
Also explain in the producer why a late extract failure isn't reported.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the latest review's summary (an export failure possibly suppressing on_error): no change to the producer, but it pointed at a real gap in the consumer, fixed in 684d1df.
Producer, unchanged. When exporting a batch fails inside extract_data, the producer returns the error and reports it through on_error if it still can. It drops the report once the stream is finishing or released, because nothing may follow release, and an on_error after end-of-stream would contradict the end of the stream. Consumers may extract that late (arrow-go does), so on_error can't be guaranteed for a failed extract. That comes from the spec. There's now a comment explaining this in ReportError.
Consumer, fixed. On a failed extract_data, the consumer used to drop the batch and wait for the producer's on_error. If none came, the stream ended normally with that batch missing. It now records the failure as a stream error, with the error code, and returns non-zero so the producer stops. The new FailedExtractIsReportedWithoutOnError test uses a fake producer that fails extract_data and releases without on_error. It fails with the old behavior and passes with the fix.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because the proxy’s cancellation path can deadlock when a producer synchronously invokes release while the proxy lock is held.
On the latest review's summary (the proxy's cancel path deadlocking if a producer calls release synchronously): no behavior change, now documented in 2693c2e.
It's true of the proxy in general. proxy_request and proxy_cancel hold a non-reentrant spinlock while calling the real producer, and outer_release takes the same lock, so a producer that called release from inside request or cancel would spin forever.
It can't happen with this proxy, though. The proxy is test-only, and the only producer it forwards to is Apache.Arrow's. That producer's request and cancel only update state: they start the produce loop with Task.Run, and cancellation of a pending source read also runs on the thread pool. So release always comes from another thread, and outer_release just waits briefly for the forwarded call to return. A comment in proxy.c now states that assumption.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because the change contains moderate hidden issues in async error propagation and partial schema-export cleanup, making its behavior and resource safety too complex to approve confidently.
No open issues 🎉
Review effort: Lite
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's Changed
Adds producer and consumer support for the experimental C Async Device Stream Interface (
ArrowAsyncDeviceStreamHandler,ArrowAsyncProducer,ArrowAsyncTask). The new API is marked[Experimental("ArrowAsyncDeviceStreamApi")].Producer:
CArrowAsyncDeviceStreamProducer.Produce(IArrowArrayStream, handler*)request(n), so handler callbacks are never concurrent and the source stream is never read from two threads. Arequestmade insideon_schemadoesn't start delivery untilon_schemahas returned.request(n <= 0)getson_error(EINVAL)and thenrelease.requestaftercanceldoes nothing.cancelends withreleaseand noon_error.extract_datais followed byon_error.release.ArrowAsyncProducerafterrelease, as the spec allows.extract_dataworks even after that, since some consumers extract later.Consumer:
CArrowAsyncDeviceStreamConsumer.CreateStream(out handler*, queueSize = 16)IArrowArrayStream. It requestsqueueSizebatches up front, then asks for more in chunks as the reader drains them.releasecallback, so callers never have to work out when it's safe to free.Testing
request(n <= 0). Cancellation is covered in several forms: disposing before a producer attaches, interrupting a source read that's waiting for input, and a producer that callsreleasesynchronously fromcancel. Some tests use a test consumer or producer written directly against the C callbacks with their own delegate types, so on .NET Framework they also cover Call C Data Interface callbacks through any delegate type on .NET Framework #491.test/arrow-go-interop, in both directions: round trips, empty streams, errors, cancellation from either side, and arrow-go rejecting a schema. Data enters and leaves the shim through the synchronous C stream interface, so the only async step is the one being tested. A proxy in the shim works around two arrow-go consumer bugs (seeproxy.h):request()on the producer after the handler has been released arrow-go#1572: arrow-go callsrequest()on the producer after release.on_schema, arrow-go'sreleaseuses state it has already freed.CI
go-interopjob on Linux, macOS and Windows. It's separate from the integration job, which is already long. It builds the shim and runs only the interop tests, withARROW_GO_INTEROP_REQUIRED=1so a missing shim fails instead of skipping.testjob doesn't need Go.This is a draft partly to find out whether
windows-latesthas a gcc that cgo can use. If it doesn't, the Windows leg ofgo-interopwill fail at the shim build, and its toolchain step printsgo env CCand the compiler version.🤖 Generated with Claude Code