Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 53 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ and result sets cross as Apache Arrow IPC byte streams.
| Path | Contents |
| --- | --- |
| `src/Apache.DataFusion/` | Managed SDK (public API + interop) |
| `src/Apache.DataFusion.TableProviders.*/` | Managed database table providers |
| `native/` | Rust `cdylib` exposing the C ABI |
| `proto/` | Protobuf contracts shared by C# and Rust |
| `tests/Apache.DataFusion.Tests/` | xUnit test suite |
Expand All @@ -30,8 +31,9 @@ and result sets cross as Apache Arrow IPC byte streams.

## Prerequisites

- **.NET SDK 9.0.200+** — required to build the `.slnx` solution format
(pinned to `9.0.301` in `global.json`). The libraries still target `net8.0`.
- **.NET SDK 10.0.300+** — required to build the `.slnx` solution format
(pinned to `10.0.300` in `global.json`). The core SDK targets `net8.0`;
database table-provider packages target `net10.0`.
- **Rust toolchain** (stable) — to build the native `cdylib`.

## Install
Expand Down Expand Up @@ -132,6 +134,9 @@ Each scenario is an independent, runnable console project under `examples/`:
| `Apache.DataFusion.Sample.Streaming` | Stream Arrow batches and read typed columns |
| `Apache.DataFusion.Sample.SessionConfig` | Configure the session builder + observability |
| `Apache.DataFusion.Sample.Excel` | Read an `.xlsx` spreadsheet and run a SQL query |
| `Apache.DataFusion.Sample.MySql` | Query a MySQL table through a managed streaming provider |
| `Apache.DataFusion.Sample.MongoDB` | Query a MongoDB collection through a managed streaming provider |
| `Apache.DataFusion.Sample.ClickHouse` | Query a ClickHouse table through the official `ClickHouse.Driver` |

Run any of them with:

Expand All @@ -156,7 +161,8 @@ dotnet run --project examples/Apache.DataFusion.Sample.Sql
- **Table providers**: `SimpleTableProvider` backed by Arrow IPC and a native `MemTable`;
`StreamingTableProvider`, a lazy callback-based provider whose `Scan` is invoked by the
engine and streamed in over the Arrow C Data Interface (connect any managed source, e.g.
ADO.NET, with no native database driver).
ADO.NET, with no native database driver). Managed provider packages currently include
MySQL, MongoDB, and ClickHouse.
- **Scalar UDF**: zero-argument `Int64` managed callbacks.
- **Typed exceptions**: `DataFusionException` (base), plus `DataFusionPlanException`,
`DataFusionExecutionException`, `DataFusionIoException`,
Expand Down Expand Up @@ -213,6 +219,50 @@ Console.WriteLine(df.Count());
may run on native worker threads and more than once over the table's lifetime; each call
must return a new, independently consumable stream, which the engine disposes once read.

### Managed database providers

Managed database provider packages are built on `StreamingTableProvider`. They keep database
drivers on the C# side and push simple projection, filter, and limit requests down to the
source query when possible.

| Provider package | Extension method | Driver |
| --- | --- | --- |
| `Apache.DataFusion.TableProviders.MySql` | `RegisterMySql` | `MySqlConnector` |
| `Apache.DataFusion.TableProviders.MongoDB` | `RegisterMongoDb` | `MongoDB.Driver` |
| `Apache.DataFusion.TableProviders.ClickHouse` | `RegisterClickHouse` | official `ClickHouse.Driver` |

```csharp
using Apache.DataFusion;
using Apache.DataFusion.TableProviders.ClickHouse;

using SessionContext context = new();
context.RegisterClickHouse("audit_log", new ClickHouseTableOptions
{
ConnectionString = "Host=localhost;Port=8123;Username=default;Password=;Database=default",
Query = """SELECT * FROM audit_log""",
});

using DataFrame df = context.Sql("""
SELECT *
FROM audit_log
LIMIT 10
""");
df.Show();
```

The registered DataFusion table name is the first argument to the extension method. The
provider's `Query` points at the remote database table or subquery. For example, if you
register `audit_log`, query `FROM audit_log` in DataFusion SQL; if you register `orders`,
query `FROM orders`.

The sample projects use environment variables for connection strings:

| Sample | Environment variable |
| --- | --- |
| `Apache.DataFusion.Sample.MySql` | `DATAFUSION_MYSQL_CONNECTION` |
| `Apache.DataFusion.Sample.MongoDB` | `DATAFUSION_MONGODB_CONNECTION` |
| `Apache.DataFusion.Sample.ClickHouse` | `DATAFUSION_CLICKHOUSE_CONNECTION` |

## Native packaging

Native binaries are packaged through RID-specific projects under `packages/`, placing the
Expand Down
2 changes: 1 addition & 1 deletion examples/Apache.DataFusion.Sample.ClickHouse/Program.cs
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
using Apache.DataFusion;
using Apache.DataFusion.TableProviders.ClickHouse;

using ClickHouse.Client.ADO;
using ClickHouse.Driver.ADO;

string? connectionString = Environment.GetEnvironmentVariable("DATAFUSION_CLICKHOUSE_CONNECTION");
if (string.IsNullOrWhiteSpace(connectionString))
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
</ItemGroup>

<ItemGroup>
<PackageReference Include="ClickHouse.Client" Version="7.14.0" />
<PackageReference Include="ClickHouse.Driver" Version="1.2.0" />
</ItemGroup>

</Project>
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
using Apache.Arrow.Ipc;
using Apache.DataFusion.TableProviders.ClickHouse.Sql;

using ClickHouse.Client.ADO;
using ClickHouse.Driver.ADO;

namespace Apache.DataFusion.TableProviders.ClickHouse;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
using Apache.Arrow.Ipc;
using Apache.DataFusion.TableProviders.ClickHouse.Sql;

using ClickHouse.Client.ADO;
using ClickHouse.Driver.ADO;

namespace Apache.DataFusion.TableProviders.ClickHouse;

Expand Down
Loading