Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
root = true

[*.{cs,vb}]
# Keep C# using directives in one block when the IDE organizes imports.
dotnet_sort_system_directives_first = true
dotnet_separate_import_directive_groups = false
dotnet_diagnostic.IDE0055.severity = warning
dotnet_diagnostic.IDE0005.severity = warning
64 changes: 50 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,10 @@ Each scenario is an independent, runnable console project under `examples/`:
| `Apache.DataFusion.Sample.Streaming` | Stream Arrow batches and read typed columns |
| `Apache.DataFusion.Sample.SessionConfig` | Configure the session builder + observability |
| `Apache.DataFusion.Sample.Excel` | Read an `.xlsx` spreadsheet and run a SQL query |
| `Apache.DataFusion.Sample.FederatedQuery` | Run one SQL query across SQLite and CSV sources |
| `Apache.DataFusion.Sample.MySql` | Query a MySQL table through a managed streaming provider |
| `Apache.DataFusion.Sample.PostgreSql` | Query a PostgreSQL table through a managed streaming provider |
| `Apache.DataFusion.Sample.Sqlite` | Query a SQLite table through a managed streaming provider |
| `Apache.DataFusion.Sample.MongoDB` | Query a MongoDB collection through a managed streaming provider |
| `Apache.DataFusion.Sample.ClickHouse` | Query a ClickHouse table through the official `ClickHouse.Driver` |

Expand Down Expand Up @@ -162,7 +165,7 @@ dotnet run --project examples/Apache.DataFusion.Sample.Sql
`StreamingTableProvider`, a lazy callback-based provider whose `Scan` is invoked by the
engine and streamed in over the Arrow C Data Interface (connect any managed source, e.g.
ADO.NET, with no native database driver). Managed provider packages currently include
MySQL, MongoDB, and ClickHouse.
MySQL, PostgreSQL, SQLite, MongoDB, and ClickHouse.
- **Scalar UDF**: zero-argument `Int64` managed callbacks.
- **Typed exceptions**: `DataFusionException` (base), plus `DataFusionPlanException`,
`DataFusionExecutionException`, `DataFusionIoException`,
Expand Down Expand Up @@ -222,44 +225,77 @@ must return a new, independently consumable stream, which the engine disposes on
### Managed database providers

Managed database provider packages are built on `StreamingTableProvider`. They keep database
drivers on the C# side and push simple projection, filter, and limit requests down to the
source query when possible.
drivers on the C# side, register discovered tables and collections in the current
`SessionContext`, and push simple projection, filter, and limit requests down when possible.

| Provider package | Extension method | Driver |
| --- | --- | --- |
| `Apache.DataFusion.TableProviders.MySql` | `RegisterMySql` | `MySqlConnector` |
| `Apache.DataFusion.TableProviders.PostgreSql` | `RegisterPostgreSql` | `Npgsql` |
| `Apache.DataFusion.TableProviders.Sqlite` | `RegisterSqlite` | `Microsoft.Data.Sqlite` |
| `Apache.DataFusion.TableProviders.MongoDB` | `RegisterMongoDb` | `MongoDB.Driver` |
| `Apache.DataFusion.TableProviders.ClickHouse` | `RegisterClickHouse` | official `ClickHouse.Driver` |

```csharp
using Apache.DataFusion;
using Apache.DataFusion.TableProviders.ClickHouse;
using Apache.DataFusion.TableProviders.PostgreSql;

using SessionContext context = new();
context.RegisterClickHouse("audit_log", new ClickHouseTableOptions
{
ConnectionString = "Host=localhost;Port=8123;Username=default;Password=;Database=default",
Query = """SELECT * FROM audit_log""",
});
context.RegisterPostgreSql("Host=localhost;Username=postgres;Password=pass;Database=datafusion_sample");

using DataFrame df = context.Sql("""
SELECT *
FROM datafusion_orders
LIMIT 10
""");
df.Show();
```

SQL database providers register tables and views from the selected database in one call:

```csharp
using Apache.DataFusion;
using Apache.DataFusion.TableProviders.MySql;

using SessionContext context = new();
context.RegisterMySql("Server=localhost;User ID=root;Password=pass;Database=datafusion_sample");

using DataFrame df = context.Sql("""
SELECT *
FROM datafusion_orders
LIMIT 10
""");
df.Show();
```

SQLite uses the same registration shape for a local database file:

```csharp
using Apache.DataFusion;
using Apache.DataFusion.TableProviders.Sqlite;

using SessionContext context = new();
context.RegisterSqlite("Data Source=datafusion_sample.sqlite");

using DataFrame df = context.Sql("""
SELECT *
FROM audit_log
FROM datafusion_orders
LIMIT 10
""");
df.Show();
```

The registered DataFusion table name is the first argument to the extension method. The
provider's `Query` points at the remote database table or subquery. For example, if you
register `audit_log`, query `FROM audit_log` in DataFusion SQL; if you register `orders`,
query `FROM orders`.
Database registration uses each discovered table or collection name as the DataFusion table
name by default. Single-table registration overloads are also available when you only want
to expose one table.

The sample projects use environment variables for connection strings:

| Sample | Environment variable |
| --- | --- |
| `Apache.DataFusion.Sample.MySql` | `DATAFUSION_MYSQL_CONNECTION` |
| `Apache.DataFusion.Sample.PostgreSql` | `DATAFUSION_POSTGRESQL_CONNECTION` |
| `Apache.DataFusion.Sample.Sqlite` | `DATAFUSION_SQLITE_CONNECTION` |
| `Apache.DataFusion.Sample.MongoDB` | `DATAFUSION_MONGODB_CONNECTION` |
| `Apache.DataFusion.Sample.ClickHouse` | `DATAFUSION_CLICKHOUSE_CONNECTION` |

Expand Down
5 changes: 5 additions & 0 deletions datafusion-csharp.slnx
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,17 @@
<Project Path="src/Apache.DataFusion.TableProviders.ClickHouse/Apache.DataFusion.TableProviders.ClickHouse.csproj" />
<Project Path="src/Apache.DataFusion.TableProviders.MongoDB/Apache.DataFusion.TableProviders.MongoDB.csproj" />
<Project Path="src/Apache.DataFusion.TableProviders.MySql/Apache.DataFusion.TableProviders.MySql.csproj" />
<Project Path="src/Apache.DataFusion.TableProviders.PostgreSql/Apache.DataFusion.TableProviders.PostgreSql.csproj" />
<Project Path="src/Apache.DataFusion.TableProviders.Sqlite/Apache.DataFusion.TableProviders.Sqlite.csproj" />
<Project Path="src\Apache.DataFusion\Apache.DataFusion.csproj" />
</Folder>
<Folder Name="/examples/">
<Project Path="examples/Apache.DataFusion.Sample.ClickHouse/Apache.DataFusion.Sample.ClickHouse.csproj" />
<Project Path="examples/Apache.DataFusion.Sample.FederatedQuery/Apache.DataFusion.Sample.FederatedQuery.csproj" />
<Project Path="examples/Apache.DataFusion.Sample.MongoDB/Apache.DataFusion.Sample.MongoDB.csproj" />
<Project Path="examples/Apache.DataFusion.Sample.MySql/Apache.DataFusion.Sample.MySql.csproj" />
<Project Path="examples/Apache.DataFusion.Sample.PostgreSql/Apache.DataFusion.Sample.PostgreSql.csproj" />
<Project Path="examples/Apache.DataFusion.Sample.Sqlite/Apache.DataFusion.Sample.Sqlite.csproj" />
<Project Path="examples\Apache.DataFusion.Sample.Aggregate\Apache.DataFusion.Sample.Aggregate.csproj" />
<Project Path="examples\Apache.DataFusion.Sample.DataFrame\Apache.DataFusion.Sample.DataFrame.csproj" />
<Project Path="examples\Apache.DataFusion.Sample.Excel\Apache.DataFusion.Sample.Excel.csproj" />
Expand Down
8 changes: 2 additions & 6 deletions examples/Apache.DataFusion.Sample.ClickHouse/Program.cs
Original file line number Diff line number Diff line change
Expand Up @@ -52,16 +52,12 @@ INSERT INTO datafusion_orders (id, customer, total) VALUES
}

using SessionContext context = new();
context.RegisterClickHouse("orders", new ClickHouseTableOptions
{
ConnectionString = connectionString,
Query = """SELECT id, customer, total FROM datafusion_orders""",
});
context.RegisterClickHouse(connectionString);

Console.WriteLine("Customer spend from a ClickHouse-backed streaming table:");
using DataFrame df = context.Sql("""
SELECT customer, SUM(total) AS spend
FROM orders
FROM datafusion_orders
WHERE total >= 10
GROUP BY customer
ORDER BY customer
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>

<ItemGroup>
<ProjectReference Include="..\..\src\Apache.DataFusion\Apache.DataFusion.csproj" />
<ProjectReference Include="..\..\src\Apache.DataFusion.TableProviders.Sqlite\Apache.DataFusion.TableProviders.Sqlite.csproj" />
</ItemGroup>

<ItemGroup>
<PackageReference Include="Microsoft.Data.Sqlite" Version="10.0.9" />
</ItemGroup>

</Project>
91 changes: 91 additions & 0 deletions examples/Apache.DataFusion.Sample.FederatedQuery/Program.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

using Apache.DataFusion;
using Apache.DataFusion.TableProviders.Sqlite;
using Microsoft.Data.Sqlite;

string sqliteConnectionString = CreateSqliteOrders();
string customerRegionsPath = WriteCustomerRegionsCsv();

using SessionContext context = new();
context.RegisterSqlite(sqliteConnectionString);
context.RegisterCsv("customer_regions", customerRegionsPath, new CsvReadOptions { HasHeader = true });

Console.WriteLine("Federated query joining SQLite orders with CSV customer regions:");
using DataFrame df = context.Sql("""
SELECT r.region, o.customer, SUM(o.total) AS spend
FROM datafusion_orders AS o
INNER JOIN customer_regions AS r
ON o.customer = r.customer
GROUP BY r.region, o.customer
ORDER BY r.region, o.customer
""");
df.Show();

static string CreateSqliteOrders()
{
string directory = WorkDirectory();
string databasePath = Path.Combine(directory, "orders.sqlite");
string connectionString = new SqliteConnectionStringBuilder
{
DataSource = databasePath,
}.ToString();

using SqliteConnection connection = new(connectionString);
connection.Open();
using SqliteCommand command = connection.CreateCommand();
command.CommandText = """
CREATE TABLE IF NOT EXISTS datafusion_orders (
id INTEGER PRIMARY KEY,
customer TEXT NOT NULL,
total REAL NOT NULL
);

DELETE FROM datafusion_orders;

INSERT INTO datafusion_orders (id, customer, total) VALUES
(1, 'alice', 19.99),
(2, 'bob', 7.50),
(3, 'alice', 100.00),
(4, 'carol', 42.25);
""";
command.ExecuteNonQuery();

return connectionString;
}

static string WriteCustomerRegionsCsv()
{
string path = Path.Combine(WorkDirectory(), "customer_regions.csv");
File.WriteAllText(
path,
"""
customer,region
alice,west
bob,east
carol,central
""");
return path;
}

static string WorkDirectory()
{
string directory = Path.Combine(Path.GetTempPath(), "datafusion-csharp-examples", "federated-query");
Directory.CreateDirectory(directory);
return directory;
}
3 changes: 1 addition & 2 deletions examples/Apache.DataFusion.Sample.MongoDB/Program.cs
Original file line number Diff line number Diff line change
Expand Up @@ -68,11 +68,10 @@
]);

using SessionContext context = new();
context.RegisterMongoDb("orders", new MongoDbTableOptions
context.RegisterMongoDb(new MongoDbDatabaseOptions
{
ConnectionString = connectionString,
DatabaseName = databaseName,
CollectionName = collectionName,
});

Console.WriteLine("Customer spend from a MongoDB-backed streaming table:");
Expand Down
10 changes: 3 additions & 7 deletions examples/Apache.DataFusion.Sample.MySql/Program.cs
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@
// specific language governing permissions and limitations
// under the License.

using MySqlConnector;
using Apache.DataFusion;
using Apache.DataFusion.TableProviders.MySql;
using MySqlConnector;

string? connectionString = Environment.GetEnvironmentVariable("DATAFUSION_MYSQL_CONNECTION");
if (string.IsNullOrWhiteSpace(connectionString))
Expand Down Expand Up @@ -51,16 +51,12 @@ INSERT INTO datafusion_orders (id, customer, total) VALUES
}

using SessionContext context = new();
context.RegisterMySql("orders", new MySqlTableOptions
{
ConnectionString = connectionString,
Query = """SELECT id, customer, total FROM datafusion_orders""",
});
context.RegisterMySql(connectionString);

Console.WriteLine("Customer spend from a MySQL-backed streaming table:");
using DataFrame df = context.Sql("""
SELECT customer, SUM(total) AS spend
FROM orders
FROM datafusion_orders
GROUP BY customer
ORDER BY customer
""");
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>

<ItemGroup>
<ProjectReference Include="..\..\src\Apache.DataFusion\Apache.DataFusion.csproj" />
<ProjectReference Include="..\..\src\Apache.DataFusion.TableProviders.PostgreSql\Apache.DataFusion.TableProviders.PostgreSql.csproj" />
</ItemGroup>

<ItemGroup>
<PackageReference Include="Npgsql" Version="10.0.3" />
</ItemGroup>

</Project>
63 changes: 63 additions & 0 deletions examples/Apache.DataFusion.Sample.PostgreSql/Program.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

using Apache.DataFusion;
using Apache.DataFusion.TableProviders.PostgreSql;
using Npgsql;

string? connectionString = Environment.GetEnvironmentVariable("DATAFUSION_POSTGRESQL_CONNECTION");
if (string.IsNullOrWhiteSpace(connectionString))
{
Console.WriteLine("Set DATAFUSION_POSTGRESQL_CONNECTION to run this sample.");
Console.WriteLine("Example:");
Console.WriteLine(" Host=localhost;Username=postgres;Password=pass;Database=datafusion_sample");
return;
}

using (NpgsqlConnection connection = new(connectionString))
{
connection.Open();
using NpgsqlCommand command = connection.CreateCommand();
command.CommandText = """
CREATE TABLE IF NOT EXISTS datafusion_orders (
id INTEGER PRIMARY KEY,
customer VARCHAR(64) NOT NULL,
total DOUBLE PRECISION NOT NULL
);

DELETE FROM datafusion_orders;

INSERT INTO datafusion_orders (id, customer, total) VALUES
(1, 'alice', 19.99),
(2, 'bob', 7.50),
(3, 'alice', 100.00),
(4, 'carol', 42.25);
""";
command.ExecuteNonQuery();
}

using SessionContext context = new();
context.RegisterPostgreSql(connectionString);

Console.WriteLine("Customer spend from a PostgreSQL-backed streaming table:");
using DataFrame df = context.Sql("""
SELECT customer, SUM(total) AS spend
FROM datafusion_orders
GROUP BY customer
ORDER BY customer
""");
df.Show();
Loading
Loading