Skip to content

Commit c8018e4

Browse files
committed
docs: add user guide page on building plans via datafusion-proto
1 parent a691900 commit c8018e4

2 files changed

Lines changed: 202 additions & 0 deletions

File tree

‎docs/source/user-guide/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ quickstart
3636
sessioncontext
3737
dataframe
3838
parquet
39+
proto-plans
3940
project-status
4041
```
4142

Lines changed: 201 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,201 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
# Logical plans via datafusion-proto
21+
22+
`SessionContext.fromProto(byte[])` accepts a serialized DataFusion
23+
`LogicalPlanNode` and returns a lazy `DataFrame`. This is useful when you
24+
already have a plan produced by another DataFusion-aware tool, or when
25+
you want to construct the plan programmatically with finer-grained
26+
control than the `sql` or DataFrame APIs.
27+
28+
The protobuf Java classes are generated by the build into the
29+
`org.apache.datafusion.protobuf` (plan and expression nodes) and
30+
`datafusion_common` (scalar values, schema, column references, file
31+
formats) packages. The Maven build downloads pinned `.proto` files from
32+
the matching upstream DataFusion tag on first build, then generates the
33+
Java classes locally — see the
34+
[Contributor Guide](../contributor-guide/updating-datafusion-version.md)
35+
for how to bump the version.
36+
37+
## A minimal plan
38+
39+
The smallest interesting plan is a projection of a literal over an
40+
empty input. It is useful as a sanity check and exercises serialization
41+
end-to-end without touching any storage.
42+
43+
```java
44+
import org.apache.arrow.memory.RootAllocator;
45+
import org.apache.arrow.vector.IntVector;
46+
import org.apache.arrow.vector.VectorSchemaRoot;
47+
import org.apache.arrow.vector.ipc.ArrowReader;
48+
import org.apache.datafusion.DataFrame;
49+
import org.apache.datafusion.SessionContext;
50+
import org.apache.datafusion.protobuf.EmptyRelationNode;
51+
import org.apache.datafusion.protobuf.LogicalExprNode;
52+
import org.apache.datafusion.protobuf.LogicalPlanNode;
53+
import org.apache.datafusion.protobuf.ProjectionNode;
54+
55+
import datafusion_common.DatafusionCommon;
56+
57+
LogicalPlanNode plan =
58+
LogicalPlanNode.newBuilder()
59+
.setProjection(
60+
ProjectionNode.newBuilder()
61+
.setInput(
62+
LogicalPlanNode.newBuilder()
63+
.setEmptyRelation(
64+
EmptyRelationNode.newBuilder().setProduceOneRow(true).build())
65+
.build())
66+
.addExpr(
67+
LogicalExprNode.newBuilder()
68+
.setLiteral(
69+
DatafusionCommon.ScalarValue.newBuilder().setInt32Value(1).build())
70+
.build())
71+
.build())
72+
.build();
73+
74+
try (var allocator = new RootAllocator();
75+
SessionContext ctx = new SessionContext();
76+
DataFrame df = ctx.fromProto(plan.toByteArray());
77+
ArrowReader reader = df.collect(allocator)) {
78+
reader.loadNextBatch();
79+
VectorSchemaRoot batch = reader.getVectorSchemaRoot();
80+
IntVector col = (IntVector) batch.getVector(0);
81+
System.out.println(col.get(0)); // 1
82+
}
83+
```
84+
85+
`fromProto` performs the same logical-planning, optimization, and
86+
physical-planning pipeline as `sql`; the result is a lazy
87+
[`DataFrame`](dataframe.md) that only executes when you pull results.
88+
89+
## Scanning a Parquet file via ListingTableScanNode
90+
91+
A `ListingTableScanNode` reads one or more files of the same format
92+
from disk. Unlike `registerParquet`, it does not require the table to
93+
be in the catalog — the scan node carries everything DataFusion needs:
94+
the file paths, the schema, the projection, the file format, and the
95+
target partition count.
96+
97+
The scan node's `schema` field is a `datafusion_common.Schema`, not an
98+
Arrow `Schema`. Convert between the two with the helper in
99+
`org.apache.datafusion.proto.SchemaConverter`:
100+
101+
```java
102+
import org.apache.arrow.vector.types.pojo.Schema;
103+
import org.apache.datafusion.proto.SchemaConverter;
104+
105+
Schema arrow = ctx.tableSchema("lineitem");
106+
DatafusionCommon.Schema schemaProto = SchemaConverter.toProto(arrow);
107+
```
108+
109+
The full example: register the file once to introspect its schema,
110+
then build a plan that scans the same file, sorts by `l_orderkey`,
111+
and fetches the first row. This is equivalent to
112+
`SELECT l_orderkey FROM lineitem ORDER BY l_orderkey LIMIT 1`.
113+
114+
```java
115+
import org.apache.datafusion.protobuf.BareTableReference;
116+
import org.apache.datafusion.protobuf.ListingTableScanNode;
117+
import org.apache.datafusion.protobuf.ProjectionColumns;
118+
import org.apache.datafusion.protobuf.SortExprNode;
119+
import org.apache.datafusion.protobuf.SortNode;
120+
import org.apache.datafusion.protobuf.TableReference;
121+
122+
String path = "/path/to/lineitem.parquet";
123+
124+
try (var allocator = new RootAllocator();
125+
SessionContext ctx = new SessionContext()) {
126+
127+
ctx.registerParquet("lineitem", path);
128+
DatafusionCommon.Schema schemaProto =
129+
SchemaConverter.toProto(ctx.tableSchema("lineitem"));
130+
131+
LogicalExprNode orderKeyCol =
132+
LogicalExprNode.newBuilder()
133+
.setColumn(DatafusionCommon.Column.newBuilder().setName("l_orderkey").build())
134+
.build();
135+
136+
LogicalPlanNode plan =
137+
LogicalPlanNode.newBuilder()
138+
.setSort(
139+
SortNode.newBuilder()
140+
.setInput(
141+
LogicalPlanNode.newBuilder()
142+
.setListingScan(
143+
ListingTableScanNode.newBuilder()
144+
.setTableName(
145+
TableReference.newBuilder()
146+
.setBare(
147+
BareTableReference.newBuilder()
148+
.setTable("lineitem")
149+
.build())
150+
.build())
151+
.addPaths(path)
152+
.setFileExtension(".parquet")
153+
.setSchema(schemaProto)
154+
.setProjection(
155+
ProjectionColumns.newBuilder()
156+
.addColumns("l_orderkey")
157+
.build())
158+
.setParquet(
159+
DatafusionCommon.ParquetFormat.getDefaultInstance())
160+
.setTargetPartitions(1)
161+
.build())
162+
.build())
163+
.addExpr(
164+
SortExprNode.newBuilder()
165+
.setExpr(orderKeyCol)
166+
.setAsc(true)
167+
.setNullsFirst(false)
168+
.build())
169+
.setFetch(1)
170+
.build())
171+
.build();
172+
173+
try (DataFrame df = ctx.fromProto(plan.toByteArray());
174+
ArrowReader reader = df.collect(allocator)) {
175+
reader.loadNextBatch();
176+
// ...
177+
}
178+
}
179+
```
180+
181+
## When to use proto plans
182+
183+
The `sql` and DataFrame APIs are the right choice for most workloads.
184+
Reach for `fromProto` when you need one of:
185+
186+
- **Cross-tool interop.** Accept plans produced by another
187+
DataFusion-based system (a planner, a scheduler, a query frontend).
188+
- **Programmatic plan construction.** Build the plan node tree directly
189+
instead of going through SQL parsing, useful for tools that compile
190+
their own surface language to DataFusion.
191+
- **Plan persistence.** Serialize a plan to bytes, store or transmit
192+
it, and execute it later — possibly in a different process or on a
193+
different machine.
194+
195+
## Schema conversion support
196+
197+
`SchemaConverter.toProto` and `SchemaConverter.fromProto` support the
198+
primitive Arrow types this project's tests exercise: `Bool`, signed and
199+
unsigned integer types 8 through 64 bits, `Float32`, `Float64`, `Utf8`,
200+
`Utf8View`, `LargeUtf8`, `Date32`, and `Decimal128`. Anything else
201+
raises `UnsupportedOperationException` naming the offending type.

0 commit comments

Comments
 (0)