Summary
When it comes to representing large virtual tables with substrait-explain, it can sometimes get a little visually cluttered when everything has to be displayed on a single line. If we implemented a format to allow for every row of data to have it's own line, it'd be much easier to read.
Motivation
By having every row of data have it's own line instead of a single massive line, it makes things much easier to read
Proposed Solution
When converting from substrait-explain to protobuf, instead of parsing the entire massive table, we parse the rows line by line. And when we convert from protobuf to substrait-explain, we print the rows within the virtual tables line by line.
Additional Context
Going from this:
== Query
=== Plan
Root[id, name, age]
Sort[($1, &DescNullsFirst) => $0, $1, $2]
Read:Virtual[(1:i32, 'Alice':string?, 25:i32), (15:i32, 'Bob':string?, 30:i32), (5:i32, null:string?, 20:i32) => id:i32, name:string?, age:i32]
To this:
== Query
=== Plan
Root[id, name, age]
Sort[($1, &DescNullsFirst) => $0, $1, $2]
Read:Virtual[
(1:i32, 'Alice':string?, 25:i32),
(15:i32, 'Bob':string?, 30:i32),
(5:i32, null:string?, 20:i32)
=> id:i32, name:string?, age:i32]
Is much more readable.
There are several formats that can be done though:
== Query
=== Plan
Root[id, name, age]
Sort[($1, &DescNullsFirst) => $0, $1, $2]
Read:Virtual[
(1:i32, 'Alice':string?, 25:i32),
(15:i32, 'Bob':string?, 30:i32),
(5:i32, null:string?, 20:i32)
=> id:i32, name:string?, age:i32]
Summary
When it comes to representing large virtual tables with substrait-explain, it can sometimes get a little visually cluttered when everything has to be displayed on a single line. If we implemented a format to allow for every row of data to have it's own line, it'd be much easier to read.
Motivation
By having every row of data have it's own line instead of a single massive line, it makes things much easier to read
Proposed Solution
When converting from substrait-explain to protobuf, instead of parsing the entire massive table, we parse the rows line by line. And when we convert from protobuf to substrait-explain, we print the rows within the virtual tables line by line.
Additional Context
Going from this:
To this:
Is much more readable.
There are several formats that can be done though: