Skip to content

add materialized subqueries #7640

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nwt wants to merge 1 commit into
base: main
Choose a base branch
from
+5,303 −4,718
Open

add materialized subqueries #7640

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 21 additions & 1 deletion book/src/super-sql/declarations/queries.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

A query may be bound to an identifier as a named query with the syntax
```
let <name> = ( <query> )
let <name> = [ materialized ] ( <query> )
```
where `<name>` is an [identifier](../queries.md#identifiers)
and `<query>` is any standalone [query](../queries.md) that sources its own input.
Expand All @@ -27,6 +27,11 @@ an array subquery.

To create a query that can operate on its input, declare an [operator](operators.md).

With `materialized`, the query is evaluated once, its output is cached,

@mccanne mccanne Nov 11, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would leave out the "cache" concept as that is mechanism leaking into the semantic definition here

@nwt nwt Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. And I just noticed that the next paragraph already includes an explanation that avoids mechanism in the context of automatic materialization. I'm going to integrate materialize better with that since the current addition feels disjointed.

and every reference to the query produces that same output. This is
useful when a query is expensive or has results that vary across
evaluations.

A common use case for a named query is to compute a complex query that returns a scalar,
then embedding that scalar result in an [expression](../expressions/intro.md). Even though the named query
appears syntactically as a subquery in this case, the result is efficient
Expand All @@ -50,6 +55,21 @@ from hello

---

_A materialized named query produces the same output each time it is referenced_

```mdtest-spq
# spq
let t = materialized (values now())
values t, t, t
| count(distinct this)
# input
# expected output
1
```

---

_Use an array subquery if multiple values expected_

```mdtest-spq
Expand Down
21 changes: 21 additions & 0 deletions book/src/super-sql/expressions/subqueries.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,27 @@ Then, for each input value `3` and `4`, the result is emitted, e.g.,
{that:4,count:3}
```

A subquery may also be made independent explicitly by placing the keyword
`materialized` before its parentheses, as in
```
materialized ( <query> )
```
A materialized subquery is evaluated just once and receives a single `null`

@mccanne mccanne Nov 11, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"is evaluated just once and returns the initial result on each subsequent invocation"

... "a materialized subquery has no access to the expression context in which it is evaluated... the body of the subquery has no parent operator and like all such queries, its input is a single null value"

^^^ I'm not proposing this exact wording... just this direction

value as its input, so it need not begin with a source. For example,
the subquery `materialized (values now())` is evaluated once even though
it does not begin with a `from` operator.

```mdtest-spq
# spq
values 1,2,3
| values materialized (values now())
| count(distinct this)
# input
# expected output
1
```

## Correlated Subqueries

When a subquery appears within a [SQL operator](../sql/intro.md),
Expand Down
26 changes: 24 additions & 2 deletions book/src/super-sql/sql/with.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@
A [WITH](with.md) clause may precede any
[SQL operator](intro.md#sql-operator) and has the form
```
WITH <alias> AS (
WITH <alias> AS [ [ NOT ] MATERIALIZED ] (
<sql-op>
)
[ , <alias> AS ( <sql-op> ) ... ]
[ , <alias> AS [ [ NOT ] MATERIALIZED ] ( <sql-op> ) ... ]
```
where
* `<alias>` is a table alias with optional columns as defined
Expand All @@ -21,6 +21,10 @@ but the CTE body must be a [SQL operator](intro.md#sql-operator)
and the CTE name can be used only with a [FROM](from.md) clause
and is not accessible in an expression.

By default, or when `NOT MATERIALIZED` is given, the body of a CTE is evaluated
each time the CTE is referenced. When `MATERIALIZED` is given, the body is
evaluated at most once and every reference to the CTE sees the same result.

The table aliases form a lexical scope
which is available in any `FROM` clause defined within the SQL operator
that follows the `WITH` clause and any `FROM` clauses recursively
Expand Down Expand Up @@ -49,6 +53,24 @@ SELECT * FROM hello

---

_A materialized CTE is evaluated once_
```mdtest-spq
# spq
WITH T AS MATERIALIZED (
SELECT now() AS x
)
SELECT x FROM T
UNION ALL
SELECT x FROM T
| count(distinct this)
# input

# expected output
1
```

---

_A first CTE referenced in a second CTE_
```mdtest-spq
# spq
Expand Down
9 changes: 5 additions & 4 deletions compiler/ast/decl.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,11 @@ type PragmaDecl struct {
}

type QueryDecl struct {
Kind string `json:"kind" unpack:""`
Name *ID `json:"name"`
Body Seq `json:"body"`
Loc `json:"loc"`
Kind string `json:"kind" unpack:""`
Name *ID `json:"name"`
Materialized bool `json:"materialized"`
Body Seq `json:"body"`
Loc `json:"loc"`
}

type TypeDecl struct {
Expand Down
9 changes: 5 additions & 4 deletions compiler/ast/expr.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,10 +195,11 @@ type (
Loc `json:"loc"`
}
SubqueryExpr struct {
Kind string `json:"kind" unpack:""`
Body Seq `json:"body"`
Array bool `json:"array"`
Loc `json:"loc"`
Kind string `json:"kind" unpack:""`
Materialized bool `json:"materialized"`
Body Seq `json:"body"`
Array bool `json:"array"`
Loc `json:"loc"`
}
SubstringExpr struct {
Kind string `json:"kind" unpack:""`
Expand Down
11 changes: 6 additions & 5 deletions compiler/dag/expr.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,10 @@ import (
)

type MainExpr struct {
Types []byte `json:"types"`
Funcs []*FuncDef `json:"funcs"`
Expr Expr `json:"expr"`
Types []byte `json:"types"`
Funcs []*FuncDef `json:"funcs"`
Materialized []*MaterializedDef `json:"materialized"`
Expr Expr `json:"expr"`
}

type (
Expand Down Expand Up @@ -144,8 +145,8 @@ type (
Nulls order.Nulls `json:"nulls"`
}
SubqueryExpr struct {
Kind string `json:"kind" unpack:""`
Correlated bool `json:"correlated"`
Kind string `json:"kind" unpack:""`
Materialized bool `json:"materialized"`
// SQL is true for a subquery in a SQL expression, whose value is
// null when the subquery produces no values. Otherwise, the value
// is none.
Expand Down
76 changes: 44 additions & 32 deletions compiler/dag/op.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,10 @@ import (
)

type Main struct {
Types []byte `json:"types"`
Funcs []*FuncDef `json:"funcs"`
Body Seq `json:"body"`
Types []byte `json:"types"`
Funcs []*FuncDef `json:"funcs"`
Materialized []*MaterializedDef `json:"materialized"`
Body Seq `json:"body"`
}

type FuncDef struct {
Expand All @@ -31,6 +32,12 @@ type FuncDef struct {
Expr Expr `json:"expr"`
}

type MaterializedDef struct {
Kind string `json:"kind" unpack:""`
Name string `json:"name"`

@mccanne mccanne Nov 11, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why names? They don't appear in the user facing model. Like functions, it seems like they should be a slice that is ref'd by slot in the slice.

@mccanne mccanne Nov 11, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm maybe the names relate to the querydecl name? They can't live in a flat table because they are lexically scoped.

Body Seq `json:"body"`
}

type Op interface {
opNode()
}
Expand Down Expand Up @@ -124,6 +131,10 @@ type (
Message string `json:"message"`
Meta string `json:"meta"`
}
MaterializedRefOp struct {
Kind string `json:"kind" unpack:""`
Name string `json:"name"`

@mccanne mccanne Nov 11, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Name -> Slot ?
(see other comment)

}
MergeOp struct {
Kind string `json:"kind" unpack:""`
Exprs []SortExpr `json:"exprs"`
Expand Down Expand Up @@ -206,35 +217,36 @@ type (
}
)

func (*AggregateOp) opNode() {}
func (*CombineOp) opNode() {}
func (*CountOp) opNode() {}
func (*DebugOp) opNode() {}
func (*DistinctOp) opNode() {}
func (*DropOp) opNode() {}
func (*FilterOp) opNode() {}
func (*ForkOp) opNode() {}
func (*FuseOp) opNode() {}
func (*HashJoinOp) opNode() {}
func (*HeadOp) opNode() {}
func (*InferOp) opNode() {}
func (*JoinOp) opNode() {}
func (*LoadOp) opNode() {}
func (*MergeOp) opNode() {}
func (*OutputOp) opNode() {}
func (*PassOp) opNode() {}
func (*PutOp) opNode() {}
func (*RenameOp) opNode() {}
func (*ScatterOp) opNode() {}
func (*SkipOp) opNode() {}
func (*SlicerOp) opNode() {}
func (*SortOp) opNode() {}
func (*SwitchOp) opNode() {}
func (*TailOp) opNode() {}
func (*TopOp) opNode() {}
func (*UniqOp) opNode() {}
func (*UnnestOp) opNode() {}
func (*ValuesOp) opNode() {}
func (*AggregateOp) opNode() {}
func (*CombineOp) opNode() {}
func (*CountOp) opNode() {}
func (*DebugOp) opNode() {}
func (*DistinctOp) opNode() {}
func (*DropOp) opNode() {}
func (*FilterOp) opNode() {}
func (*ForkOp) opNode() {}
func (*FuseOp) opNode() {}
func (*HashJoinOp) opNode() {}
func (*HeadOp) opNode() {}
func (*InferOp) opNode() {}
func (*JoinOp) opNode() {}
func (*LoadOp) opNode() {}
func (*MaterializedRefOp) opNode() {}
func (*MergeOp) opNode() {}
func (*OutputOp) opNode() {}
func (*PassOp) opNode() {}
func (*PutOp) opNode() {}
func (*RenameOp) opNode() {}
func (*ScatterOp) opNode() {}
func (*SkipOp) opNode() {}
func (*SlicerOp) opNode() {}
func (*SortOp) opNode() {}
func (*SwitchOp) opNode() {}
func (*TailOp) opNode() {}
func (*TopOp) opNode() {}
func (*UniqOp) opNode() {}
func (*UnnestOp) opNode() {}
func (*ValuesOp) opNode() {}

// Scanner sources also implement Op and all have suffix "Scan".
type (
Expand Down
2 changes: 2 additions & 0 deletions compiler/dag/unpack.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ var unpacker = unpack.New(
LoadOp{},
MapCallExpr{},
MapExpr{},
MaterializedDef{},
MaterializedRefOp{},
MergeOp{},
NullScan{},
OutputOp{},
Expand Down
16 changes: 13 additions & 3 deletions compiler/describe/analyze.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,11 @@ func AnalyzeDAG(ctx context.Context, main *dag.Main, src *exec.Environment) (*In
entry := main.Body
var err error
var info Info
if info.Sources, err = describeSources(ctx, src.DB(), entry[0]); err != nil {
materialized := make(map[string]*dag.MaterializedDef)
for _, m := range main.Materialized {
materialized[m.Name] = m
}
if info.Sources, err = describeSources(ctx, src.DB(), materialized, entry[0]); err != nil {
return nil, err
}
sortKeys, err := optimizer.New(ctx, src).SortKeys(entry)
Expand Down Expand Up @@ -100,18 +104,24 @@ func AnalyzeDAG(ctx context.Context, main *dag.Main, src *exec.Environment) (*In
return &info, nil
}

func describeSources(ctx context.Context, root *db.Root, o dag.Op) ([]Source, error) {
func describeSources(ctx context.Context, root *db.Root, materialized map[string]*dag.MaterializedDef, o dag.Op) ([]Source, error) {
switch o := o.(type) {
case *dag.ForkOp:
var s []Source
for _, p := range o.Paths {
out, err := describeSources(ctx, root, p[0])
out, err := describeSources(ctx, root, materialized, p[0])
if err != nil {
return nil, err
}
s = append(s, out...)
}
return s, nil
case *dag.MaterializedRefOp:
def, ok := materialized[o.Name]
if !ok || len(def.Body) == 0 {
return nil, fmt.Errorf("internal error: undefined materialized query %q", o.Name)
}
return describeSources(ctx, root, materialized, def.Body[0])
case *dag.NullScan:
return []Source{&Null{Kind: "Null"}}, nil
case *dag.FileScan:
Expand Down
2 changes: 1 addition & 1 deletion compiler/optimizer/demand.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ func (a *demandAnalyzer) demandForSimpleOp(op dag.Op, downstream demand.Demand)
a.demandForExpr(df.Expr)
}
return demand.None()
case *dag.HTTPScan, *dag.ListerScan, *dag.NullScan, *dag.TableMetaScan, *dag.TableScan:
case *dag.HTTPScan, *dag.ListerScan, *dag.MaterializedRefOp, *dag.NullScan, *dag.TableMetaScan, *dag.TableScan:
return demand.None()
case *dag.RobotScan:
return demand.Union(a.demandForExpr(op.Expr), a.demandForExpr(op.Method), a.demandForExpr(op.Headers), a.demandForExpr(op.Body))
Expand Down
22 changes: 18 additions & 4 deletions compiler/optimizer/optimizer.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,22 @@ func walkEntries(seq dag.Seq, post func(dag.Seq) (dag.Seq, error)) (dag.Seq, err
// source's pushdown predicate. This should be called before ParallelizeScan().
// TBD: we need to do pushdown for search/cut to optimize columnar extraction.
func (o *Optimizer) Optimize(main *dag.Main) error {
seq := main.Body
for _, m := range main.Materialized {
seq, err := o.optimizeSeq(m.Body)
if err != nil {
return err
}
m.Body = seq
}
seq, err := o.optimizeSeq(main.Body)
if err != nil {
return err
}
main.Body = seq
return nil
}

func (o *Optimizer) optimizeSeq(seq dag.Seq) (dag.Seq, error) {
seq = liftFilterOps(seq)
seq = mergeFilters(seq)
seq = mergeValuesOps(seq)
Expand All @@ -135,7 +150,7 @@ func (o *Optimizer) Optimize(main *dag.Main) error {
seq = mergeFilters(seq)
seq, err := o.optimizeSourcePaths(seq)
if err != nil {
return err
return nil, err
}
seq = removePassOps(seq)
table := analyzeDemand(seq)
Expand All @@ -144,8 +159,7 @@ func (o *Optimizer) Optimize(main *dag.Main) error {
}
setProjections(table)
setPushdownUnordered(seq, false)
main.Body = seq
return nil
return seq, nil
}

func (o *Optimizer) OptimizeDeleter(main *dag.Main, replicas int) error {
Expand Down
Loading
Loading