clarify none and missing in docs

book/src/SUMMARY.md

@@ -61,6 +61,7 @@
         - [Errors](super-sql/types/error.md)
         - [Named Types](super-sql/types/named.md)
         - [Nulls](super-sql/types/null.md)
+        - [None and Missing](super-sql/types/none.md)
     - [Operators](super-sql/operators/intro.md)
         - [aggregate](super-sql/operators/aggregate.md)
         - [assert](super-sql/operators/assert.md)

book/src/formats/model.md

@@ -77,16 +77,22 @@ type `T` or null is a [union](#25-union) of `T` and `null`,
 e.g., a nullable string has type `string|null`.
 
 The _missing_ type is a primitive type whose only value is `missing`,
-which represents the absence of a value, e.g., in a record type formed by
-[type fusion](../super-sql/type-fusion.md), the value of
-a field that is absent from the original record.
-
-The _none_ type is a primitive type whose only value is `none`.  Its
-primary purpose is to be cast to an [option](#29-option) type, yielding
-that option's "none" value.  It is also the element type of an empty array
+which marks a field that is absent from some of the values described by
+one type, e.g., a field of a record type formed by
+[type fusion](../super-sql/type-fusion.md) that is absent from some of
+the original records.
+
+The _none_ type is a primitive type whose only value is `none`, which
+means there is no value, e.g., the result of
+[ok](../super-sql/functions/generics/ok.md) applied to an error.
+Cast to an [option](#29-option) type, `none` yields that option's "none"
+value.  It is also the element type of an empty array
 or set and the key and value type of an empty map whose types are not
 otherwise known.
 
+See [None and Missing](../super-sql/types/none.md) for how SuperSQL
+treats these two values.
+
 ### 1.2 All and Any
 
 The _any_ type is the top type of the type system: every type conforms

book/src/super-sql/expressions/comparisons.md

@@ -103,12 +103,10 @@ values {isNull:this is null,isNotNull:this is not null}
 1
 null
 2
-missing
 # expected output
 {isNull:false,isNotNull:true}
 {isNull:true,isNotNull:false}
 {isNull:false,isNotNull:true}
-{isNull:false,isNotNull:true}
 ```
 
 ---

book/src/super-sql/types/error.md

@@ -146,6 +146,17 @@ values x
 error("no such field x")
 ```
 
+SuperSQL does have a type called `missing`, but it plays a different role.
+When values of different shapes are described by one type, e.g., after
+[type fusion](../type-fusion.md), the type must say which fields are absent
+from some of the values.  A field of type `int64|missing`, for instance,
+is an `int64` in the records that include the field and absent from records
+that do not include it.
+So `missing` describes the shape of data rather than being a value that a
+query produces: reading such a field where it's absent is still an error,
+as above.
+See [None and Missing](none.md) for how `missing` relates to `none`.
+
 ## Examples
 
 ---

book/src/super-sql/types/intro.md

@@ -8,6 +8,7 @@ comprising
 [sum types](union.md),
 [named types](named.md),
 the [null type](null.md),
+the [none and missing types](none.md),
 and first-class
 [errors](error.md) and [types](type.md).
 

book/src/super-sql/types/none.md

@@ -0,0 +1,130 @@
+# None and Missing
+
+SuperSQL has two types for "nothing here," each with just one value:
+* `none`, a value meaning that there is no value, and
+* `missing`, which marks a field that's absent from some of the values
+  described by one type.
+
+They answer different questions.  A `none` says _the answer is nothing_,
+while `missing` says _the field isn't there_.  You compute with `none`.
+You don't compute with `missing`: SuperSQL reports it so you can say
+what you want, usually by converting it to `none` with
+[ok](../functions/generics/ok.md).
+
+## None
+
+A plain `none` is the result of an expression that produces nothing, e.g.,
+* [ok](../functions/generics/ok.md) of an error,
+* `?.` applied to `none`,
+* a [type assertion](../expressions/type-assertions.md) that fails,
+* a [match](../expressions/match.md) with no matching case and no `else`,
+  or
+* a pipe subquery that produces no values.
+
+A `none` is handled quietly rather than reported:
+* it's dropped from the output of the [values](../operators/values.md)
+  operator, so it's never passed to a downstream operator,
+* it disappears from a record expression, an array, a set, or a map,
+* [aggregate functions](../aggregates/intro.md) skip it,
+* `??` replaces it with another value, and
+* `?.` passes it through a field access.
+
+For example,
+```mdtest-spq
+# spq
+values {a:this, b:ok(10/this)}, ok(10/this) ?? "bad value"
+# input
+2
+0
+# expected output
+{a:2,b:5}
+5
+{a:0}
+"bad value"
+```
+
+The `none` of an [option type](../../formats/model.md#29-option), e.g.,
+`none::int64`, isn't a plain `none`.  It's a value of its option type, and does
+it does not disappear from record fields or from array/set/map values like
+a plain `none` does.
+
+## Missing
+
+A `missing` never comes from an expression.  It appears only in data whose
+different shapes are described by one type, e.g., the records produced by
+the [fuse](../operators/fuse.md) operator:
+```mdtest-spq
+# spq
+fuse
+# input
+{a:1}
+{b:"x"}
+# expected output
+{a:1::(int64|missing),b:missing::(string|missing)}
+{a:missing::(int64|missing),b:"x"::(string|missing)}
+```
+Here, the type `int64|missing` says that `a` is absent from some of the
+values, and each record holds `missing` for the field it doesn't have.
+
+Reading a field where it's absent is an [error](error.md), just as it is for
+a field that doesn't exist at all:
+```mdtest-spq
+# spq
+fuse | values b
+# input
+{a:1}
+{b:"x"}
+# expected output
+error("no such field b")
+"x"::(string|missing)
+```
+
+When the types of a query's input are known, a field that may be absent is
+reported before the query runs:
+```mdtest-spq fails
+# spq
+values {a:1},{b:2} | values a ?? 0
+# input
+
+# expected output
+field may be missing (consider ok()) at line 1, column 29:
+values {a:1},{b:2} | values a ?? 0
+                            ~
+```
+Note that `??` and `?.` handle `none`, not `missing`.  To handle a field
+that may be absent, convert it to `none` with
+[ok](../functions/generics/ok.md), then drop the `none` or replace it with
+`??`:
+```mdtest-spq
+# spq
+values {a:1},{b:2} | values {input:this, ok:a.ok(), with_default:a.ok() ?? 0}
+# input
+
+# expected output
+{input:{a:1},ok:1,with_default:1}
+{input:{b:2},with_default:0}
+```
+
+## Why Two Types?
+
+A `none` from `ok()` or a failed type assertion is something the query asked
+for, so it's safe to drop it or replace it without comment.
+
+A field that may be absent is different.  It usually means the data doesn't
+look the way the query expects, e.g., the field name is misspelled, or only
+some of the input has the field.  If an absent field quietly became `none`,
+such mistakes would silently drop values from the output.
+
+Keeping the two apart lets SuperSQL drop `none` quietly while still catching
+every read of a field that may not be there.  Writing `ok()` makes the choice
+explicit: where the field is absent, treat it as nothing.
+
+This resembles the distinction between `null` and `missing` in
+[SQL++](https://asterixdb.apache.org/docs/0.9.7.1/sqlpp/manual.html), but in SuperSQL a field that isn't
+there is an error rather than a value (see [Errors](error.md#missing)),
+and `missing` only marks such fields in the types and values of data.
+
+>[!NOTE]
+> The [SUP](../../formats/sup.md#26-null-missing-and-none-values) format spells
+> the missing value `missing` so that data with absent fields can be written
+> and read, but `missing` isn't intended for use in queries.