From 9be758573ec5e8112c347d740ae6640926a2bf79 Mon Sep 17 00:00:00 2001 From: Steven McCanne Date: Fri, 6 Nov 2026 08:06:29 -0800 Subject: [PATCH 1/2] update data model, SUP, and BSUP specs --- book/src/SUMMARY.md | 1 - book/src/command/formats.md | 3 +- book/src/command/input.md | 2 +- book/src/command/output.md | 19 +- book/src/database/api.md | 1 - book/src/database/format.md | 14 +- book/src/database/intro.md | 3 +- book/src/dev/integrations/zeek/types.md | 2 +- book/src/dev/libraries/python.md | 9 +- book/src/formats/bsup.md | 1262 +++++++++++------------ book/src/formats/csup.md | 400 ------- book/src/formats/intro.md | 12 +- book/src/formats/model.md | 266 +++-- book/src/formats/sup.md | 342 +++--- book/src/intro.md | 5 +- book/src/super-sql/types/enum.md | 4 +- book/src/tutorials/jq.md | 4 +- book/src/tutorials/super-structured.md | 11 +- 18 files changed, 1075 insertions(+), 1285 deletions(-) delete mode 100644 book/src/formats/csup.md diff --git a/book/src/SUMMARY.md b/book/src/SUMMARY.md index afc4872425..319d3119e8 100644 --- a/book/src/SUMMARY.md +++ b/book/src/SUMMARY.md @@ -209,7 +209,6 @@ - [Data Model](formats/model.md) - [Super (SUP)](formats/sup.md) - [Super Binary (BSUP)](formats/bsup.md) - - [Super Column (CSUP)](formats/csup.md) !!- [Database](database/intro.md) !! - [API](database/api.md) !! - [Format](database/format.md) diff --git a/book/src/command/formats.md b/book/src/command/formats.md index 550f2851e2..cbde7985fc 100644 --- a/book/src/command/formats.md +++ b/book/src/command/formats.md @@ -6,7 +6,6 @@ The supported [input](input.md) and [output](output.md) formats include the foll |-----------|------|-----------|------------------------------------------| | `arrows` | yes | `.arrows` | [Arrow IPC Stream Format](https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format) | | `bsup` | yes | `.bsup` | [BSUP](../formats/bsup.md) | -| `csup` | yes | `.csup` | [CSUP](../formats/csup.md) | | `csv` | yes | `.csv` | [Comma-Separated Values (RFC 4180)](https://www.rfc-editor.org/rfc/rfc4180.html) | | `json` | yes | `.json`, `.jsonl`, `.ndjson` | [JSON (RFC 8259)](https://www.rfc-editor.org/rfc/rfc8259.html) | | `line` | no | `.text`, `.txt` | n/a | One text value per line | @@ -17,6 +16,6 @@ The supported [input](input.md) and [output](output.md) formats include the foll >[!NOTE] > Best performance is typically achieved when operating on data in binary columnar formats -> such as [CSUP](../formats/csup.md), +> such as [BSUP](../formats/bsup.md), > [Parquet](https://github.com/apache/parquet-format), or > [Arrow](https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format). diff --git a/book/src/command/input.md b/book/src/command/input.md index 115bd7d741..0b67381591 100644 --- a/book/src/command/input.md +++ b/book/src/command/input.md @@ -96,7 +96,7 @@ Note that the `line` format cannot be automatically detected and requires `-i` or `(format line)` for reading. >[!NOTE] -> Parquet and CSUP require a seekable input and cannot be operated upon +> Parquet requires a seekable input and cannot be operated upon > when read on standard input. ## Type Checking diff --git a/book/src/command/output.md b/book/src/command/output.md index 21f8ee13c3..204b16f968 100644 --- a/book/src/command/output.md +++ b/book/src/command/output.md @@ -16,10 +16,6 @@ output format is [SUP](../formats/sup.md). Otherwise, the default format is [BSUP](../formats/bsup.md). These defaults may be overridden with `-f`, `-s`, or `-S`. ->[!NOTE] -> While BSUP is currently default, a forthcoming release will change -> CSUP to default after CSUP supports streaming. - Since SUP is a common format choice for interactive use, the `-s` flag is shorthand for `-f sup`. Also, `-S` is a shortcut for `-f sup` with `-pretty 2` as @@ -33,9 +29,9 @@ is a shortcut for `-f json` and `-J` is a shortcut for pretty-printing JSON. > causes an occasional surprise > (e.g., forgetting `-f` or `-s` in a scripted test that works fine on the > command line but fails in CI). However, this avoids problematic performance where a -> data pipeline deployed to production accidentally uses SUP instead of CSUP. +> data pipeline deployed to production accidentally uses SUP instead of BSUP. > Since `super` gracefully handles any input, this would be hard to detect. -> Alternatively, making CSUP always be default would cause much annoyance when +> Alternatively, making BSUP always be default would cause much annoyance when > binary data is written to the terminal. If no query is specified with `-c`, the inputs are scanned without modification @@ -89,10 +85,17 @@ produces When pretty printing, colorization is enabled by default when writing to a terminal, and can be disabled with `-color false`. +## BSUP Output + +By default, [BSUP](../formats/bsup.md) output is written as column frames, +which let a reader read only the columns it needs. The `-rows` flag writes +row frames instead. The `-framecap` flag sets the target number of values +in each frame (default 10000). + ## Pipeline-friendly Formats -Though they're compressed formats, CSUP and BSUP data are self-describing and -stream-oriented and thus are pipeline friendly. +Though it's a compressed format, BSUP data is self-describing and +stream-oriented and thus is pipeline friendly. Since data is self-describing you can simply take super-structured output of one command and pipe it to the input of another. It doesn't matter if the value diff --git a/book/src/database/api.md b/book/src/database/api.md index 4685117c12..44949ed0a0 100644 --- a/book/src/database/api.md +++ b/book/src/database/api.md @@ -546,7 +546,6 @@ The following table shows the supported MIME types and where they can be used. | ---------------- | --------- | -------- | ------------------------------------- | | Arrow IPC Stream | yes | yes | `application/vnd.apache.arrow.stream` | | BSUP | yes | yes | `application/x-bsup` | -| CSUP | yes | yes | `application/x-csup` | | CSV | yes | yes | `text/csv` | | JSON | yes | yes | `application/json` | | Line | yes | yes | `application/x-line` | diff --git a/book/src/database/format.md b/book/src/database/format.md index 8060172a69..e07653cc66 100644 --- a/book/src/database/format.md +++ b/book/src/database/format.md @@ -76,20 +76,16 @@ Since the object's name is globally unique and the resulting object is immutable, there is no possible write concurrency to manage with respect to a given object. -A data object is composed of the primary data object stored as one or two objects -(for sequence and/or vector layout). - -Data objects may be either in sequence form (i.e., BSUP) or vector form (i.e., CSUP), -or both forms may be present as a query optimizer may choose to use whatever -representation is more efficient. -When both sequence and vector data objects are present, they must contain the same -underlying Super data. +A data object is stored as a single [BSUP](../formats/bsup.md) object, +whose column frames hold the data in vector form and whose +[zone maps](../formats/bsup.md#86-zone-maps) allow a query to skip frames +that can't contain matching values. Immutable objects are named as follows: |object type|name| |-----------|----| -|sequence data|`/data/.bsup`| +|data|`/data/.bsup`| `` is the KSUID of the data object. diff --git a/book/src/database/intro.md b/book/src/database/intro.md index 5f3459db66..006f700fdd 100644 --- a/book/src/database/intro.md +++ b/book/src/database/intro.md @@ -150,8 +150,7 @@ replication easy to support and deploy. The cloud objects that comprise a database, e.g., data objects, commit history, transaction journals, partial aggregations, etc., -are stored as super-structured data, i.e., either as [row-based Super Binary (BSUP)](../formats/bsup.md) -or [Super Columnar (CSUP)](../formats/csup.md). +are stored as super-structured data in [Super Binary (BSUP)](../formats/bsup.md). This makes introspection of the database structure straightforward as many key database data structures can be queried with metadata queries and presented to a client for further processing by downstream tooling. diff --git a/book/src/dev/integrations/zeek/types.md b/book/src/dev/integrations/zeek/types.md index 5e68ff514e..bfbe8cf97f 100644 --- a/book/src/dev/integrations/zeek/types.md +++ b/book/src/dev/integrations/zeek/types.md @@ -3,7 +3,7 @@ As the [super-structured data model](../../../formats/model.md) was in many ways inspired by the [Zeek TSV log format](https://docs.zeek.org/en/current/tutorial/logs.html#zeek-log-formats), SuperDB's rich storage formats ([SUP](../../../formats/sup.md), -[CSUP](../../../formats/csup.md), etc.) maintain comprehensive interoperability +[BSUP](../../../formats/bsup.md), etc.) maintain comprehensive interoperability with Zeek. When Zeek is configured to output its logs in diff --git a/book/src/dev/libraries/python.md b/book/src/dev/libraries/python.md index be743a48ff..45ba7b7e55 100644 --- a/book/src/dev/libraries/python.md +++ b/book/src/dev/libraries/python.md @@ -23,7 +23,7 @@ interacts with the database via the REST API provided by > > The full benefits of super-structured data (including improved performance) > are expected when there is a native Python library implementation of -> [CSUP](../../formats/csup.md). Stay tuned. +> [BSUP](../../formats/bsup.md). Stay tuned. ## Installation @@ -185,17 +185,16 @@ by inspecting the beginning of the uploaded stream. This works for most formats: | Zeek | `application/x-zeek` | | Arrow IPC stream | `application/vnd.apache.arrow.stream` | -Three formats cannot be auto-detected over the API and **require an explicit +Two formats cannot be auto-detected over the API and **require an explicit `mime_type`**: | Format | MIME type | Reason | |---|---|---| -| CSUP | `application/x-csup` | Detection requires seekable input | | Parquet | `application/x-parquet` | Detection requires seekable input | | Line | `application/x-line` | Auto-detection not supported | -CSUP and Parquet store identifying metadata outside the stream header (Parquet's -footer is at the end of the file), so the server cannot detect them from an HTTP +Parquet stores identifying metadata outside the stream header (its +footer is at the end of the file), so the server cannot detect it from an HTTP request body. Line format is plain text with no identifying structure and is intentionally excluded from auto-detection. diff --git a/book/src/formats/bsup.md b/book/src/formats/bsup.md index 2e98ae160f..50124c6513 100644 --- a/book/src/formats/bsup.md +++ b/book/src/formats/bsup.md @@ -2,188 +2,115 @@ ## 1. Introduction -Super Binary (BSUP) is an efficient, sequence-oriented serialization format for -[super-structured data](model.md). - -BSUP is "row oriented" and -analogous to [Apache Avro](https://avro.apache.org) but does not -require schema definitions as it instead utilizes the fine-grained type system -of the super data model. -This binary format is based on machine-readable data types with an -encoding methodology inspired by Avro, -[Parquet](https://en.wikipedia.org/wiki/Apache_Parquet), -[Protocol Buffers](https://developers.google.com/protocol-buffers), and -[Apache Arrow](https://arrow.apache.org/docs/format/Columnar.html). - -To this end, BSUP embeds all type information -in the stream itself while having a binary serialization format that -allows "lazy parsing" of fields such that -only the fields of interest in a stream need to be deserialized and interpreted. -Unlike Avro, BSUP embeds its "schemas" in the data stream as types and thereby admits -an efficient multiplexing of heterogeneous data types by prepending to each -data value a simple integer identifier to reference its type. - -Since no external schema definitions exist in BSUP, a "type context" is constructed -on the fly by composing dynamic type definitions embedded in the format. -BSUP can be readily adapted to systems like -[Apache Kafka](https://kafka.apache.org/) which utilize schema registries, -by having a connector translate the schemas implied in the -BSUP stream into registered schemas and vice versa. Better still, Kafka could -be used natively with BSUP obviating the need for the schema registry. - -Multiple BSUP streams with different type contexts are easily merged because the -serialization of values does not depend on the details of -the type context. One or more streams can be merged by simply merging the -input contexts into an output context and adjusting the type reference of -each value in the output sequence. The values need not be traversed -or otherwise rewritten to be merged in this fashion. - -## 2. The BSUP Format - -A BSUP stream comprises a sequence of frames where -each frame contains one of three types of data: -_types_, _values_, or externally-defined _control_. - -A stream is punctuated by the end-of-stream value `0xff`. - -Each frame header includes a length field -allowing an implementation to easily skip from frame to frame. - -Each frame begins with a single-byte version number followed by -a single byte "frame code": -``` - 7 6 5 4 3 2 1 0 - +-+-+-+-+-+-+-+-+ - |1| VERSION | - +-+-+-+-+-+-+-+-+ - |X|C| T| L| - +-+-+-+-+-+-+-+-+ - - VERSION: 7 bits - - The BSUP version number. The upper bit of the version byte must be 1. - - X: 1 bit - - Unused. - - C: 1 bit - - Indicates compressed frame data. - - T: 2 bits - - Type of frame data. - - 00: Types - 01: Values - 10: Control - 11: undefined - - L: 4 bits - - Low-order bits of frame length. -``` - -Following the frame code is its encoded length followed by a "frame payload" -of bytes of said length: -``` - -``` -The length encoding utilizes a variable-length unsigned integer called herein a `uvarint`: - ->[!NOTE] -> Inspired by Protocol Buffers, -> a `uvarint` is an unsigned, variable-length integer encoded as a sequence of -> bytes consisting of N-1 bytes with bit 7 clear and the Nth byte with bit 7 set, -> whose value is the base-128 number composed of the digits defined by the lower -> 7 bits of each byte from least-significant digit (byte 0) to -> most-significant digit (byte N-1). - -The frame payload's length is equal to the value of the `uvarint` following the -frame code times 16 plus the low 4-bit integer value `L` field in the frame code. - -If the `C` bit is set in the frame code, then the frame payload following the -frame length is compressed and has the form: -``` - -``` -where -* `` is a single byte indicating the compression format of the compressed payload, -* `` is a `uvarint` encoding the size of the uncompressed payload, and -* `` is a bytes sequence whose length equals -the outer frame length less 1 byte for the compression format and the encoded length -of the `uvarint` size field. - -The `compressed payload` is compressed according to the compression algorithm -specified by the `format` byte. Each frame is compressed independently -such that the compression algorithm's state is not carried from frame to frame -(thereby enabling parallel decoding). - -The `` value is redundant with the compressed payload -but is useful to an implementation to deterministically -size decompression buffers in advance of decoding. - -Of the 256 possible values for the `` byte, only type `0` is currently -defined and specifies that `` contains an -[LZ4 block](https://github.com/lz4/lz4/blob/master/doc/lz4_Block_format.md). - ->[!NOTE] -> This arrangement of frames separating types and values allows -> for efficient scanning and parallelization. In general, values depend -> on type definitions but as long as all of the types are known when -> values are used, decoding can be done in parallel. Likewise, since -> each block is independently compressed, the blocks can be decompressed -> in parallel. Moreover, efficient filtering can be carried out over -> uncompressed data before it is deserialized into native data structures, -> e.g., allowing entire frames to be discarded based on -> heuristics, e.g., knowing a filtering predicate can't be true based on a -> quick scan of the data perhaps using the Boyer-Moore algorithm to determine -> that a comparison with a string constant would not work for any -> value in the buffer. - -Whether the payload was originally uncompressed or was decompressed, it is -then interpreted according to the `T` bits of the frame code as a -* [types frame](#21-types-frame), -* [values frame](#22-values-frame), or -* [control frame](#23-control-frame). - -### 2.1 Types Frame - -A _types frame_ encodes a sequence of type definitions for complex types -and establishes a "type ID" for each such definition. -Type IDs for the "primitive types" -are predefined with the IDs listed in the [Primitive Types](#3-primitive-types) table. - -Each definition, or "typedef", -consists of a typedef code followed by its type-specific encoding as described below. -A table of typedefs is encoded as the number of typedefs followed by two -sections: +Super Binary (BSUP) is an efficient binary serialization format for +[super-structured data](model.md). It holds data in either of two +orientations, and a single BSUP file may mix them: + +* _row frames_ encode a sequence of values one after another, a bit like + [Apache Avro](https://avro.apache.org), and +* _column frames_ encode a sequence of values as vectors, a bit like + [Apache Parquet](https://parquet.apache.org/) or + [Apache Arrow](https://arrow.apache.org/docs/format/Columnar.html). + +Unlike Avro and Parquet, BSUP has no external schema. Each frame carries +the definitions of the types of its values in a compact _typedefs table_, +so values of many different types may be freely intermixed in a single +sequence and each frame can be decoded on its own. + +A BSUP file is organized into _super frames_, each a sequence of frames +followed by a _super footer_ that summarizes the super frame. The super +footer holds the _fused type_ of the super frame's values (a single type to +which all of its values conform) and, for column frames, +[zone maps](#86-zone-maps) of the minimum and maximum value of each +column of each frame. This allows a reader to determine the type of an +entire file and to skip frames that can't match a query without reading +the frames themselves. + +Since each frame is self-contained and each super frame ends with a footer +that records its size, BSUP files may simply be concatenated to form a +larger, valid BSUP file, which can be traversed either forward from its +start or backward from its end. + +This specification describes BSUP version 0.31. + +## 2. Conventions + +All fixed-size integers in headers, footers, and data are encoded in +little-endian byte order unless stated otherwise. + +A `uvarint` is an unsigned, variable-length integer as defined by +[Protocol Buffers](https://protobuf.dev/programming-guides/encoding/#varints): +a sequence of bytes with bit 7 set in each byte except the last, whose value +is the base-128 number composed of the lower 7 bits of each byte from +least-significant digit (byte 0) to most-significant digit. + +A _counted string_ is a `uvarint` length followed by that many bytes of +UTF-8 text. + +The byte layouts below show fixed-size fields with their size in bytes +(e.g., ``) and variable-size fields without one. + +## 3. File Structure + +A BSUP file is a sequence of super frames. A super frame is a sequence of +one or more frames followed by a super footer: +``` +| frame | frame | ... | super footer | frame | frame | ... | super footer | ... +``` +Each frame and each super footer begins with a 4-byte magic string that +identifies it: + +| Magic | Section | +|--------|------------------------------------------------| +| `SUPR` | [row frame](#7-row-frames) | +| `SUPX` | [control frame](#74-control-frames) | +| `SUPC` | [column frame](#8-column-frames) | +| `SUPS` | [super footer](#9-super-footer) | + +The frames of a super frame are either all row frames or all column frames, +except that control frames may appear in either. Row super frames and +column super frames may be intermixed in a file. + +The magic is followed by a 2-byte version number, which encodes a +major version in its upper 6 bits and a minor version in its lower 10 bits. +Implementations may read data with the same major version as their own, +though an implementation might not understand types added by a later minor +version. Major version 0 is reserved for development and there is no +compatibility between its minor versions. The current version is 0.31. + +Each frame header and super footer holds its own size, so a reader can +traverse a file forward from its start, skipping over any frame +it doesn't need. Each super footer also ends with a copy of its size and +holds the size of its super frame, so a reader can also traverse a +file backward from its end, e.g., to read the fused types and zone maps +of a file's super frames without reading any frames. + +An implementation may bound the sizes of these sections. The reference +implementation limits a frame to 2 GiB, a super frame to 512 GiB, and the +typedefs, locator, and zone map sections and the super footer to 100 MiB. + +## 4. Typedefs Tables + +A typedefs table defines a set of complex types and assigns each a +_type ID_. Each frame has its own typedefs tables (a row frame has one and a +column frame two) and a type ID is meaningful only with respect to its table. + +The primitive types have predefined type IDs less than 48 as listed in the +[Primitive Types](#5-primitive-types) table. The typedefs in a table are +numbered in order starting at 48, so the type ID of a typedef is implied by +its position and isn't encoded. + +A typedefs table is encoded as the number of typedefs, the offset of each +typedef, and the typedefs: ``` ------------------------------------------------------------ ||...|...| ------------------------------------------------------------ ``` -The first section has the offset of each typedef in type ID order, where -each offset is relative to the start of the second section, and the -second section has the typedefs. `` and each offset are encoded -as a `uvarint`. The offsets allow a typedef to be found by its type ID -without decoding the typedefs that precede it. An empty table is encoded -as zero bytes. +where `` and each offset are `uvarint`s and each offset is relative +to the start of the first typedef, so the first offset is always 0. The +offsets allow a typedef to be found by its type ID without decoding the +typedefs that precede it. An empty table is encoded as zero bytes. -The typedefs are numbered in the order encountered starting at 30 -(as the largest primary type ID is 29). Types refer to other types -by their type ID. Note that the type ID of a typedef is implied by its -position in the sequence and is not explicitly encoded. - -A typedef other than a named type may refer only to types with smaller -type IDs, i.e., to primitive types and to typedefs that precede it. -A named type's inner type may refer to any type, which allows recursive -types, but a cycle of references must include a type that is not named, -e.g., a named type may not refer to itself directly or through other -named types alone. A table that breaks these rules is malformed. - -The typedef codes are defined as follows: +Each typedef is a 1-byte typedef code followed by its type-specific encoding: | Code | Complex Type | |------|--------------| @@ -195,518 +122,579 @@ The typedef codes are defined as follows: | 5 | enum | | 6 | error | | 7 | named type | -| 8 | fusion type | - -Any references to a type ID in the body of a typedef are encoded as a `uvarint`. - -#### 2.1.1 Record Typedef - -A record typedef creates a new type ID equal to the next stream type ID -with the following structure: -``` ----------------------------------------------------------------------- -|0x00||...| ----------------------------------------------------------------------- -``` -Record types consist of an ordered set of fields where each field consists of -a name and its type. Unlike JSON, the ordering of the fields is significant -and must be preserved through any APIs that consume, process, and emit BSUP records. +| 8 | fusion | +| 9 | option | +| 10 | decimal | -A record type is encoded as a count of fields, i.e., `` from above, -followed by the field definitions, -where a field definition is a field name, followed by its optionality coded -as a single byte, followed by a type ID, i.e., -`` followed by `` etc. as indicated above. +Within a typedef, a reference to another type is that type's ID encoded as a +`uvarint`, except for the inner type of a named type (see below). -The field names in a record must be unique. - -The `` value is encoded as a `uvarint`. +A typedef other than a named type may refer only to types with smaller +type IDs, i.e., to primitive types and to typedefs that precede it. +A named type may refer to any type, which allows recursive +types, but a cycle of references must include a type that is not named, +i.e., a named type may not refer to itself directly or through other +named types alone. A table that breaks these rules is malformed. -The field name is encoded as a UTF-8 string defining a "BSUP identifier". -The UTF-8 string -is further encoded as a "counted string", which is the `uvarint` encoding -of the length of the string followed by that many bytes of UTF-8 encoded -string data. +An encoder writes at most one typedef for any type in a table. ->[!NOTE] -> As defined by [Super (SUP)](sup.md), a field name can be any valid UTF-8 string much like JSON -> objects can be indexed with arbitrary string keys (via index operator) -> even if the field names available to the dot operator are restricted -> by language syntax for identifiers. +### 4.1 Record Typedef -The optionality byte indicates "optional" for the value 1 and mandatory for -the value 0. +``` +|0x00||...| +``` +A record type is an ordered list of fields, each a name and a type. +`` is a `uvarint`, each name is a counted string, and the +names of a record's fields must be unique. Unlike JSON, the order of the +fields is significant. -The type ID follows the field name and is encoded as a `uvarint`. +A field that may be absent from a value has an [option](#410-option-typedef) +type. -#### 2.1.2 Array Typedef +### 4.2 Array Typedef -An array type is encoded as simply the type code of the elements of -the array encoded as a `uvarint`: ``` ----------------- |0x01|| ----------------- ``` +where `` is the type of the array's elements. -#### 2.1.3 Set Typedef +### 4.3 Set Typedef -A set type is encoded as the type ID of the -elements of the set, encoded as a `uvarint`: ``` ----------------- |0x02|| ----------------- ``` +where `` is the type of the set's elements. -#### 2.1.4 Map Typedef +### 4.4 Map Typedef -A map type is encoded as the type code of the key -followed by the type code of the value. ``` --------------------------- -|0x03||| --------------------------- +|0x03||| ``` -Each `` is encoded as `uvarint`. -#### 2.1.5 Union Typedef +### 4.5 Union Typedef -A union typedef creates a new type ID equal to the next stream type ID -with the following structure: ``` ------------------------------------------ |0x04||...| ------------------------------------------ ``` -A union type consists of an ordered set of types -encoded as a count of the number of types, i.e., `` from above, -followed by the type IDs comprising the types of the union. -The type IDs of a union must be unique. - -The `` and the type IDs are all encoded as `uvarint`. - -`` cannot be 0. +where `` is a `uvarint` and the type IDs are those of the union's +member types, which must be distinct, may not themselves be unions, and are +sorted in the canonical order of types. The position of a member type +in this list is its _tag_. -#### 2.1.6 Enum Typedef +### 4.6 Enum Typedef -An enum type is encoded as a `uvarint` representing the number of symbols -in the enumeration followed by the names of each symbol. ``` --------------------------------- -|0x05||...| --------------------------------- +|0x05||...| ``` -`` is encoded as `uvarint`. -The names have the same UTF-8 format as record field names and are encoded -as counted strings following the same convention as record field names. +where `` is a `uvarint` and each symbol is a counted string. -#### 2.1.7 Error Typedef +### 4.7 Error Typedef -An error type is encoded as follows: ``` ----------------- |0x06|| ----------------- ``` -which defines a new error type for error values that have the underlying type -indicated by ``. +where `` is the type of the error's value. -#### 2.1.8 Named Type Typedef +### 4.8 Named Type Typedef -A named type defines a new type ID that binds a name to a previously existing type ID. - -A named type is encoded as follows: ``` ----------------------- -|0x07|| ----------------------- +|0x07||| ``` -where `` is an identifier representing the new type name with a new type ID -allocated as the next available type ID in the stream that refers to the -existing type ID ``. `` is encoded as a `uvarint` and `` -is encoded as a `uvarint` representing the length of the name in bytes, -followed by that many bytes of UTF-8 string. +where `` is a counted string and `` is the ID of the named +type's underlying type encoded as a 4-byte, _big-endian_ integer. +(The fixed size allows an encoder to define a recursive type by writing +the named typedef before its underlying type is known and filling in its +type ID later.) -As indicated in the [data model](model.md), -it is an error to define a type name that has the same name as a primitive type, -and it is permissible to redefine a previously defined type name with a -type that differs from the previous definition. +A name may be bound to only one type in a typedefs table. It is an error +to name a type with the name of a primitive type. -#### 2.1.8 Fusion Typedef +### 4.9 Fusion Typedef -A fusion type is encoded as follows: ``` ----------------- |0x08|| ----------------- -``` -which defines a new fusion type for fusion values that have the underlying type -indicated by ``. - -### 2.2 Values Frame - -A _values frame_ is a sequence of values each encoded as the value's type ID, -encoded as a `uvarint`, followed by its tag-encoded serialization as described below. - -Since a single type ID encodes the entire value's structure, no additional -type information is needed. Also, the value encoding follows the structure -of the type explicitly so the type is not needed to parse the structure of the -value, but rather only its semantics. - -It is an error for a value to reference a type ID that has not been -previously defined by a typedef scoped to the stream in which the value -appears. - -The value is encoded using a "tag-encoding" scheme -that captures the structure of both primitive types and the recursive -nature of complex types. This structure is encoded -explicitly in every value and the boundaries of each value and its -recursive nesting can be parsed without knowledge of the type or types of -the underlying values. This admits an efficient implementation -for traversing the values, inclusive of recursive traversal of complex values, -whereby the inner loop need not consult and interpret the type ID of each element. - -#### 2.2.1 Tag-Encoding of Values - -Each value is prefixed with a length "tag" indicating its encoded length in bytes. - -Zero-length values are possible as many data types -have a meaningful empty interpretation, for example, an -empty array, the empty record, etc. - -The tag itself is encoded as a `uvarint`. - -#### 2.2.2 Tag-Encoded Body of Primitive Values - -Following the tag encoding is the value encoded in N bytes as described above. -A typed value with a `value` of length `N` is interpreted as described in the -[Primitive Types](#3-primitive-types) table. The type information needed to -interpret all of the value elements of a complex type are all implied by the -top-level type ID of the values frame. For example, the type ID could indicate -a particular record type, which recursively provides the type information -for all of the elements within that record, including other complex types -embedded within the top-level record. - -Note that because the tag indicates the length of the value, there is no need -to use varint encoding of integer values. Instead, an integer value is encoded -using the full 8 bits of each byte in little-endian order. Signed values, -before encoding, are shifted left one bit, and the sign bit stored as bit 0. -For negative numbers, the remaining bits are negated so that the upper bytes -tend to be zero-filled for small integers. - -#### 2.2.3 Tag-Encoded Body of Complex Values - -The body of a length-N container for a complex value -comprises zero or more tag-encoded values, -where the values are encoded as follows: - -| Type | Value | -|----------|-------------------------------------------| -| `array` | concatenation of elements | -| `set` | normalized concatenation of elements | -| `record` | option bits and concatenation of elements | -| `map` | concatenation of key and value elements | -| `union` | concatenation of tag and value | -| `enum` | position of enum element | -| `error` | wrapped element | -| `fusion` | element followed by subtype value | - -Since N, the byte length of any of these container values, is known, -there is no need to encode a count of the -elements present. Also, since the type ID is implied by the typedef -of any complex type, each value is encoded without its type ID. - -For sets, the concatenation of elements must be normalized so that the -sequence of bytes encoding each element's tag-counted value is -lexicographically greater than that of the preceding element. - -For records, when its type has optional fields, -a bit vector of length NF is encoded as a tag-encoded value of -floor((NF+7)/8) bytes to indicate the omission of an optional value -where NF is the number of optional fields in the record. -The field order of optional fields determines their position -in the bit vector with bit numbers 0-7 (least significant to most significant) -in the first byte, number 8-15 in the second byte, and so forth. -Following the option bits is a concatenation -of elements comprising the mandatory values and the optional values that are present -all in field order. - -A union value is encoded as a container with two elements. The first -element, called the tag, is the `uvarint` encoding of the -positional index determining the type of the value in reference to the -union's list of defined types, and the second element is the value -encoded according to that type. - -An enum value is represented as the `uvarint` encoding of the -positional index of that value's symbol in reference to the enum's -list of defined symbols. - -A map value is encoded as a container whose elements are alternating -tag-encoded keys and values, with keys and values encoded according to -the map's key type and value type, respectively. - -The concatenation of elements must be normalized so that the -sequence of bytes encoding each tag-counted key (of the key/value pair) is -lexicographically greater than that of the preceding key (of the preceding -key/value pair). - -### 2.3 Control Frame - -A _control frame_ contains an application-defined control message. - -Control frames are available to higher-layer protocols and are carried -in BSUP as a convenient signaling mechanism. A BSUP implementation -may skip over all control frames and is guaranteed by -this specification to decode all of the data as described herein even if such -frames provide additional semantics on top of the base BSUP format. - -The body of a control frame is a control message and may be JSON, -SUP, BSUP, arbitrary binary, or UTF-8 text. The serialization of the control -frame body is independent of the stream containing the control frame. - -Any control message not known by a BSUP data receiver shall be ignored. - -The delivery order of control messages with respect to the delivery -order of values of the BSUP stream should be preserved by an API implementing -BSUP serialization and deserialization. -In this way, system endpoints that communicate using BSUP can embed -protocol directives directly into the stream as control payloads -in an order-preserving semantics rather than defining additional -layers of encapsulation and synchronization between such layers. - -A control frame has the following form: -``` -------------------------- -|||| -------------------------- -``` -where -* `` is a single byte indicating whether the body is encoded -as BSUP (0), JSON (1), SUP (2), an arbitrary UTF-8 string (3), or arbitrary binary data (4), -* `` is a `uvarint` encoding the length in bytes of the body -(exclusive of the length 1 encoding byte), and -* `` is a control message whose semantics are outside the scope of -the base BSUP specification. - -If the encoding type is BSUP, the embedded BSUP data -starts and ends a single BSUP stream independent of the outer BSUP stream. - -### 2.4 End of Stream - -A BSUP stream must be terminated by an end-of-stream marker. -A new BSUP stream may begin immediately after an end-of-stream marker. -Each such stream has its own, independent type context. - -In this way, the concatenation of BSUP streams (or BSUP files containing -BSUP streams) results in a valid BSUP data sequence. - -For example, a large BSUP file can be arranged into multiple, smaller streams -to facilitate random access at stream boundaries. -This benefit comes at the cost of some additional overhead -- -the space consumed by stream boundary markers and repeated type definitions. -Choosing an appropriate stream size that balances this overhead with the -benefit of enabling random access is left up to implementations. - -End-of-stream markers are also useful in the context of sending BSUP over Kafka, -as a receiver can easily resynchronize with a live Kafka topic by -discarding incomplete frames until a frame is found that is terminated -by an end-of-stream marker (presuming the sender implementation aligns -the BSUP frames on Kafka message boundaries). - -A end-of-stream marker is encoded as follows: -``` ------- -|0xff| ------- -``` - -After this marker, all previously read -typedefs are invalidated and the "next available type ID" is reset to -the initial value of 30. To represent subsequent values that use a -previously defined type, the appropriate typedef control code must -be re-emitted -(and note that the typedef may now be assigned a different ID). - -## 3. Primitive Types - -For each BSUP primitive type, the following table describes: -* its type ID, and -* the interpretation of a length `N` [value frame](#22-values-frame). - -All fixed-size multi-byte sequences representing machine words -are serialized in little-endian format. - -| Type | ID | N | BSUP Value Interpretation | -|--------------|---:|:--------:|------------------------------------------------| -| `uint8` | 0 | variable | unsigned int of length N | -| `uint16` | 1 | variable | unsigned int of length N | -| `uint32` | 2 | variable | unsigned int of length N | -| `uint64` | 3 | variable | unsigned int of length N | -| `uint128` | 4 | variable | unsigned int of length N | -| `uint256` | 5 | variable | unsigned int of length N | -| `int8` | 6 | variable | signed int of length N | -| `int16` | 7 | variable | signed int of length N | -| `int32` | 8 | variable | signed int of length N | -| `int64` | 9 | variable | signed int of length N | -| `int128` | 10 | variable | signed int of length N | -| `int256` | 11 | variable | signed int of length N | -| `duration` | 12 | variable | signed int of length N as ns | -| `time` | 13 | variable | signed int of length N as ns since epoch | -| `float16` | 14 | 2 | 2 bytes of IEEE 64-bit format | -| `float32` | 15 | 4 | 4 bytes of IEEE 64-bit format | -| `float64` | 16 | 8 | 8 bytes of IEEE 64-bit format | -| `float128` | 17 | 16 | 16 bytes of IEEE 64-bit format | -| `float256` | 18 | 32 | 32 bytes of IEEE 64-bit format | -| `decimal32` | 19 | 4 | 4 bytes of IEEE decimal format | -| `decimal64` | 20 | 8 | 8 bytes of IEEE decimal format | -| `decimal128` | 21 | 16 | 16 bytes of IEEE decimal format | -| `decimal256` | 22 | 32 | 32 bytes of IEEE decimal format | -| `bool` | 23 | 1 | one byte 0 (false) or 1 (true) | -| `bytes` | 24 | variable | N bytes of value | -| `string` | 25 | variable | UTF-8 byte sequence | -| `ip` | 26 | 4 or 16 | 4 or 16 bytes of IP address | -| `net` | 27 | 8 or 32 | 8 or 32 bytes of IP prefix and subnet mask | -| `type` | 28 | variable | type value byte sequence [as defined below](#4-type-values) | -| `null` | 29 | 0 | no value, always represents an undefined value | -| `none` | 30 | 0 | appears in empty sets/arrays/maps | - -## 4. Type Values - -As the super data model supports first-class types and because the BSUP design goals -require that value serializations cannot change across type contexts, type values -must be encoded in a fashion that is independent of the type context. -Thus, a serialized type value encodes the entire type in a canonical form -according to the recursive definition in this section. - -The type value of a primitive type (include type `type`) is its primitive ID, -serialized as a single byte. - -The type value of a complex type is serialized recursively according to the -complex type it represents as described below. - -### 4.1 Record Type Value - -A record type value has the form: -``` ---------------------------------------------------------- -|30|||...| ---------------------------------------------------------- -``` -where `` is the number of fields in the record encoded as a `uvarint`, -`` is a bit vector of length `` indicating which fields are optional, -`` etc. are the field names encoded as in the -record typedef, and each `` is a recursive encoding of a type value. - -### 4.2 Array Type Value - -An array type value has the form: -``` --------------- -|31|| --------------- -``` -where `` is a recursive encoding of a type value. - -### 4.3 Set Type Value - -A set type value has the form: -``` --------------- -|32|| --------------- ``` -where `` is a recursive encoding of a type value. +where `` is the fusion's supertype. + +### 4.10 Option Typedef + +``` +|0x09|| +``` +where `` is the type of an option's value when it has one. The +underlying type may not be an option, `missing`, or `none`. + +### 4.11 Decimal Typedef + +``` +|0x0a||| +``` +where `` and `` are `uvarint`s. Decimal values are +reserved and not yet defined. + +## 5. Primitive Types + +The primitive types have the following type IDs. The table also describes the +_body_ of a value of each type (see [Values](#73-values)). + +| Type | ID | Body | +|--------------------|---:|------------------------------------------------------------| +| `uint8` | 0 | unsigned integer as a variable-length integer (see below) | +| `uint16` | 1 | " | +| `uint32` | 2 | " | +| `uint64` | 3 | " | +| `uint128` | 4 | reserved | +| `int8` | 5 | signed integer as a variable-length integer (see below) | +| `int16` | 6 | " | +| `int32` | 7 | " | +| `int64` | 8 | " | +| `int128` | 9 | reserved | +| `duration` | 10 | signed integer count of nanoseconds | +| `timestamp(0)` | 11 | reserved | +| `timestamp(3)` | 12 | reserved | +| `timestamp(6)` | 13 | reserved | +| `timestamp` | 14 | signed integer count of nanoseconds since the Unix epoch | +| `timestamp_ntz(0)` | 15 | reserved | +| `timestamp_ntz(3)` | 16 | reserved | +| `timestamp_ntz(6)` | 17 | reserved | +| `timestamp_ntz` | 18 | reserved | +| `date` | 19 | reserved | +| `time(0)` | 20 | reserved | +| `time(3)` | 21 | reserved | +| `time(6)` | 22 | reserved | +| `time` | 23 | reserved | +| `float16` | 24 | 2 bytes of IEEE 754 binary16 | +| `float32` | 25 | 4 bytes of IEEE 754 binary32 | +| `float64` | 26 | 8 bytes of IEEE 754 binary64 | +| `bool` | 27 | 1 byte, 0 (false) or 1 (true) | +| `bytes` | 28 | the bytes | +| `string` | 29 | UTF-8 text | +| `ip` | 30 | 4 or 16 bytes of IPv4 or IPv6 address in network byte order | +| `net` | 31 | address followed by mask, each 4 or 16 bytes as for `ip` | +| `interval` | 32 | reserved | +| `uuid` | 33 | 16 bytes | +| `geometry` | 34 | reserved | +| `geography` | 35 | reserved | +| `type` | 36 | [type value](#6-type-values) | +| `null` | 37 | empty | +| `missing` | 38 | empty | +| `none` | 39 | empty | +| `all` | 40 | a value of any type (see [Fusion Values](#733-complex-values)) | +| `any` | 41 | none: no value has type `any`, the top type of the type system | + +IDs 42 through 47 are reserved for future primitive types. A type whose +body is "reserved" has a type ID, so it may appear in a type, but its values +are not yet defined. + +An unsigned integer is encoded as the minimum number of bytes needed to hold +it in little-endian order, so zero has an empty body. +A signed integer `i` is first mapped to an unsigned integer +as `i<<1` when `i` is nonnegative and as `(-i)<<1 | 1` when `i` is negative, +which is then encoded as an unsigned integer. (The unsigned integer 1 +represents the minimum 64-bit signed integer.) +Since the length of a body is always known, these variable-length integers +need no further delimiters. + +A `net` value's address is masked by its mask, whose leading one bits give +the prefix length. + +## 6. Type Values + +A type value (a value of type `type`, the fused type of a super frame, or the +subtype of a fusion value) does not depend on any typedefs table, so it is the +same in every frame. The body of a type value is the type's encoding as +follows. A primitive type is its type ID encoded as a single byte. A complex type is +encoded recursively as a 1-byte code followed by: + +| Code | Type | Encoding | +|------|--------------|----------------------------------------------------------------------| +| 48 | record | `uvarint` number of fields, then each field's name and type value | +| 49 | array | element type value | +| 50 | set | element type value | +| 51 | map | key type value, value type value | +| 52 | union | `uvarint` number of types, then each type value in canonical order | +| 53 | enum | `uvarint` number of symbols, then each symbol | +| 54 | error | value type value | +| 55 | named (definition) | name, underlying type value | +| 56 | named (reference) | name | +| 57 | fusion | supertype type value | +| 58 | option | value type value | +| 59 | decimal | `uvarint` precision, `uvarint` scale | + +where names and symbols are counted strings. + +The first occurrence of a named type in a type value, in a left-to-right +depth-first traversal, is a definition and later occurrences are references, +which allows recursive types. The name binding is local to the type +value. + +## 7. Row Frames + +A row frame encodes a sequence of values, each with its type ID in the +frame's typedefs table. + +### 7.1 Row Frame Header + +A row frame begins with a 26-byte header: + +| Offset | Size | Field | +|-------:|-----:|-------------------------------------------------------------| +| 0 | 4 | magic, `SUPR` (or `SUPX` for a control frame) | +| 4 | 2 | version | +| 6 | 8 | frame size, which includes the header | +| 14 | 4 | type ID of the frame's fused type in its typedefs table | +| 18 | 8 | size of the typedefs section | + +### 7.2 Row Frame Sections + +The header is followed by a typedefs section, which is the frame's +[typedefs table](#4-typedefs-tables), and the values section, which fills +the rest of the frame: +``` +|||| +``` +The values section is a sequence of values, each encoded as its type ID +followed by its length-prefixed body: +``` +|||... +``` +where `` and `` are `uvarint`s. + +### 7.3 Values + +A value is encoded as a _body_ whose length is known from its context. +Where values are concatenated, each value is _length-prefixed_, i.e., +encoded as the `uvarint` length of its body followed by its body: +``` +|| +``` +The body of a value of a complex type is a _container_, which is the +concatenation of the length-prefixed values described below. This structure is +explicit in every value so the boundaries of each value and its nested +values can be found without knowledge of their types, which allows +values to be traversed efficiently and only the parts of interest to be +interpreted. + +#### 7.3.1 Primitive Values + +The body of a primitive value is described in the +[Primitive Types](#5-primitive-types) table. -### 4.4 Map Type Value - -A map type value has the form: -``` --------------------------- -|33||| --------------------------- -``` -where `` and `` are recursive encodings of type values. +#### 7.3.2 Type Values -### 4.5 Union Type Value +The body of a value of type `type` encodes the entire type canonically as +described in [Type Values](#6-type-values). + +#### 7.3.3 Complex Values + +The body of a value of a complex type is: + +| Type | Body | +|----------|-------------------------------------------------------------------| +| record | container of the values of its fields in field order | +| array | container of its elements | +| set | container of its elements, normalized (see below) | +| map | container of its keys and values alternating, normalized | +| union | container of its tag as an unsigned integer and its value | +| enum | the position of its symbol as an unsigned integer | +| error | the body of its value | +| named | the body of its value as its underlying type | +| option | container of 0 and its value if it has one, else container of 1 | +| fusion | container of its value and the [type value](#6-type-values) of its subtype | + +The integers in these bodies (e.g., a union's tag) are encoded as +in [Primitive Types](#5-primitive-types), so a union value of tag 0 begins +with a length-prefixed value of length 0. -A union type value has the form: -``` ------------------------------------ -|34||...| ------------------------------------ -``` -where `` is the number of types in the union encoded as a `uvarint` -and each `` is a recursive definition of a type value. +A set's elements are normalized by sorting them by their length-prefixed +bytes and removing duplicates. Likewise, a map's entries are normalized +by sorting them by the length-prefixed bytes of their keys and removing entries +with duplicate keys. -### 4.6 Enum Type Value +A fusion value is its value encoded as a value of the fusion's supertype +followed by the type value of the value's actual type. A value of type +`fusion(all)` holds the body of a value of any type, which is +interpreted according to the subtype. -An enum type value has the form: -``` ------------------------------- -|35||...| ------------------------------- -``` -where `` and each symbol name is encoded as in an enum typedef. +### 7.4 Control Frames -### 4.7 Error Type Value +A control frame has the same format as a row frame but the magic `SUPX`. +Its values are application-defined control messages, which are delivered +in order with the values of the file (e.g., to signal the end of a +logical stream to a receiver). A reader that doesn't understand a +control message may ignore it. -An error type value has the form: -``` ------------ -|36|| ------------ -``` -where `` is the type value of the error. +## 8. Column Frames -### 4.8 Named Type Type Value - -A named type type value may appear either as a definition or a reference. -When a named type is referenced, it must have been previously -defined in the type value in accordance with a left-to-right depth-first-search (DFS) -traversal of the type. - -A named type definition has the form: -``` --------------------- -|37|| --------------------- -``` -where `` is encoded as in a named type typedef -and `` is a recursive encoding of a type value. This creates -a binding between the given name and the indicated type value only within the -scope of the encoded value and does not affect the type context. -This binding may be changed by another named type definition -of the same name in the same type value according to the DFS order. - -An named type reference has the form: -``` ------------ -|38|| ------------ -``` -It is an error for a named type reference to appear in a type value with a name -that has not been previously defined according to the DFS order. - -### 4.9 Fusion Type Value - -A fusion type value has the form: -``` ------------ -|39|| ------------ -``` -where `` is the type value of the fusion supertype. +A column frame encodes a sequence of values as one or more vectors. +A vector holds a sequence of values of a single type, which is described by +the type's _locators_: its length and the location in the +frame's data section of each _segment_ of the data that encodes its +values. For example, a vector of `int64` values has one segment of +integers and a vector of records has a vector for each of its fields. +If the frame's values have more than one type, its values are a _dynamic_ +of several vectors, each of a different type, with a tag for each value that +indicates its vector. -## 5. Compression Types +A reader that needs only some of the values of a frame (e.g., the +values of some record fields) reads the locators and then only the +segments it needs. -This section specifies values for the `` byte of a -[compressed value message block](#2-the-bsup-format) -and the corresponding algorithms for the `` byte sequence. +### 8.1 Column Frame Header -As new compression algorithms are specified, they will be documented -here without any need to change the other sections of the BSUP specification. +A column frame begins with a 42-byte header: -Of the 256 possible values for the `` byte, only type `0` is currently -defined and specifies that `` contains an -[LZ4 block](https://github.com/lz4/lz4/blob/master/doc/lz4_Block_format.md). +| Offset | Size | Field | +|-------:|-----:|-------------------------------------------------------------| +| 0 | 4 | magic, `SUPC` | +| 4 | 2 | version | +| 6 | 8 | frame size, which includes the header | +| 14 | 4 | type ID of the frame's fused type in its type values table | +| 18 | 8 | size of the vector typedefs section | +| 26 | 8 | size of the locators section | +| 34 | 8 | size of the type values section | + +### 8.2 Column Frame Sections + +The header is followed by four sections: +``` +|||||| +``` + +* The _vector typedefs_ section is a [typedefs table](#4-typedefs-tables) of + the types of the frame's vectors. +* The _locators_ section holds the [locators](#83-locators) of the frame's + vectors, which refer to types in the vector typedefs table. +* The _type values_ section is a [typedefs table](#4-typedefs-tables) of + the types of the frame's type values and the subtypes of its fusion + values, and the frame's fused type. These types are held apart from + the vector types since a reader typically needs them only for the + vectors it reads. +* The _data_ section fills the rest of the frame and holds the + segments of the frame's vectors. + +### 8.3 Locators + +The locators section begins with the number of root vectors and the type ID +of each: +``` +||...||...| +``` +where `` and each type ID are `uvarint`s. If there is more than one +root vector, the values of the frame form a dynamic and `` is a +[segment](#84-segments) of [unsigned 32-bit integers](#85-segment-data), +one for each value, whose value is the index of the root vector that holds +the value, so the frame's length is the sum of the lengths of the root +vectors. Otherwise, `` is absent and the single root vector holds the +frame's values. Each `` is the locators of a root vector. + +The locators of a vector are encoded by traversing its type in depth-first +order. The locators of a vector begin with a 1-byte _kind_: + +| Kind | Name | Description | +|------|-------|-------------------------------------------------------------| +| 0 | plain | a vector whose values are encoded by type as described below | +| 1 | const | a vector whose values are all the same value | +| 2 | dict | a vector of a primitive type encoded as a dictionary | +| 3 | none | the values of an option vector whose values are all none | + +In the descriptions below, `` is a `uvarint` length of a vector and +each `` is a [segment](#84-segments). + +A _const_ vector is +``` +|0x01||| +``` +where `` is a `uvarint` that is 0 for no value or the body length +plus 1 followed by the body of the vector's value +(see [Values](#73-values)). A const vector of +count 0 is an empty vector. A const vector may have any type, including +the reserved primitive types and decimal types, which have no plain encoding. + +A _dict_ vector is +``` +|0x02|||| +``` +where `` is the locators of a plain vector of the dict's distinct +values (at most 256), `` is a segment of unsigned 32-bit integers +whose `k`th integer is the number of occurrences of the `k`th distinct value, +and `` is an uncompressed segment of one byte for each value of the +vector whose byte is the position of its value among the distinct values. +The vector's length is the length of ``. Only a vector of a primitive +type may be a dict vector. + +A _none_ vector is +``` +|0x03|| +``` +which may appear only as the values of an option vector (see below). + +A _plain_ vector is encoded according to its type: + +| Type | Locators after the kind byte | +|------|------------------------------| +| `uint8`...`uint64`, `int8`...`int64`, `duration`, `timestamp` | ``: a segment of 64-bit integers | +| `float16`, `float32`, `float64` | ``: a segment of 8-byte IEEE 754 binary64 values | +| `bool` | ``: a segment of bits | +| `bytes`, `string`, `uuid`, `all` | ``: a [bytes table](#85-segment-data) of the values' bodies | +| `ip`, `net` | ``: a bytes table of the values' bodies | +| `type` | ``: a segment of unsigned 32-bit type IDs in the type values table | +| `null`, `missing`, `none` | `` | +| record | the locators of each field's vector in field order, or `` if the record has no fields | +| array, set | `` | +| map | `` | +| union | `...` | +| enum | ``: a segment of 64-bit unsigned integers, each the position of a symbol | +| error | `` | +| named | `` | +| fusion | `` | +| option | `` | + +where: +* the vector of a record's field has the record's length; +* `` of an array, set, or map is a segment of `+1` unsigned + 32-bit integers, starting with 0, where the elements of the `k`th value + are the elements of `` (or the entries of `` and ``) + from offset `k` up to offset `k+1`; +* `` of a union is a segment of unsigned 32-bit integers that gives + each value's union tag and `` is the locators of the vector of the + values whose tag is `k-1`, in order. If the union has two member types, + `` instead holds the lengths of alternating runs of values with + tag 1 and tag 0, starting with a run of tag 1 (which may be 0), and is + empty if all values have tag 0; +* `` of an error or named vector is the locators of the vector of + its values, which has its length; +* `` of a fusion vector is the locators of the vector of its values + as values of the fusion's supertype (or of type `all`, for `fusion(all)`), + which has its length, and `` is a segment of unsigned 32-bit + integers, one for each value, that is the ID of its subtype in the type + values table; and +* `` of an option vector is a segment of unsigned 32-bit integers, + one for each value, that is 0 if the value has a value and 1 if + not, and `` is the locators of the vector of the values that + are present, in order. If all the values are present or none are, + `` is empty, and if none are, `` is a none vector of the + option's length. + +A vector's length is encoded only where it can't be derived from its +children's lengths, e.g., a record's length is the length of its first +field. + +Vectors may be nested no more than 10,000 deep, counting each +vector (e.g., an array of records of `int64` values is three vectors deep). + +### 8.4 Segments + +A segment is a contiguous range of the frame's data section. It is encoded +in the locators as its compression format, the length of its data +when uncompressed, and, if it's compressed, its length in the data section: +``` +|||[]| +``` +where the lengths are `uvarint`s and `` is present only if +`` is not 0. The compression formats are: + +| Format | Compression | +|--------|-----------------------------------------------------------------------------------| +| 0 | none | +| 1 | [LZ4 block](https://github.com/lz4/lz4/blob/master/doc/lz4_Block_format.md) | + +The offset of a segment isn't encoded. Instead, the segments of a frame are +written to its data section in the order in which they appear in the locators +section with no gaps between them, so each segment begins where the +previous one ends and the first begins at the start of the data section. +The segments must fill the data section exactly. + +A dynamic occurs only at the root of a column frame. Values of a union +type, by contrast, are encoded as union vectors, which retain the union type. + +### 8.5 Segment Data + +The data of the segments (after any decompression) are encoded as follows: + +* A segment of 64-bit integers (signed or unsigned) or of unsigned 32-bit + integers holds them compressed with the bit-packing integer codec of + [`github.com/ronanh/intcomp`](https://github.com/ronanh/intcomp) + (`CompressInt64`, `CompressUint64`, or `CompressUint32`) as that codec's + little-endian 64-bit or 32-bit words. Such a segment's compression format + is always 0. +* A segment of 8-byte binary64 values holds the values in order. + Values of `float16` and `float32` vectors are widened to binary64. +* A segment of bits holds bit `k` of the vector in bit `k%8` of byte `k/8`, + padded to a multiple of 8 bytes. +* A _bytes table_ is two segments: the concatenation of the values' bodies + and a segment of `+1` unsigned 32-bit offsets, starting with 0, where + the body of the `k`th value is the bytes from offset `k` up to offset `k+1`. + The offsets segment may be empty when `` is 0. + +The encoder decides whether to compress a segment of binary64 +values, bits, or bytes with LZ4, e.g., only when compression saves space. + +### 8.6 Zone Maps + +The zone maps section of the [super footer](#9-super-footer) of a column +super frame holds, for each column frame of the super frame, the minimum +and maximum values of the frame's _columns_, i.e., of its vectors of +primitive values, which a reader can use to skip frames that can't contain +values that match a query. + +The section is the concatenation of a +[length-prefixed](#73-values) container +for each column frame in order. A frame's container holds the +length-prefixed offset of the frame from the start of its super frame, as an +8-byte integer, followed by a zone map for each of its columns, sorted by +column path: +``` +||...| +``` +where each value is length-prefixed and each zone map is +``` +|||||| +``` +where `` is a container of the column's path, the +names of the record fields that lead to the column from the root of the +frame's values, as length-prefixed strings, and `` and +`` are the 2-byte primitive type IDs of the bodies of +the minimum value `` and maximum value ``. + +A column's zone map summarizes the values of all the vectors at its path, +including the values of const and dict vectors. Only values of integer, +float, `duration`, `timestamp`, `bytes`, `string`, `uuid`, `ip`, and `net` +types are summarized, along with enum values as the `uint64` positions of +their symbols. The values of named, error, option, union, and fusion vectors +(other than `fusion(all)`) are summarized at the path of the vector, and the +values of other complex types (e.g., the elements of arrays) are not +summarized. Since the values at a path may have different types (e.g., the +members of a union), a zone map's minimum and maximum may have different +types, which are compared by the order of super-structured values. + +A frame whose root values are not records summarizes them at the empty path. + +## 9. Super Footer + +A super footer ends each super frame: + +| Offset | Size | Field | +|-------:|-----:|------------------------------------------------------------------| +| 0 | 4 | magic, `SUPS` | +| 4 | 2 | version | +| 6 | 8 | footer size, which includes the whole footer and its trailer | +| 14 | 8 | size of the fused type section | +| 22 | 8 | size of the zone maps section | +| 30 | 8 | super frame size, the total size of the frames of the super frame | +| 38 | | fused type section | +| | | zone maps section | +| | 8 | footer size again (the trailer) | + +The fused type section is the [type value](#6-type-values) of the fused type +of all the values of the super frame. A reader can determine the type of a +file without reading its frames by traversing the super footers backward +from the end of the file, fusing their fused types. Starting at the end of +a super footer, its trailer gives the footer's start, and the +super frame size gives the start of its super frame, which is the end of the +previous super footer, if there is one. + +The zone maps section holds the [zone maps](#86-zone-maps) of a super frame +of column frames and is empty for a super frame of row frames. + +An empty super frame (one with no frames) has no super footer. diff --git a/book/src/formats/csup.md b/book/src/formats/csup.md deleted file mode 100644 index 483aa66b95..0000000000 --- a/book/src/formats/csup.md +++ /dev/null @@ -1,400 +0,0 @@ -# Super Column (CSUP) Format - -> _TODO: this is out of date and needs to be updated._ - -Super Columnar (CSUP) is a file format based on -the [super-structured data model](model.md) where data is stacked to form columns. -Its purpose is to provide for efficient analytics and search over -bounded-length sequences of super-structured data -that is stored in columnar form. - -Like [Parquet](https://github.com/apache/parquet-format), -CSUP provides an efficient representation for semi-structured data, -but unlike Parquet, CSUP is not based on schemas and does not require -a schema to be declared when writing data to a file. Instead, -it exploits the nature of super-structured data: columns of data -self-organize around their type structure. - -## CSUP Files - -A CSUP file encodes a bounded, ordered sequence of values. -To provide for efficient access to subsets of CSUP-encoded data (e.g., columns), -the file is presumed to be accessible via random access -(e.g., range requests to a cloud object store or seeks in a Unix file system). - -A CSUP file can be stored entirely as one storage object -or split across separate objects that are treated -together as a single CSUP entity. While the format provides much flexibility -for how data is laid out, it is left to an implementation to lay out data -in intelligent ways for efficient sequential read accesses of related data. - -## Column Streams - -The CSUP data abstraction is built around a collection of _column streams_. - -There is one column stream for each top-level type encountered in the input where -each column stream is encoded according to its type. For top-level complex types, -the embedded elements are encoded recursively in additional column streams -as described below. For example, -a [record column](#record-column) encodes a [presence column](#presence-columns) encoding any null value for -each field then encodes each non-null field recursively, whereas -an [array column](#array-column) encodes a sequence of "lengths" and encodes each -element recursively. - -Values are reconstructed one by one from the column streams by picking values -from each appropriate column stream based on the type structure of the value and -its relationship to the various column streams. For hierarchical records -(i.e., records inside of records, or records inside of arrays inside of records, etc.), -the reconstruction process is recursive (as described below). - -## The Physical Layout - -The overall layout of a CSUP file is comprised of the following sections, -in this order: -* the data section, -* the reassembly section, and -* the trailer. - -This layout allows an implementation to buffer metadata in -memory while writing column data in a natural order to the -data section (based on the volume statistics of each column), -then write the metadata into the reassembly section along with the trailer -at the end. This allows a stream to be converted to a CSUP file -in a single pass. - ->[!NOTE] -> That said, the layout is -> flexible enough that an implementation may optimize the data layout with -> additional passes or by writing the output to multiple files then -> merging them together (or even leaving the CSUP entity as separate files). - -### The Data Section - -The data section contains raw data values organized into _segments_, -where a segment is a seek offset and byte length relative to the -data section. Each segment contains a sequence of -[primitive-type values](model.md#1-primitive-types), -encoded as counted-length byte sequences where the counted-length is -variable-length encoded as in the [Super Binary (BSUP) specification](bsup.md). -Segments may be compressed. - -There is no information in the data section for how segments relate -to one another or how they are reconstructed into columns. They are just -blobs of BSUP data. - ->[!NOTE] -> Unlike Parquet, there is no explicit arrangement of the column chunks into -> row groups but rather they are allowed to grow at different rates so a -> high-volume column might be comprised of many segments while a low-volume -> column must just be one or several. This allows scans of low-volume record types -> (the "mice") to perform well amongst high-volume record types (the "elephants"), -> i.e., there are not a bunch of seeks with tiny reads of mice data interspersed -> throughout the elephants. -> -> The mice/elephants model creates an interesting and challenging layout -> problem. If you let the row indexes get too far apart (call this "skew"), then -> you have to buffer very large amounts of data to keep the column data aligned. -> This is the point of row groups in Parquet, but the model here is to leave it -> up to the implementation to do layout as it sees fit. You can also fall back -> to doing lots of seeks and that might work perfectly fine when using SSDs but -> this also creates interesting optimization problems when sequential reads work -> a lot better. There could be a scan optimizer that lays out how the data is -> read that lives under the column stream reader. Also, you can make tradeoffs: -> if you use lots of buffering on ingest, you can write the mice in front of the -> elephants so the read path requires less buffering to align columns. Or you can -> do two passes where you store segments in separate files then merge them at close -> according to an optimization plan. - -### The Reassembly Section - -The reassembly section provides the information needed to reconstruct -column streams from segments, and in turn, to reconstruct the original values -from column streams, i.e., to map columns back to composite values. - ->[!NOTE] -> Of course, the reassembly section also provides the ability to extract just subsets of columns -> to be read and searched efficiently without ever needing to reconstruct -> the original rows. How well this performs is up to any particular -> CSUP implementation._ -> -> Also, the reassembly section is in general vastly smaller than the data section -> so the goal here isn't to express information in cute and obscure compact forms -> but rather to represent data in an easy-to-digest, programmer-friendly form that -> leverages BSUP. - -The reassembly section is a BSUP stream. Unlike Parquet, -which uses an externally described schema -(via [Thrift](https://thrift.apache.org/)) to describe -analogous data structures, we simply reuse BSUP here. - -#### The Super Types - -This reassembly stream encodes 2*N+1 values, where N is equal to the number -of top-level types that are present in the encoded input. -To simplify terminology, we call a top-level type a "super type", -e.g., there are N unique super types encoded in the CSUP file. - -These N super types are defined by the first N values of the reassembly stream -and are encoded as a null value of the indicated super type. -A super type's integer position in this sequence defines its identifier -encoded in the [super column](#the-super-column). This identifier is called -the super ID. - -> _TODO: Change the first N values to type values instead of nulls?_ - -The next N+1 records contain reassembly information for each of the N super types -where each record defines the column streams needed to reconstruct the original -values. - -#### Segment Maps - -The foundation of column reconstruction is based on _segment maps_. -A segment map is a list of the segments from the data area that are -concatenated to form the data for a column stream. - -Each segment map that appears within the reassembly records is represented -with an array of records that represent seek ranges conforming to this -type signature: -``` -[{offset:uint64,length:uint32,mem_length:uint32,compression_format:uint8}] -``` - -In the rest of this document, we will refer to this type as `` for -shorthand and refer to the concept as a "segmap". - ->[!NOTE] -> We use the type name "segmap" to emphasize that this information represents -> a set of byte ranges where data is stored and must be read from *rather than* -> the data itself. - -#### The Super Column - -The first of the N+1 reassembly records defines the "super column", where this column -represents the sequence of [super types](#the-super-types) of each original value, i.e., indicating -which super type's column stream to select from to pull column values to form -the reconstructed value. -The sequence of super types is defined by each type's super ID (as defined above), -0 to N-1, within the set of N super types. - -The super column stream is encoded as a sequence of BSUP-encoded `int32` primitive values. -While there are a large number of entries in the super column (one for each original row), -the cardinality of super IDs is small in practice so this column -will compress very significantly, e.g., in the special case that all the -values in the CSUP file have the same super ID, -the super column will compress trivially. - -The reassembly map appears as the next value in the reassembly section -and is of type ``. - -#### The Reassembly Records - -Following the root reassembly map are N reassembly maps, one for each unique super type. - -Each reassembly record is a record of type ``, as defined below, -where each reassembly record appears in the same sequence as the original N schemas. -Note that there is no "any" type in the super data model, but rather this terminology is used -here to refer to any of the concrete type structures that would appear -in a given CSUP file. - -In other words, the reassembly record of the super column -combined with the N reassembly records collectively define the original sequence -of data values in the original order. -Taken in pieces, the reassembly records allow efficient access to sub-ranges of the -rows, to subsets of columns of the rows, to sub-ranges of columns of the rows, and so forth. - -This simple top-down arrangement, along with the definition of the other -column structures below, is all that is needed to reconstruct all of the -original data. - ->[!NOTE] -> Each row reassembly record has its own layout of columnar -> values and there is no attempt made to store like-typed columns from different -> schemas in the same physical column. - -The notation `` refers to any instance of the five column types: -* [``](#record-column), -* [``](#array-column), -* [``](#union-column), -* [``](#map-column), or -* [``](#primitive-column). - -Note that when decoding a column, all type information is known -from the super type in question so there is no need -to encode the type information again in the reassembly record. - -#### Record Column - -A `` is defined recursively in terms of the column types of -its fields, i.e., other types that represent arrays, unions, or primitive types -and has the form: -``` -{ - :{column:,presence:}, - :{column:,presence:}, - ... - :{column:,presence:} -} -``` -where -* `` through `` are the names of the top-level fields of the -original row record, -* the `column` fields are column stream definitions for each field, and -* the [`presence` columns](#presence-columns) are `int32` BSUP column streams comprised of a -run-length encoding of the locations of column values in their respective rows, -when there are null values. - -If there are no null values, then the `presence` field contains an empty ``. -If all of the values are null, then the `column` field is null (and the `presence` -contains an empty ``). For an empty ``, there is no -corresponding data stored in the data section. Since a `` is an -array, an empty `` is simply the empty array value `[]`. - -#### Array Column - -An `` has the form: -``` -{values:,lengths:} -``` -where -* `values` represents a continuous sequence of values of the array elements -that are sliced into array values based on the length information, and -* `lengths` encodes an `int32` sequence of values that represent the length -of each array value. - -The `` structure is used for both arrays and sets. - -#### Map Column - -A `` has the form: -``` -{key:,value:} -``` -where -* `key` encodes the column of map keys and -* `value` encodes the column of map values. - -#### Union Column - -A `` has the form: -``` -{columns:[],tags:} -``` -where -* `columns` is an array containing the reassembly information for each tagged union value -in the same column order implied by the union type, and -* `tags` is a column of `int32` values where each subsequent value encodes -the tag of the union type indicating which column the value falls within. - -> _TODO: Change code to conform to columns array instead of record{c0,c1,...}_ - -The number of times each value of `tags` appears must equal the number of values -in each respective column. - -#### Primitive Column - -A `` is a `` that defines a column stream of -primitive values. - -#### Presence Columns - -The presence column is logically a sequence of booleans, one for each position -in the original column, indicating whether a value is null or present. -The number of values in the encoded column is equal to the number of values -present so that null values are not encoded. - -Instead the presence column is encoded as a sequence of alternating runs. -First, the number of values present is encoded, then the number of values not present, -then the number of values present, and so forth. These runs are then stored -as `int32` values in the presence column (which may be subject to further -compression based on segment compression). - -### The Trailer - -After the reassembly section is a BSUP stream with a single record defining -the "trailer" of the CSUP file. The trailer provides a magic field -indicating the file format, a version number, -the size of the segment threshold for decomposing segments into frames, -the size of the skew threshold for flushing all segments to storage when -the memory footprint roughly exceeds this threshold, -and an array of sizes in bytes of the sections of the CSUP file. - -This type of this record has the format -``` -{magic:string,type:string,version:int64,sections:[int64],meta:{skew_thresh:int64,segment_thresh:int64}} -``` -The trailer can be efficiently found by scanning backward from the end of the -CSUP file to find a valid BSUP stream containing a single record value -conforming to the above type. - -## Decoding - -To decode an entire CSUP file into rows, the trailer is read to find the sizes -of the sections, then the BSUP stream of the reassembly section is read, -typically in its entirety. - -Since this data structure is relatively small compared to all of the columnar -data in the file, -it will typically fit comfortably in memory and it can be very fast to scan the -entire reassembly structure for any purpose. - ->[!NOTE] -> For a given query, a "scan planner" could traverse all the -> reassembly records to figure out which segments will be needed, then construct -> an intelligent plan for reading the needed segments and attempt to read them -> in mostly sequential order, which could serve as -> an optimizing intermediary between any underlying storage API and the -> CSUP decoding logic. - -To decode the "next" row, its schema index is read from the root reassembly -column stream. - -This schema index then determines which reassembly record to fetch -column values from. - -The top-level reassembly fetches column values as a ``. - -For any ``, a value from each field is read from each field's column, -accounting for the presence column indicating null, -and the results are encoded into the corresponding BSUP record value using -type information from the corresponding schema. - -For a `` a value is determined by reading the next -value from its segmap. - -For an ``, a length is read from its `lengths` segmap as an `int32` -and that many values are read from its the `values` sub-column, -encoding the result as a BSUP array value. - -For a ``, a value is read from its `tags` segmap -and that value is used to select the corresponding column stream -`c0`, `c1`, etc. The value read is then encoded as a BSUP union value -using the same tag within the union value. - -## Examples - -### Hello, world - -Start with this [Super (SUP)](sup.md) file `hello.sup`: -``` -{a:"hello",b:"world"} -{a:"goodnight",b:"gracie"} -``` - -To convert to CSUP format: -``` -super -f csup hello.sup > hello.csup -``` - -Segments in the CSUP format would be laid out like this: -``` -=== column for a -hello -goodnight -=== column for b -world -gracie -=== column for schema IDs -0 -0 -=== -``` diff --git a/book/src/formats/intro.md b/book/src/formats/intro.md index 9c9ef34e2b..b05f786dee 100644 --- a/book/src/formats/intro.md +++ b/book/src/formats/intro.md @@ -6,17 +6,15 @@ providing a unified approach to row, columnar, and human-readable formats: * [Super (SUP)](sup.md) is a human-readable format for super-structured data. All JSON documents are SUP values as the SUP format is a strict superset of the JSON syntax. -* [Super Binary (BSUP)](bsup.md) is a row-based, binary representation somewhat like -[Avro](https://avro.apache.org/) but leveraging the super data model to represent -a sequence of arbitrarily-typed values. -* [Super Columnar (CSUP)](csup.md) is columnar like +* [Super Binary (BSUP)](bsup.md) is an efficient binary representation of +a sequence of arbitrarily-typed values in row form, somewhat like +[Avro](https://avro.apache.org/), or columnar form, somewhat like [Parquet](https://parquet.apache.org/), [ORC](https://orc.apache.org/), or -[Arrow](https://arrow.apache.org/) but for super-structured data. +[Arrow](https://arrow.apache.org/), but for super-structured data. Because all of the formats conform to the same super-structured data model, -conversions between a human-readable form, a row-based binary form, -and a row-based columnar form can +conversions between a human-readable form and a binary row or columnar form can be carried out with no loss of information. This provides the best of both worlds: the same data can be easily expressed in and converted between a human-friendly and easy-to-program text form alongside efficient row and columnar formats. diff --git a/book/src/formats/model.md b/book/src/formats/model.md index 7c4af0c269..bc10a556cf 100644 --- a/book/src/formats/model.md +++ b/book/src/formats/model.md @@ -1,59 +1,101 @@ # Super-structured Data Model Super-structured data is a collection of one or more typed data values. -Each value's type is either a "primitive type", a "complex type", the "type type", -a "named type", or the "null type". +Each value's type is either a "primitive type", a "complex type", +or a "named type". ## 1. Primitive Types -Primitive types include signed and unsigned integers, IEEE binary and decimal -floating point, string, byte sequence, Boolean, IP address, IP network, -null, and a first-class type _type_. - -There are 30 types of primitive values defined as follows: - -| Name | Definition | -|------------|-------------------------------------------------| -| `uint8` | unsigned 8-bit integer | -| `uint16` | unsigned 16-bit integer | -| `uint32` | unsigned 32-bit integer | -| `uint64` | unsigned 64-bit integer | -| `uint128` | unsigned 128-bit integer | -| `uint256` | unsigned 256-bit integer | -| `int8` | signed 8-bit integer | -| `int16` | signed 16-bit integer | -| `int32` | signed 32-bit integer | -| `int64` | signed 64-bit integer | -| `int128` | signed 128-bit integer | -| `int256` | signed 256-bit integer | -| `duration` | signed 64-bit integer as nanoseconds | -| `time` | signed 64-bit integer as nanoseconds from epoch | -| `float16` | IEEE-754 binary16 | -| `float32` | IEEE-754 binary32 | -| `float64` | IEEE-754 binary64 | -| `float128` | IEEE-754 binary128 | -| `float256` | IEEE-754 binary256 | -| `decimal32` | IEEE-754 decimal32 | -| `decimal64` | IEEE-754 decimal64 | -| `decimal128` | IEEE-754 decimal128 | -| `decimal256` | IEEE-754 decimal256 | -| `bool` | the Boolean value `true` or `false` | -| `bytes` | a bounded sequence of 8-bit bytes | -| `string` | a UTF-8 string | -| `ip` | an IPv4 or IPv6 address | -| `net` | an IPv4 or IPv6 address and net mask | -| `type` | a type value | -| `null` | the null type | - -The _type_ type provides for first-class types. Even though a type value can +Primitive types include signed and unsigned integers, IEEE binary +floating point, durations and timestamps, string, byte sequence, Boolean, +IP address, IP network, UUID, the first-class type _type_, and a few +special types (`null`, `missing`, `none`, `all`, and `any`) +described below. + +The primitive types are defined as follows: + +| Name | Definition | +|--------------------|-------------------------------------------------| +| `uint8` | unsigned 8-bit integer | +| `uint16` | unsigned 16-bit integer | +| `uint32` | unsigned 32-bit integer | +| `uint64` | unsigned 64-bit integer | +| `uint128` | unsigned 128-bit integer (reserved) | +| `int8` | signed 8-bit integer | +| `int16` | signed 16-bit integer | +| `int32` | signed 32-bit integer | +| `int64` | signed 64-bit integer | +| `int128` | signed 128-bit integer (reserved) | +| `duration` | signed 64-bit integer as nanoseconds | +| `timestamp(0)` | timestamp with 0 fractional digits (reserved) | +| `timestamp(3)` | timestamp with 3 fractional digits (reserved) | +| `timestamp(6)` | timestamp with 6 fractional digits (reserved) | +| `timestamp` | signed 64-bit integer as nanoseconds from the Unix epoch (equivalently `timestamp(9)`) | +| `timestamp_ntz(0)` | timestamp without time zone with 0 fractional digits (reserved) | +| `timestamp_ntz(3)` | timestamp without time zone with 3 fractional digits (reserved) | +| `timestamp_ntz(6)` | timestamp without time zone with 6 fractional digits (reserved) | +| `timestamp_ntz` | timestamp without time zone with 9 fractional digits (reserved) | +| `date` | calendar date (reserved) | +| `time(0)` | time of day with 0 fractional digits (reserved) | +| `time(3)` | time of day with 3 fractional digits (reserved) | +| `time(6)` | time of day with 6 fractional digits (reserved) | +| `time` | time of day with 9 fractional digits (reserved) | +| `float16` | IEEE-754 binary16 | +| `float32` | IEEE-754 binary32 | +| `float64` | IEEE-754 binary64 | +| `bool` | the Boolean value `true` or `false` | +| `bytes` | a bounded sequence of 8-bit bytes | +| `string` | a UTF-8 string | +| `ip` | an IPv4 or IPv6 address | +| `net` | an IPv4 or IPv6 address and net mask | +| `interval` | a calendar interval (reserved) | +| `uuid` | a 128-bit universally unique identifier | +| `geometry` | a geometric shape (reserved) | +| `geography` | a geographic shape (reserved) | +| `type` | a type value | +| `null` | the type of the `null` value | +| `missing` | the type of the `missing` value | +| `none` | the type of the `none` value | +| `all` | the type of a value of any type held by a [fusion](#28-fusion) | +| `any` | the top type, to which every type conforms | + +A type marked "reserved" is part of the type system, so it may appear in +a type, but its values are not yet defined. The time zone of a +`timestamp_ntz`, `date`, or `time` value is unspecified. + +The _type_ type provides for first-class types. Even though a type value can represent a complex type, the value itself is a singleton. -Two type values are equivalent if their underlying types are equal. Since -every type in the type system is uniquely defined, type values are equal -if and only if their corresponding types are uniquely equal. +Since every type in the type system is uniquely defined, two type values +are equal if and only if the types they represent are equal. -The _null_ type is a primitive type representing only a `null` value. -A `null` value can have any type. +### 1.1 Null, Missing, and None + +The _null_ type is a primitive type whose only value is `null`. +Since `null` has its own type, a value that is either a value of some +type `T` or null is a [union](#25-union) of `T` and `null`, +e.g., a nullable string has type `string|null`. + +The _missing_ type is a primitive type whose only value is `missing`, +which represents the absence of a value, e.g., in a record type formed by +[type fusion](../super-sql/type-fusion.md), the value of +a field that is absent from the original record. + +The _none_ type is a primitive type whose only value is `none`. Its +primary purpose is to be cast to an [option](#29-option) type, yielding +that option's "none" value. It is also the element type of an empty array +or set and the key and value type of an empty map whose types are not +otherwise known. + +### 1.2 All and Any + +The _any_ type is the top type of the type system: every type conforms +to `any`, but no value has type `any`. A value cast to `any` has type +[`fusion(all)`](#28-fusion), which holds a value of any type along with its +type. + +The _all_ type is the type of the value held by a `fusion(all)` value. +It appears only within the type `fusion(all)`. ## 2. Complex Types @@ -66,12 +108,16 @@ The categories of complex types include: uniform type called the key type and the values are of a uniform type called the value type, * _union_ - a type representing values whose type is any of a specified collection of two or more unique types, -* _enum_ - a type representing a finite set of symbols typically representing categories, and -* _error_ - any value wrapped as an "error". +* _enum_ - a type representing a finite set of symbols typically representing categories, +* _error_ - any value wrapped as an "error", +* _fusion_ - a value of a supertype together with the type of the value it represents, +* _option_ - a value that is either present or absent, and +* _decimal_ - a fixed-point decimal number of a given precision and scale. The type system comprises a total order: -* The order of primitive types corresponds to the order in the table above. +* All [named types](#3-named-type) are ordered before any other types. * All primitive types are ordered before any complex types. +* The order of primitive types corresponds to the order in the table above. * The order of complex type categories corresponds to the order above. * For complex types of the same category, the order is defined below. @@ -86,41 +132,42 @@ A field name is any UTF-8 string. A field value is a value of any type. -A field is either mandatory or optional as indicated by its optionality. -The optionalities of fields in two records must be the same for those -record types to be equivalent, e.g., type `{a:string,b?:string}` is distinct -from `{a:string,b:string}`. +A field that may be absent from a record value is an _optional field_, +whose type is an [option](#29-option) type. When the field's value is +the option's none value, the field is logically absent but still present +in the value's type. A record type is uniquely defined by its ordered list of field-type pairs. The type order of two records is as follows: -* Record with fewer columns than other is ordered before the other. -* Records with the same number of columns are ordered as follows according to: +* A record type with fewer fields than the other is ordered before the other. +* Record types with the same number of fields are ordered according to * the lexicographic order of the field names from left to right, - * secondarily the optionality of a field name for two names that are the same, - * or if all the field names and optionalities are the same, the type order of the field types from left to right. + * or if all the field names are the same, the type order of the field types from left to right. ### 2.2 Array An array is an ordered sequence of zero or more values called "elements" all conforming to the same type. -An array value may be empty. An empty array may have element type `null`. +An array value may be empty. An empty array whose type is not otherwise +known has element type `none`. An array type is uniquely defined by its single element type. The type order of two arrays is defined as the type order of the two array element types. -An array of mixed-type values (such a mixed-type JSON array) is representable +An array of mixed-type values (such as a mixed-type JSON array) is representable as an array with elements of type `union`. ### 2.3 Set -A set is an unordered sequence of zero or more values called "elements" +A set is an unordered collection of zero or more unique values called "elements" all conforming to the same type. -A set may be empty. An empty set may have element type `null`. +A set may be empty. An empty set whose type is not otherwise known has +element type `none`. A set of mixed-type values is representable as a set with elements of type `union`. @@ -132,12 +179,13 @@ two set element types. ### 2.4 Map -A map represents a list of zero or more key-value pairs, where the keys +A map represents a collection of zero or more key-value pairs, where the keys have a common type and the values have a common type. -Each key across an instance of a map value must be a unique value. +The keys of a map value must be unique. -A map value may be empty. +A map value may be empty. An empty map whose type is not otherwise known +has key type and value type `none`. A map type is uniquely defined by its key type and value type. @@ -147,35 +195,37 @@ The type order of two map types is as follows: ### 2.5 Union -A union represents a value that may be any one of a specific enumeration -of two or more unique data types that comprise its "union type". +A union represents a value that may be any one of a specific set +of two or more unique types that comprise its "union type". -A union type is uniquely defined by an ordered set of unique types (which may be -other union types) where the order corresponds to the type system's total order. +A union type is uniquely defined by an ordered set of two or more unique types +where the order corresponds to the type system's total order. +A union's types may not themselves be union types, though they may be +named types whose underlying types are unions. Union values are tagged in that -any instance of a union value explicitly conforms to exactly one of the union's types. +each union value explicitly conforms to exactly one of the union's types. The union tag is an integer indicating the position of its type in the union type's ordered list of types. The type order of two union types is as follows: -* The union type with fewer types than other is ordered before the other. +* The union type with fewer types than the other is ordered before the other. * Two union types with the same number of types are ordered according to the type order of the constituent types in left to right order. ### 2.6 Enum An enum represents a symbol from a finite set of one or more unique symbols -referenced by name. An enum name may be any UTF-8 string. +referenced by name. An enum symbol may be any UTF-8 string. An enum type is uniquely defined by its ordered set of unique symbols, where the order is significant, e.g., two enum types with the same set of symbols but in different order are distinct. The type order of two enum types is as follows: -* The enum type with fewer symbols than other is ordered before the other. +* The enum type with fewer symbols than the other is ordered before the other. * Two enum types with the same number of symbols are ordered according to -the type order of the constituent types in left to right order. +the lexicographic order of their symbols from left to right. The order among enum values corresponds to the order of the symbols in the enum type. Order among enum values from different types is undefined. @@ -184,24 +234,74 @@ Order among enum values from different types is undefined. An error represents any value designated as an error. -The type order of an error is the type order of the type of its contained value. +An error type is uniquely defined by the type of its value. + +The type order of two error types is the type order of the types of +their values. + +### 2.8 Fusion + +A fusion value is a value of a type called the fusion's _supertype_ together +with the type of the value it represents, called its _subtype_. The +represented value is recovered by converting the value to its subtype. + +Fusion types are created by [type fusion](../super-sql/type-fusion.md), +which allows values of many different types to be stored as values of +a single fused type without losing their original types. +For example, the values `{a:1}` and `{b:"x"}` may be stored as +fusion values of type `fusion({a:int64|missing,b:string|missing})` +whose subtypes are `{a:int64}` and `{b:string}`. + +The type `fusion(all)` holds a value of any type, whose subtype is +the value's type. + +A fusion type is uniquely defined by its supertype. + +The type order of two fusion types is the type order of their supertypes. + +### 2.9 Option + +An option represents a value of a given type that is either present +(called "some") or absent (called "none"). The type of an option may +not be an option type, `missing`, or `none`. + +An option type is uniquely defined by the type of its value. + +An [optional record field](#21-record) is a field whose type is an +option type. + +The type order of two option types is the type order of the types of +their values. + +### 2.10 Decimal + +A decimal type is a fixed-point decimal number with a _precision_ +(the total number of decimal digits, from 1 to 38) and +a _scale_ (the number of those digits to the right of the decimal point, +which is no greater than the precision). + +A decimal type is uniquely defined by its precision and scale. +Decimal values are reserved and not yet defined. + +The type order of two decimal types is as follows: +* the order of their precisions, +* or if they are the same, then the order of their scales. ## 3. Named Type A _named type_ is a name for a specific data type. Any value can have a named type and the named type is a distinct type -from the underlying type. A named type can refer to another named type. +from its underlying type. A named type can refer to another named type +and, since a named type may refer to itself, recursive types may be defined. -The binding between a named type and its underlying type is local in scope -and need not be unique across a sequence of values. +The binding between a named type and its underlying type is unique +within a sequence of values: a type name may be bound to only one type. -A type name may be any UTF-8 string exclusive of primitive type names. +A type name may be any UTF-8 string other than the name of a primitive type. For example, if "port" is a named type for `uint16`, then two values of type "port" have the same type but a value of type "port" and a value of type `uint16` do not have the same type. -The type order of a named type is the type order of its underlying type with two -exceptions: -* A named type is ordered after its underlying type. -* Named types sharing an underlying type are ordered lexicographically by name. +Named types are ordered before all other types and are +ordered lexicographically by name. diff --git a/book/src/formats/sup.md b/book/src/formats/sup.md index 2272249929..ec3fe75b02 100644 --- a/book/src/formats/sup.md +++ b/book/src/formats/sup.md @@ -5,7 +5,7 @@ Super (SUP) is the human-readable, text-based serialization format for [super-structured data](model.md). -SUP builds upon the elegant simplicity of JSON with [type decorators](#22-type-decorators) +SUP builds upon the elegant simplicity of JSON with [type decorators](#22-type-decorators). Where the type of a value is not implied by its syntax, a type decorator is appended to the value interposed with double colons to establish a concrete type for every value expressed in source text. @@ -18,7 +18,8 @@ SUP values. A SUP text is a sequence of UTF-8 characters organized either as a bounded input or an unbounded stream. -The input text is organized as a sequence of one or more values optionally +The input text is organized as a sequence of one or more values and +[type declarations](#27-type-declarations) optionally separated by and interspersed with whitespace. Single-line (`//`) and multi-line (`/* ... */`) comments are treated as whitespace and ignored. @@ -34,8 +35,7 @@ A name is either an _identifier_ or a [quoted string](#231-strings). Names are referred to as `` below. An _identifier_ is case-sensitive and can contain Unicode letters, `$`, `_`, -and digits `[0-9]`, but may not start with a digit. An identifier cannot be -`true`, `false`, or `null`. +and digits `[0-9]`, but may not start with a digit. ### 2.2 Type Decorators @@ -62,27 +62,18 @@ The syntax of a union value decorator is where the rightmost type must be a union type if more than one decorator is present. -A decorator may also define a [named type](#258-named-type): +A decorator of a complex value applies to the values within it, e.g., ``` -::= +{p:80}::{p:uint16} ``` -which declares a new type with the indicated type name using the -implied type of the value. Type names may not be numeric, where a -numeric is a sequence of one or more characters in the set `[0-9]`. - -A decorator may also define a temporary numeric reference of the form: -``` -::= -``` -Once defined, this numeric reference may then be used anywhere a named type -is used but a named type is not created. +is the same as `{p:80::uint16}`. A record decorator matches the fields of +the record value by name, and a field of the decorator that is +[optional](#249-option-value) and absent from the record value is none, +e.g., `{a:1}::{a:int64,b?:string}` is the same as +`{a:1,b?:none::string}`. It is an error for the decorator to be type incompatible with its referenced value. -Note that the `=` sigil here disambiguates between the case that a new -type is defined, which may override a previous definition of a different type with the -same name, from the case that an existing named type is merely decorating the value. - ### 2.3 Primitive Values The type names and format for @@ -94,32 +85,29 @@ The type names and format for | `uint16` | decimal string representation of any unsigned, 16-bit integer | | `uint32` | decimal string representation of any unsigned, 32-bit integer | | `uint64` | decimal string representation of any unsigned, 64-bit integer | -| `uint128` | decimal string representation of any unsigned, 128-bit integer | -| `uint256` | decimal string representation of any unsigned, 256-bit integer | | `int8` | decimal string representation of any signed, 8-bit integer | | `int16` | decimal string representation of any signed, 16-bit integer | | `int32` | decimal string representation of any signed, 32-bit integer | | `int64` | decimal string representation of any signed, 64-bit integer | -| `int128` | decimal string representation of any signed, 128-bit integer | -| `int256` | decimal string representation of any signed, 256-bit integer | | `duration` | a _duration string_ representing signed 64-bit nanoseconds | -| `time` | an RFC 3339 UTC date/time string representing signed 64-bit nanoseconds from epoch | +| `timestamp` | an RFC 3339 date/time string representing signed 64-bit nanoseconds from epoch | | `float16` | a _non-integer string_ representing an IEEE-754 binary16 value | | `float32` | a _non-integer string_ representing an IEEE-754 binary32 value | | `float64` | a _non-integer string_ representing an IEEE-754 binary64 value | -| `float128` | a _non-integer string_ representing an IEEE-754 binary128 value | -| `float256` | a _non-integer string_ representing an IEEE-754 binary256 value | -| `decimal32` | a _non-integer string_ representing an IEEE-754 decimal32 value | -| `decimal64` | a _non-integer string_ representing an IEEE-754 decimal64 value | -| `decimal128` | a _non-integer string_ representing an IEEE-754 decimal128 value | -| `decimal256` | a _non-integer string_ representing an IEEE-754 decimal256 value | | `bool` | the string `true` or `false` | | `bytes` | a sequence of bytes encoded as a hexadecimal string prefixed with `0x` | | `string` | a double-quoted UTF-8 string | | `ip` | a string representing an IP address in [IPv4 or IPv6 format](https://tools.ietf.org/html/draft-main-ipaddr-text-rep-02#section-3) | | `net` | a string in CIDR notation representing an IP address and prefix length as defined in RFC 4632 and RFC 4291. | -| `type` | a string in canonical form as described in [Section 2.5](#25-types) | +| `uuid` | a string of 32 hexadecimal digits in five groups separated by hyphens (8-4-4-4-12) as defined in RFC 9562 | +| `type` | a type in canonical form as described in [Section 2.5](#25-types) enclosed in angle brackets | | `null` | the string `null` | +| `missing` | the string `missing` | +| `none` | the string `none` | + +The [reserved primitive types](model.md#1-primitive-types) +(e.g., `uint128`, `timestamp(3)`, and `date`) have no value format +as their values are not yet defined. Their names may appear in types. The format of a _duration string_ is an optionally-signed concatenation of decimal numbers, @@ -149,16 +137,16 @@ also be one of: A floating point value may be expressed with an integer string provided a type decorator is applied, e.g., `123::float64`. -Decimal values require type decorators. - -Of the 30 primitive types, eleven of them represent _implied-type_ values: -`int64`, `time`, `duration`, `float64`, `bool`, `bytes`, `string`, `ip`, `net`, `type`, and `null`. +The following primitive types represent _implied-type_ values: +`int64`, `uint64`, `duration`, `timestamp`, `float64`, `bool`, `bytes`, +`string`, `ip`, `net`, `uuid`, `type`, `null`, `missing`, and `none`. Values for these types are determined by the format of the value and thus do not need decorators to clarify the underlying type, e.g., ``` 123::int64 ``` -is the same as `123`. +is the same as `123`. An integer string is implied to be an `int64` unless +it is too large, in which case it is implied to be a `uint64`. Values that do not have implied types must include a type decorator to clarify its type or appear in a context for which its type is defined (i.e., as a field @@ -170,7 +158,7 @@ and thus always a primitive type. A `type` value is encoded as: * a type as [encoded below](#25-types), followed by * a right angle bracket `>`. -A `time` value corresponds to 64-bit Unix epoch nanoseconds and thus +A `timestamp` value corresponds to 64-bit Unix epoch nanoseconds and thus not all possible RFC 3339 date/time strings are valid. In addition, nanosecond epoch times overflow on April 11, 2262. @@ -203,7 +191,7 @@ is not a valid escape sequence. The behavior of an implementation that encounters such invalid sequences in a `string` type is undefined. These escaping rules apply also to quoted field names in record values and -record types as well as enum symbols. +record types, enum symbols, and type names. ### 2.4 Complex Values @@ -214,8 +202,10 @@ and conform to the super data model's complex types: [set](model.md#23-set), [map](model.md#24-map), [union](model.md#25-union), -[enum](model.md#26-enum), and -[error](model.md#27-error). +[enum](model.md#26-enum), +[error](model.md#27-error), +[option](model.md#29-option), and +[fusion](model.md#28-fusion). Complex values have an implied type when their constituent values all have implied types. @@ -231,6 +221,11 @@ any optionally-decorated value inclusive of other records. Each name/value pair is called a _field_. There may be zero or more fields. +An [optional field](#249-option-value) has the form +``` + ?: +``` + #### 2.4.2 Array Value An array value has the form: @@ -241,27 +236,27 @@ If the elements of the array are not of uniform type, then the implied type of the array elements is a union of the types present. An array value may be empty. An empty array value without a type decorator is -presumed to be an empty array of type `null`. +presumed to be an empty array of type `[none]`. #### 2.4.3 Set Value A set value has the form: ``` -|[ , , ... ]| +set[ , , ... ] ``` -where the indicated values must be distinct. +where duplicate values are removed. If the elements of the set are not of uniform type, then the implied type of the set elements is a union of the types present. A set value may be empty. An empty set value without a type decorator is -presumed to be an empty set of type `null`. +presumed to be an empty set of type `set[none]`. #### 2.4.4 Map Value A map value has the form: ``` -|{ : , : , ... }| +map{ : , : , ... } ``` where zero or more comma-separated, key/value pairs are present. @@ -270,7 +265,7 @@ avoid ambiguity, whitespace must separate an IPv6 key from the colon that follows it. An empty map value without a type decorator is -presumed to be an empty map of type `|{null: null}|`. +presumed to be an empty map of type `map{none:none}`. #### 2.4.5 Union Value @@ -296,7 +291,8 @@ contained enum type is defined by the complex type's decorator. A sequence of enum values might look like this: ``` -"HEADS"::(flip=(enum(HEADS,TAILS))) +type flip=enum(HEADS,TAILS) +"HEADS"::flip "TAILS"::flip "HEADS"::flip ``` @@ -309,10 +305,56 @@ error() ``` where `` is any value. +#### 2.4.8 Fusion Value + +A fusion value has the form: +``` +fusion(, ) +``` +where `` is a value of the fusion's supertype and `` +is a [type value](#23-primitive-values) of its subtype, e.g., +``` +fusion(1::(int64|string), ) +``` + +A value of type `fusion(all)` has the form: +``` +::any +``` +where the value's subtype is the type of ``. + +#### 2.4.9 Option Value + +An option value whose value is present has the form: +``` +some() +``` +where `` may not be an option value, +and an option value whose value is absent has the form: +``` +none:: +``` +where `` is the type of the option's value or the option type itself, +e.g., `none::int64` and `none::option(int64)` are the same value. +A value whose type is not an option type +may also be decorated with an option type, e.g., `1::option(int64)` is +the same as `some(1)`. + +In a record value, an optional field `?:` is a field whose +value is an option value, where a `` that is not an option value +is the same as `some()`, e.g., +``` +{a?:1,b?:none::string} +``` +is the same as `{a:some(1),b:none::string}`. + ### 2.5 Types A primitive type is simply the name of the primitive type, i.e., `string`, -`uint16`, etc. Complex types are defined as follows. +`uint16`, etc. The `timestamp`, `timestamp_ntz`, and `time` types +may include a precision of 0, 3, 6, or 9 fractional digits, e.g., +`timestamp(3)`, where `timestamp` is the same as `timestamp(9)`. +Complex types are defined as follows. #### 2.5.1 Record Type @@ -326,6 +368,9 @@ where `` is a [SUP name](#21-names) and The order of the record fields is significant, e.g., type `{a:int32,b:int32}` is distinct from type `{b:int32,a:int32}`. +An optional field has the form ` ?: `, which is the same as +` : option()`. + #### 2.5.2 Array Type An _array type_ has the form: @@ -337,14 +382,14 @@ An _array type_ has the form: A _set type_ has the form: ``` -|[ ]| +set[ ] ``` #### 2.5.4 Map Type A _map type_ has the form: ``` -|{ : }| +map{ : } ``` where `` is the type of the keys and `` is the type of the values. @@ -353,9 +398,17 @@ type of the values. A _union type_ has the form: ``` -( , , ... ) + | | ... ``` -where there are at least two types in the list. +where there are at least two types in the list and +none is itself a union type. +A union type may be enclosed in parentheses, which are needed +where a union type would otherwise be ambiguous, e.g., when a decorator +is a union type as in `1::(int64|string)`. + +Since a union type is an ordered set of types in the type system's order, +the order of the types in the list is not significant, e.g., +`string|int64` is the same type as `int64|string`. #### 2.5.6 Enum Type @@ -377,35 +430,89 @@ where `` is the type of the underlying values wrapped as an error. #### 2.5.8 Named Type -A named type has the form: +A named type is referenced by its name, which is a [SUP name](#21-names), +and is defined by a [type declaration](#27-type-declarations). + +#### 2.5.9 Fusion Type + +A _fusion type_ has the form: ``` - = +fusion( ) ``` -where a new type is defined with the given name and type. +where `` is the fusion's supertype. -When a named type appears in a complex value, the new type name may be -referenced by any subsequent value in left-to-right depth-first order. +#### 2.5.10 Option Type -For example, +An _option type_ has the form: ``` -{p1:80::(port=uint16), p2: 8080::port} +option( ) ``` -is valid but +where `` is the type of the option's value. + +#### 2.5.11 Decimal Type + +A _decimal type_ has the form: ``` -{p1:80::port, p2: 8080::(port=uint16)} +decimal( [, ] ) ``` -is invalid. +where `` and `` are integers and the scale defaults to 0. + +### 2.6 Null, Missing, and None Values -Named types may be redefined, in which case subsequent references -resolve to the most recent definition according to -* sequence order across values, or -* left-to-right depth-first order within a complex value. +The [null value](model.md#11-null-missing-and-none) is represented by +the string `null` and has type `null`. A null value may be decorated +with `null` or with a union type that includes `null`, e.g., +``` +null::(int64|null) +``` -### 2.6 Null Value +The missing value is represented by the string `missing` and +has type `missing`. -The null value is represented by the string `null`. +The none value is represented by the string `none` and +has type `none`. When decorated with a type that is not `none`, +it is an [option value](#249-option-value). + +### 2.7 Type Declarations + +A _type declaration_ binds a name to a type and has the form +``` +type = +``` +where `` is a [SUP name](#21-names) and `` is any type. +A type name may be any name other than the name of a +[primitive type](#23-primitive-values) (including `all` and `any`). + +Type declarations appear in the sequence of values before or between values +but never inside a value, e.g., `{a:type x=int64 1}` is invalid. +A declaration binds the name for the value that follows it and for +all subsequent values in the sequence, so a type is declared once +and then referenced by name in any number of values, e.g., +``` +type port=uint16 +{src:80::port,dst:8080::port} +{src:443::port,dst:9000::port} +``` -A value of any type can be null. +A type name may be bound to only one type in a sequence of values. +A name may be declared again with the same type, but it is an error to +declare a name with a different type. + +The declarations preceding a value are bound together, so they may refer to +each other in any order and to themselves, which allows recursive and +mutually recursive types, e.g., +``` +type tree={value:int64,children:forest} +type forest=[tree] +{value:1,children:[{value:2,children:[]}]}::tree +``` +A declaration may not refer to a name that is declared only after a +subsequent value. Declarations at the end of the input that are not followed +by a value have no effect. + +When an implementation formats values as SUP, it emits the declaration of +each named type in a value's type (including the types of its type values) +before the first value in which the named type appears. ## 3. Examples @@ -417,15 +524,13 @@ There's no need for a type declaration here. It's explicitly a string. A relational table might look like this: ``` -{ city: "Berkeley", state: "CA", population: 121643::uint32 }::=city_schema -{ city: "Broad Cove", state: "ME", population: 806::uint32 }::=city_schema -{ city: "Sioux Falls", state: "SD", population: 192517::uint32 }::=city_schema -``` -The text here depicts three record values. It defines a type called `city_schema` -and the inferred type of the `city_schema` has the signature: -``` -{ city:string, state:string, population:uint32 } +type city_schema={city:string,state:string,population:uint32} +{city:"Berkeley",state:"CA",population:121643}::city_schema +{city:"Broad Cove",state:"ME",population:806}::city_schema +{city:"Sioux Falls",state:"SD",population:192517}::city_schema ``` +The text here depicts three record values. It declares a type called `city_schema` +and decorates each record value with the type. When all the values in a sequence have the same record type, the sequence can be interpreted as a _table_, where record values form the _rows_ and the fields of the records form the _columns_. In this way, these @@ -434,42 +539,45 @@ three records form a relational table conforming to the schema `city_schema`. In contrast, text representing a semi-structured sequence of log lines might look like this: ``` +type socket={addr:ip,port:uint16} +type conn={info:string,src:socket,dst:socket} +type access_list={info:string,nets:[net]} { info: "Connection Example", - src: { addr: 10.1.1.2, port: 80::uint16 }::=socket, - dst: { addr: 10.0.1.2, port: 20130::uint16 }::=socket -}::=conn + src: { addr: 10.1.1.2, port: 80 }, + dst: { addr: 10.0.1.2, port: 20130 } +}::conn { info: "Connection Example 2", - src: { addr: 10.1.1.8, port: 80::uint16 }::=socket, - dst: { addr: 10.1.2.88, port: 19801::uint16 }::=socket -}::=conn + src: { addr: 10.1.1.8, port: 80 }, + dst: { addr: 10.1.2.88, port: 19801 } +}::conn { info: "Access List Example", nets: [ 10.1.1.0/24, 10.1.2.0/24 ] -}::=access_list +}::access_list { metric: "A", ts: 2020-11-24T08:44:09.586441-08:00, value: 120 } { metric: "B", ts: 2020-11-24T08:44:20.726057-08:00, value: 0.86 } { metric: "A", ts: 2020-11-24T08:44:32.201458-08:00, value: 126 } { metric: "C", ts: 2020-11-24T08:44:43.547506-08:00, value: { x:10, y:101 } } ``` -In this case, the first record defines not just a record type -with named type `conn`, but also a second embedded record type called `socket`. -Decorators are used where a type is not inferred from -the value itself: +In this case, the type declarations define a record type called `socket`, +which is embedded in a second record type called `conn`: * `socket` is a record with typed fields `addr` and `port` where `port` is an unsigned 16-bit integer, and * `conn` is a record with typed fields `info`, `src`, and `dst`. -The subsequent value defines a type called `access_list`. In this case, -the `nets` field is an array of networks and illustrates the helpful range of -primitive types. Note that the syntax here implies -the type of the array, as it is inferred from the type of the elements. +The decorator of each `conn` value establishes the type of its fields, so +the `port` values need no decorators of their own. + +The `access_list` type is a record whose +`nets` field is an array of networks and illustrates the helpful range of +primitive types. Finally, there are four more values that show SUP's efficacy for representing metrics. Here, there are no type decorators as all of the field types are implied by their syntax, and hence, the top-level record type is implied. For instance, the `ts` field is an RFC 3339 date and time string, -unambiguously the primitive type `time`. Further, +unambiguously the primitive type `timestamp`. Further, note that the `value` field takes on different types and even a complex record type on the last line. In this case, there is a different top-level record type implied by each of the three variations of type of the `value` field. @@ -482,17 +590,18 @@ For example, union and enum values must both appear in a context that defines their type. ``` - = | | + = | - = . + = | - = | | + = "type" "=" - = "::=" + = | = "::" | "::" - = | | | | | | + = | | | | | | + | | = primitive value as defined above @@ -500,7 +609,7 @@ that defines their type. = "," | - = ":" + = ":" | "?:" = | @@ -512,11 +621,9 @@ that defines their type. = "," | - = "|[" "]|" | "|[" "]|" + = "set[" "]" | "set[" "]" - = "::" - - = "|{" "}|" | "|{" "}|" + = "map{" "}" | "map{" "}" = | "," @@ -526,9 +633,14 @@ that defines their type. = "error(" ")" + = "fusion(" "," ")" + + = "some(" ")" + = | | | | - | | | - | | | + | | | | + | | | | + "(" ")" = uint8 | uint16 | etc. as defined above @@ -536,11 +648,13 @@ that defines their type. = "," | - = ":" + = ":" | "?:" + + = "[" "]" - = "[" "]" | "[" "]" + = "set[" "]" - = "|[" "]|" | "|[" "]|" + = "map{" ":" "}" = "|" @@ -550,13 +664,11 @@ that defines their type. = "," | - = "{" "," "}" - - = = + = "error(" ")" - = as defined above + = "fusion(" ")" - = [0-9]+ + = "option(" ")" - = "error(" ")" + = "decimal(" ")" | "decimal(" "," ")" ``` diff --git a/book/src/intro.md b/book/src/intro.md index 7dd297a4a6..5316212e92 100644 --- a/book/src/intro.md +++ b/book/src/intro.md @@ -80,10 +80,9 @@ Likewise, when extracting and serializing data from a query, there is never any loss of information as the [super-structured formats](formats/intro.md) capture all aspects of the strongly-typed data whether in -[human-readable form](formats/sup.md), -[binary row-like form](formats/bsup.md), +[human-readable form](formats/sup.md) or -[columnar-like form](formats/csup.md). +[binary row or columnar form](formats/bsup.md). ## The `super` Command diff --git a/book/src/super-sql/types/enum.md b/book/src/super-sql/types/enum.md index 208148a9f9..e5ee239f8d 100644 --- a/book/src/super-sql/types/enum.md +++ b/book/src/super-sql/types/enum.md @@ -23,8 +23,8 @@ and this is a value of that type: ``` Enum serialization in the SUP format is fairly verbose as the set of -symbols must be enumerated anywhere the type appears. In the binary formats -of BSUP and CSUP, the enum symbols are encoded efficiently just once. +symbols must be enumerated anywhere the type appears. In the binary +BSUP format, the enum symbols are encoded efficiently just once. ## Examples diff --git a/book/src/tutorials/jq.md b/book/src/tutorials/jq.md index 56e752ebc4..532fba1e50 100644 --- a/book/src/tutorials/jq.md +++ b/book/src/tutorials/jq.md @@ -171,8 +171,8 @@ JSON when needed. Providing human-readability without losing detail is what [SUP](../formats/sup.md) is all about. SUP is nice because it has a comprehensive type system and you can -go from SUP to an efficient binary row format ([Super Binary, BSUP](../formats/bsup.md)) -and columnar ([Super Columnar, CSUP](../formats/csup.md)) --- and vice versa --- +go from SUP to an efficient binary format in row or columnar form +([Super Binary, BSUP](../formats/bsup.md)) --- and vice versa --- with complete fidelity and no loss of information. In this tour, we'll stick to SUP, though for large data sets [BSUP is much faster](../tutorials/performance.md). diff --git a/book/src/tutorials/super-structured.md b/book/src/tutorials/super-structured.md index e31ead77dc..5c852d0b77 100644 --- a/book/src/tutorials/super-structured.md +++ b/book/src/tutorials/super-structured.md @@ -276,14 +276,13 @@ providing a unified approach to row, columnar, and human-readable formats: * [Super (SUP)](../formats/sup.md) is a human-readable format for super-structured data. All JSON documents are SUP values as the SUP format is a strict superset of the JSON syntax. -* [Super Binary (BSUP)](../formats/bsup.md) is a row-based, binary representation somewhat like -Avro but leveraging the super data model to represent a sequence of arbitrarily-typed -values. -* [Super Column (CSUP)](../formats/csup.md) is columnar like Parquet or ORC but also -embodies the super data model for heterogeneous and self-describing schemas. +* [Super Binary (BSUP)](../formats/bsup.md) is an efficient binary representation +of a sequence of arbitrarily-typed values in either rows, somewhat like Avro, or +columns, somewhat like Parquet or ORC, that embodies the super data model for +heterogeneous and self-describing schemas. Because all of the formats conform to the same super data model, conversions between -a human-readable form, a row-based binary form, and a row-based columnar form can +a human-readable form and a binary row or columnar form can be trivially carried out with no loss of information. This is the best of both worlds: the same data can be easily expressed in and converted between a human-friendly and easy-to-program text form alongside efficient row and columnar formats. From 374e67193087f056072a6a3f37e82bfcce321b4f Mon Sep 17 00:00:00 2001 From: Steven McCanne Date: Mon, 9 Nov 2026 15:55:13 -0800 Subject: [PATCH 2/2] address PR feedback --- book/src/formats/model.md | 2 +- book/src/super-sql/types/enum.md | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/book/src/formats/model.md b/book/src/formats/model.md index bc10a556cf..3c0d198dee 100644 --- a/book/src/formats/model.md +++ b/book/src/formats/model.md @@ -103,7 +103,7 @@ Complex types are composed of primitive types and/or other complex types. The categories of complex types include: * _record_ - an ordered collection of zero or more named values called fields, * _array_ - an ordered sequence of zero or more values called elements, -* _set_ - a set of zero or more unique values called elements, +* _set_ - an unordered collection of zero or more unique values called elements, * _map_ - a collection of zero or more key/value pairs where the keys are of a uniform type called the key type and the values are of a uniform type called the value type, diff --git a/book/src/super-sql/types/enum.md b/book/src/super-sql/types/enum.md index e5ee239f8d..436362f349 100644 --- a/book/src/super-sql/types/enum.md +++ b/book/src/super-sql/types/enum.md @@ -24,7 +24,8 @@ and this is a value of that type: Enum serialization in the SUP format is fairly verbose as the set of symbols must be enumerated anywhere the type appears. In the binary -BSUP format, the enum symbols are encoded efficiently just once. +[BSUP](../../formats/bsup.md) format, +the enum symbols are encoded efficiently just once. ## Examples