Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion book/src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -209,7 +209,6 @@
- [Data Model](formats/model.md)
- [Super (SUP)](formats/sup.md)
- [Super Binary (BSUP)](formats/bsup.md)
- [Super Column (CSUP)](formats/csup.md)
!!- [Database](database/intro.md)
!! - [API](database/api.md)
!! - [Format](database/format.md)
Expand Down
3 changes: 1 addition & 2 deletions book/src/command/formats.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ The supported [input](input.md) and [output](output.md) formats include the foll
|-----------|------|-----------|------------------------------------------|
| `arrows` | yes | `.arrows` | [Arrow IPC Stream Format](https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format) |
| `bsup` | yes | `.bsup` | [BSUP](../formats/bsup.md) |
| `csup` | yes | `.csup` | [CSUP](../formats/csup.md) |
| `csv` | yes | `.csv` | [Comma-Separated Values (RFC 4180)](https://www.rfc-editor.org/rfc/rfc4180.html) |
| `json` | yes | `.json`, `.jsonl`, `.ndjson` | [JSON (RFC 8259)](https://www.rfc-editor.org/rfc/rfc8259.html) |
| `line` | no | `.text`, `.txt` | n/a | One text value per line |
Expand All @@ -17,6 +16,6 @@ The supported [input](input.md) and [output](output.md) formats include the foll

>[!NOTE]
> Best performance is typically achieved when operating on data in binary columnar formats
> such as [CSUP](../formats/csup.md),
> such as [BSUP](../formats/bsup.md),
> [Parquet](https://github.com/apache/parquet-format), or
> [Arrow](https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format).
2 changes: 1 addition & 1 deletion book/src/command/input.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ Note that the `line` format cannot be automatically detected and
requires `-i` or `(format line)` for reading.

>[!NOTE]
> Parquet and CSUP require a seekable input and cannot be operated upon
> Parquet requires a seekable input and cannot be operated upon
> when read on standard input.

## Type Checking
Expand Down
19 changes: 11 additions & 8 deletions book/src/command/output.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,6 @@ output format is [SUP](../formats/sup.md).
Otherwise, the default format is [BSUP](../formats/bsup.md).
These defaults may be overridden with `-f`, `-s`, or `-S`.

>[!NOTE]
> While BSUP is currently default, a forthcoming release will change
> CSUP to default after CSUP supports streaming.

Since SUP is a common format choice for interactive use,
the `-s` flag is shorthand for `-f sup`.
Also, `-S` is a shortcut for `-f sup` with `-pretty 2` as
Expand All @@ -33,9 +29,9 @@ is a shortcut for `-f json` and `-J` is a shortcut for pretty-printing JSON.
> causes an occasional surprise
> (e.g., forgetting `-f` or `-s` in a scripted test that works fine on the
> command line but fails in CI). However, this avoids problematic performance where a
> data pipeline deployed to production accidentally uses SUP instead of CSUP.
> data pipeline deployed to production accidentally uses SUP instead of BSUP.
> Since `super` gracefully handles any input, this would be hard to detect.
> Alternatively, making CSUP always be default would cause much annoyance when
> Alternatively, making BSUP always be default would cause much annoyance when
> binary data is written to the terminal.

If no query is specified with `-c`, the inputs are scanned without modification
Expand Down Expand Up @@ -89,10 +85,17 @@ produces
When pretty printing, colorization is enabled by default when writing to a terminal,
and can be disabled with `-color false`.

## BSUP Output

By default, [BSUP](../formats/bsup.md) output is written as column frames,
which let a reader read only the columns it needs. The `-rows` flag writes
row frames instead. The `-framecap` flag sets the target number of values
in each frame (default 10000).

## Pipeline-friendly Formats

Though they're compressed formats, CSUP and BSUP data are self-describing and
stream-oriented and thus are pipeline friendly.
Though it's a compressed format, BSUP data is self-describing and
stream-oriented and thus is pipeline friendly.

Since data is self-describing you can simply take super-structured output
of one command and pipe it to the input of another. It doesn't matter if the value
Expand Down
1 change: 0 additions & 1 deletion book/src/database/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -546,7 +546,6 @@ The following table shows the supported MIME types and where they can be used.
| ---------------- | --------- | -------- | ------------------------------------- |
| Arrow IPC Stream | yes | yes | `application/vnd.apache.arrow.stream` |
| BSUP | yes | yes | `application/x-bsup` |
| CSUP | yes | yes | `application/x-csup` |
| CSV | yes | yes | `text/csv` |
| JSON | yes | yes | `application/json` |
| Line | yes | yes | `application/x-line` |
Expand Down
14 changes: 5 additions & 9 deletions book/src/database/format.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,20 +76,16 @@ Since the object's name is globally unique and the
resulting object is immutable, there is no possible write concurrency to manage
with respect to a given object.

A data object is composed of the primary data object stored as one or two objects
(for sequence and/or vector layout).

Data objects may be either in sequence form (i.e., BSUP) or vector form (i.e., CSUP),
or both forms may be present as a query optimizer may choose to use whatever
representation is more efficient.
When both sequence and vector data objects are present, they must contain the same
underlying Super data.
A data object is stored as a single [BSUP](../formats/bsup.md) object,
whose column frames hold the data in vector form and whose
[zone maps](../formats/bsup.md#86-zone-maps) allow a query to skip frames
that can't contain matching values.

Immutable objects are named as follows:

|object type|name|
|-----------|----|
|sequence data|`<table-id>/data/<id>.bsup`|
|data|`<table-id>/data/<id>.bsup`|

`<id>` is the KSUID of the data object.

Expand Down
3 changes: 1 addition & 2 deletions book/src/database/intro.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,8 +150,7 @@ replication easy to support and deploy.

The cloud objects that comprise a database, e.g., data objects,
commit history, transaction journals, partial aggregations, etc.,
are stored as super-structured data, i.e., either as [row-based Super Binary (BSUP)](../formats/bsup.md)
or [Super Columnar (CSUP)](../formats/csup.md).
are stored as super-structured data in [Super Binary (BSUP)](../formats/bsup.md).
This makes introspection of the database structure straightforward as many key
database data structures can be queried with metadata queries and presented
to a client for further processing by downstream tooling.
Expand Down
2 changes: 1 addition & 1 deletion book/src/dev/integrations/zeek/types.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
As the [super-structured data model](../../../formats/model.md) was in many ways inspired by the
[Zeek TSV log format](https://docs.zeek.org/en/current/tutorial/logs.html#zeek-log-formats),
SuperDB's rich storage formats ([SUP](../../../formats/sup.md),
[CSUP](../../../formats/csup.md), etc.) maintain comprehensive interoperability
[BSUP](../../../formats/bsup.md), etc.) maintain comprehensive interoperability
with Zeek.

When Zeek is configured to output its logs in
Expand Down
9 changes: 4 additions & 5 deletions book/src/dev/libraries/python.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ interacts with the database via the REST API provided by
>
> The full benefits of super-structured data (including improved performance)
> are expected when there is a native Python library implementation of
> [CSUP](../../formats/csup.md). Stay tuned.
> [BSUP](../../formats/bsup.md). Stay tuned.

## Installation

Expand Down Expand Up @@ -185,17 +185,16 @@ by inspecting the beginning of the uploaded stream. This works for most formats:
| Zeek | `application/x-zeek` |
| Arrow IPC stream | `application/vnd.apache.arrow.stream` |

Three formats cannot be auto-detected over the API and **require an explicit
Two formats cannot be auto-detected over the API and **require an explicit
`mime_type`**:

| Format | MIME type | Reason |
|---|---|---|
| CSUP | `application/x-csup` | Detection requires seekable input |
| Parquet | `application/x-parquet` | Detection requires seekable input |
| Line | `application/x-line` | Auto-detection not supported |

CSUP and Parquet store identifying metadata outside the stream header (Parquet's
footer is at the end of the file), so the server cannot detect them from an HTTP
Parquet stores identifying metadata outside the stream header (its
footer is at the end of the file), so the server cannot detect it from an HTTP
request body. Line format is plain text with no identifying structure and is
intentionally excluded from auto-detection.

Expand Down
Loading
Loading