hyperdb-api

A pure-Rust implementation of the Hyper database API. Create, read, and manipulate Hyper database files (.hyper) without any C library dependencies.

Single-connection throughput on a 100M row benchmark: 68.9M rows/sec inserts with the async AsyncArrowInserter, 25.0M rows/sec with the sync Inserter, 31.1M rows/sec full-scan queries (see docs/BENCHMARK_GUIDE.md)
Streaming by default — constant memory for billion-row results
Both sync (Connection) and async (AsyncConnection) APIs
Built-in string-native key-value store (KvStore / AsyncKvStore)
No feature flags on hyperdb-api — everything is always available (opt-in compile-time SQL validation lives behind a feature on the separate hyperdb-api-derive crate)

Installation

[dependencies]
hyperdb-api = "1.0"

Semver and re-exported crates

arrow, chrono and geo_types types appear in this crate's public API (ArrowInserter, ArrowRowset, Timestamp conversions, Geography), so hyperdb-api re-exports all three crates from its root. Use hyperdb_api::arrow (and friends), or depend on exactly the same major yourself: two majors in one binary give two incompatible RecordBatch types.

A major version bump of any re-exported crate is a major version bump of hyperdb-api. This release builds against arrow 60, chrono 0.4 and geo-types 0.7.

Runtime Requirements

The hyperd executable (Hyper database server) must be available. Set its path via:

export HYPERD_PATH=/path/to/hyperd

Quick Start

use hyperdb_api::{
    Catalog, Connection, CreateMode, HyperProcess, Inserter,
    Result, SqlType, TableDefinition,
};

fn main() -> Result<()> {
    // Start a Hyper server
    let hyper = HyperProcess::new(None, None)?;

    // Connect to a database
    let conn = Connection::new(&hyper, "example.hyper", CreateMode::CreateIfNotExists)?;

    // Create a table
    let table_def = TableDefinition::from("users")
        .add_required_column("id", SqlType::int())
        .add_required_column("name", SqlType::text());
    Catalog::new(&conn).create_table(&table_def)?;

    // Insert data using the high-performance Inserter (COPY protocol)
    {
        let mut inserter = Inserter::new(&conn, &table_def)?;
        inserter.add_row(&[&1i32, &"Alice"])?;
        inserter.add_row(&[&2i32, &"Bob"])?;
        inserter.execute()?;
    }

    // Query data
    let result = conn.execute_query("SELECT * FROM users")?;
    for row in result.rows() {
        let row = row?;
        let id: Option<i32> = row.get(0);
        let name: Option<String> = row.get(1);
        println!("User: {:?} - {:?}", id, name);
    }

    Ok(())
}

Async Quick Start

use hyperdb_api::{AsyncConnection, CreateMode, HyperProcess, Result};

#[tokio::main]
async fn main() -> Result<()> {
    let hyper = HyperProcess::new(None, None)?;
    let endpoint = hyper.require_endpoint()?;

    let conn = AsyncConnection::connect(
        endpoint,
        "example_async.hyper",
        CreateMode::CreateIfNotExists,
    ).await?;

    conn.execute_command("CREATE TABLE users (id INT, name TEXT)").await?;
    conn.execute_command("INSERT INTO users VALUES (1, 'Alice')").await?;
    conn.close().await?;
    Ok(())
}

Connection Pooling

For high-concurrency async applications, use the built-in connection pool:

use hyperdb_api::pool::{create_pool, PoolConfig};
use hyperdb_api::{CreateMode, HyperProcess, Result};

#[tokio::main]
async fn main() -> Result<()> {
    let hyper = HyperProcess::new(None, None)?;
    let endpoint = hyper.require_endpoint()?;

    let config = PoolConfig::new(endpoint, "pooled.hyper")
        .create_mode(CreateMode::CreateIfNotExists)
        .max_size(10);

    let pool = create_pool(config)?;

    // Get connections from the pool — returned automatically when dropped
    let conn = pool.get().await?;
    conn.execute_command("SELECT 1").await?;

    Ok(())
}

Option	Default	Description
`max_size(n)`	16	Maximum connections in pool
`create_mode(mode)`	`DoNotCreate`	Database creation mode
`auth(user, pass)`	None	Authentication credentials

gRPC Transport

The API supports gRPC as an alternative to TCP for read-only queries with Arrow IPC results. The unified Connection type auto-detects transport based on URL:

use hyperdb_api::{Connection, CreateMode};

// http:// or https:// → gRPC (auto-detected)
let conn = Connection::connect(
    "http://localhost:7484",
    "my_database.hyper",
    CreateMode::DoNotCreate
)?;

// Same query API as TCP
let mut result = conn.execute_query("SELECT * FROM users")?;

Start Hyper with gRPC enabled:

use hyperdb_api::{HyperProcess, ListenMode, Parameters};

let mut params = Parameters::new();
params.set_listen_mode(ListenMode::Both { grpc_port: 7484 });
let hyper = HyperProcess::new(None, Some(&params))?;

Transfer Mode	Best For
`SYNC`	Small results
`ASYNC`	Very large results
`ADAPTIVE` (default)	Most workloads

Query Execution

Convenience Methods

// Fetch single row (error if no rows)
let user = conn.fetch_one("SELECT * FROM users WHERE id = 1")?;

// Fetch optional row (None if no rows)
let user = conn.fetch_optional("SELECT * FROM users WHERE id = 999")?;

// Fetch all rows
let users = conn.fetch_all("SELECT * FROM users")?;

// Fetch scalar value
let count: i64 = conn.fetch_scalar("SELECT COUNT(*) FROM users")?;

Parameterized Queries

Always use parameterized queries with user input to prevent SQL injection:

let mut result = conn.query_params(
    "SELECT * FROM users WHERE name = $1 AND age >= $2",
    &[&"Alice", &30i32],
)?;

// For INSERT/UPDATE/DELETE:
let rows_affected = conn.command_params(
    "UPDATE users SET balance = $1 WHERE name = $2",
    &[&750.0f64, &"Alice"],
)?;

Supported parameter types: i16, i32, i64, f32, f64, bool, &str, String, Option<T>, Date, Time, Timestamp, OffsetTimestamp, Vec<u8>, &[u8].

To map a parameterized query's results straight into a FromRow struct (instead of raw rows), use the _as_params variants — fetch_one_as_params, fetch_all_as_params, and stream_as_params (sync and async). They combine the $1 binding above with the struct mapping of fetch_*_as / stream_as in one call:

#[derive(hyperdb_api_derive::FromRow)]
struct User { id: i32, name: String }

let users: Vec<User> = conn.fetch_all_as_params(
    "SELECT id, name FROM users WHERE org_id = $1",
    &[&42i32],
)?;

Known limitation: `OR` / `IN` lists over parameters

A hyperd defect rejects a filter on a single column that combines a bound parameter with other values through OR, IN, NOT IN or = ANY(ARRAY[...]), for example WHERE id = $1 OR id = $2 or WHERE id IN ($1, $2). The statement fails when it is prepared, with SQLSTATE XX000 ("A parameter was accessed in an execution target with too few registered parameters"). This affects every API that binds parameters, including command_params for UPDATE and DELETE. Filters that combine parameters with AND, and OR across two different columns, are not affected.

Rewrite the list as a subquery over an array built from the parameters. The values stay bound, so the query is still injection-safe:

// Fails: "SELECT * FROM users WHERE id IN ($1, $2)"
let mut result = conn.query_params(
    "SELECT * FROM users WHERE id IN (SELECT unnest(ARRAY[$1, $2]))",
    &[&1i64, &2i64],
)?;

See the Connection::query_params API docs for details.

Compile-time SQL validation (opt-in)

With the hyperdb-api-derive crate's compile-time feature, the query_as! and query_scalar! macros validate your SQL against registered #[derive(Table)] structs at build time — typos, unknown columns, and struct/SQL mismatches become compile errors (with red squigglies in VS Code) instead of runtime failures. It's entirely opt-in and off by default. See hyperdb-api-derive/README.md for setup.

Streaming Results

Results are always streaming with constant memory usage:

// Chunked iteration (batch processing, maximum performance)
let mut result = conn.execute_query("SELECT * FROM large_table")?;
while let Some(chunk) = result.next_chunk()? {
    for row in &chunk {
        let id: Option<i32> = row.get(0);
    }
}

// Row iterator (simpler, slightly more overhead)
let result = conn.execute_query("SELECT * FROM large_table")?;
for row in result.rows() {
    let row = row?;
    let id: Option<i32> = row.get(0);
}

Streaming typed structs (`stream_as`)

stream_as::<T>() maps each row to a FromRow struct lazily — the same struct mapping as fetch_all_as, but with O(chunk) memory instead of buffering every row. The column-name → index map is built once and reused across all chunks:

#[derive(hyperdb_api_derive::FromRow)]
struct User { id: i32, name: String }

// Sync — lazy iterator, constant memory regardless of result size
for row in conn.stream_as::<User>("SELECT id, name FROM users")? {
    let user = row?;
}

// Async — impl Stream<Item = Result<T>>
use futures::StreamExt; // add `futures = "0.3"` to your Cargo.toml for `.next()`
let stream = conn.stream_as::<User>("SELECT id, name FROM users");
tokio::pin!(stream);
while let Some(row) = stream.next().await {
    let user = row?;
}

The async form yields an impl Stream; driving it needs the StreamExt / TryStreamExt extension traits from the futures crate (hyperdb-api itself only depends on futures-core for the Stream type).

See docs/ROW_MAPPING.md for all five row-mapping forms, from positional row.get(0) to streaming stream_as.

Bulk Data Insertion

Inserter (COPY Protocol)

The high-performance Inserter uses HyperBinary COPY protocol for roughly 25M rows/sec on a single connection. For the fastest insert path, use the async AsyncArrowInserter instead — see docs/BENCHMARK_GUIDE.md.

let mut inserter = Inserter::new(&conn, &table_def)?;
inserter.add_row(&[&1i32, &"Widget", &9.99f64])?;
inserter.add_row(&[&2i32, &None::<&str>, &19.99f64])?;  // NULL name
inserter.execute()?;

Type-specific methods are also available (similar to the C++ API):

inserter.add_i32(1)?;
inserter.add_str("Widget")?;
inserter.add_f64(9.99)?;
inserter.end_row()?;

Column Mappings

Insert data with computed columns using MappedInserter:

let mappings = vec![
    ColumnMapping::new("id"),
    ColumnMapping::new("quantity"),
    ColumnMapping::with_expression("total", "\"quantity\" * \"unit_price\""),
];

let mut inserter = Inserter::with_column_mappings(&conn, &inserter_def, "orders", &mappings)?;

Arrow Format

Insert pre-formatted Arrow IPC data:

let mut inserter = ArrowInserter::new(&conn, &table_def)?;
inserter.insert_data(&arrow_ipc_data)?;
let rows = inserter.execute()?;

Multi-threaded Insertion

For maximum throughput on multi-core systems, use InsertChunk and ChunkSender to parallelize data encoding:

let sender = ChunkSender::new(&conn, &table_def)?;

// Workers encode chunks in parallel using InsertChunk
let chunk = InsertChunk::from_table_definition(&table_def);
// ... populate chunk ...
sender.send_chunk(chunk)?;

let total_rows = sender.finish()?;

Catalog Operations

let catalog = Catalog::new(&conn);

// List schemas and tables
let schemas = catalog.get_schema_names::<&str>(None)?;
let tables = catalog.get_table_names("public")?;

// Check existence
if catalog.has_table("public.users")? {
    let table_def = catalog.get_table_definition("public.users")?;
    println!("Columns: {}", table_def.column_count());
}

SQL-Safe Names

Type-safe SQL identifier handling with automatic escaping:

use hyperdb_api::{table_name, Name, TableName};

// Simple construction
let name = Name::try_new("users")?;

// Qualified names (dot-separated parsing)
let table: TableName = "mydb.public.users".parse()?;

// Fluent builder
let table = TableName::try_new("users")?.with_schema("public")?;

// Macros
let table = table_name!("public", "users")?;

Most API methods accept strings directly via impl TryInto<T>:

// No manual conversion needed
if catalog.has_table("public.users")? { /* ... */ }

Transactions

let txn = conn.transaction()?;
txn.execute_command("INSERT INTO users VALUES (1, 'Alice')")?;
txn.execute_command("INSERT INTO users VALUES (2, 'Bob')")?;
txn.commit()?;  // both inserts are committed; drop without commit() auto-rolls-back

See docs/TRANSACTIONS.md for full details.

Key-Value Store

Connection::kv_store (and AsyncConnection::kv_store) opens a KvStore handle — a string-native key-value store backed by a single fixed table, namespaced by store name. Values are plain strings, or any serde-serializable type via set_as/get_as.

use hyperdb_api::{Connection, CreateMode, HyperProcess, Result};

fn main() -> Result<()> {
    let hyper = HyperProcess::new(None, None)?;
    let conn = Connection::new(&hyper, "app.hyper", CreateMode::CreateIfNotExists)?;

    let kv = conn.kv_store("settings")?;
    let outcome = kv.set("theme", "dark")?;
    assert!(outcome.created);
    assert_eq!(kv.get("theme")?, Some("dark".to_string()));

    // Typed values via serde (stored as JSON).
    let outcome = kv.set_as("retries", &3u32)?;
    assert!(outcome.created);
    assert_eq!(kv.get_as::<u32>("retries")?, Some(3));

    // Write many entries atomically in one transaction.
    let batch_outcome = kv.set_batch(&[("host", "localhost"), ("port", "7483")])?;
    assert_eq!(batch_outcome.created, 2);
    assert_eq!(batch_outcome.overwritten, 0);

    // Other operations: exists, delete, size, keys, clear, and pop
    // (remove-and-return the lowest-ordered key).
    assert!(kv.exists("host")?);
    let _ = conn.kv_list_stores()?; // names of all non-empty stores
    Ok(())
}

Store names and keys must be non-empty, at most 512 bytes, and contain only ASCII A-Z a-z 0-9 _ . -; anything else returns Error::InvalidName.

Connection Features

TLS

TCP connections use TLS through the builders' tls(), with libpq sslmode names: Disable (the default), Prefer, Require, VerifyCa and VerifyFull. PoolConfig and SyncPoolConfig take the same setting; the Connection::new / connect* shortcuts stay plaintext.

use hyperdb_api::{ConnectionBuilder, TlsConfig, TlsMode};

let conn = ConnectionBuilder::new("hyper.example.com:7483")
    .database("data.hyper")
    .tls(TlsConfig::new(TlsMode::VerifyFull).root_cert("ca.pem"))
    .build()?;
assert!(conn.is_tls());

A hyperd serves TLS when started with the ssl_key and ssl_certificate settings (Parameters::set) over the TCP transport. gRPC picks TLS from its https:// scheme instead. See TlsMode for how each mode treats a server without TLS and which certificate checks it runs.

Query Cancellation

Thread-safe cancellation from another thread:

let conn = Arc::new(Connection::new(&hyper, "test.hyper", CreateMode::CreateIfNotExists)?);
// ... in another thread:
conn.cancel()?;  // Cancels running query (SQLSTATE 57014)

Notice Receiver

conn.set_notice_receiver(Some(Box::new(|notice| {
    println!("Notice: {} ({})", notice.message(), notice.severity().unwrap_or(""));
})));

Query Statistics

Per-query performance metrics from Hyper's internal log:

use hyperdb_api::LogFileStatsProvider;

conn.enable_query_stats(LogFileStatsProvider::from_process(&hyper));
conn.execute_command("SELECT * FROM users")?;

if let Some(stats) = conn.last_query_stats() {
    println!("Elapsed: {:.3}ms", stats.elapsed_s * 1000.0);
}

Logging

The API uses the tracing crate for structured logging:

# Enable via environment variable
RUST_LOG=hyperdb_api=info cargo run

# Debug-level for auth details and chunk progress
RUST_LOG=hyperdb_api=debug cargo run

Examples

Example	Description	Command
`insert_data_into_single_table`	Create table and insert data	`cargo run -p hyperdb-api --example insert_data_into_single_table`
`insert_data_into_multiple_tables`	Multiple related tables	`cargo run -p hyperdb-api --example insert_data_into_multiple_tables`
`create_hyper_file_from_csv`	Load CSV into Hyper	`cargo run -p hyperdb-api --example create_hyper_file_from_csv`
`read_and_print_data_from_existing_hyper_file`	Read and query data	`cargo run -p hyperdb-api --example read_and_print_data_from_existing_hyper_file`
`insert_data_with_expressions`	Column mappings	`cargo run -p hyperdb-api --example insert_data_with_expressions`
`insert_geospatial_data_to_a_hyper_file`	Geography column	`cargo run -p hyperdb-api --example insert_geospatial_data_to_a_hyper_file`
`arrow`	Arrow RecordBatch read/write	`cargo run -p hyperdb-api --example arrow`
`async_usage`	AsyncConnection + Tokio	`cargo run -p hyperdb-api --example async_usage`
`threaded_inserter`	Multi-threaded bulk insert	`cargo run -p hyperdb-api --example threaded_inserter`
`grpc_query`	gRPC transport + Arrow IPC	`cargo run -p hyperdb-api --example grpc_query`
`connection_pool`	Async connection pooling	`cargo run -p hyperdb-api --example connection_pool`
`transactions`	RAII guards, multi-table rollback, DDL, reconnect semantics	`cargo run -p hyperdb-api --example transactions`

Run all examples:

export HYPERD_PATH=/path/to/hyperd
./run_all_examples.sh

Companion Crates

sea-query-hyperdb

HyperDB dialect backend for sea-query — use for window functions, CTEs, complex JOINs, and type-safe query composition.

[dependencies]
sea-query = "1.0"
sea-query-hyperdb = "1.0"

use sea_query::{Expr, ExprTrait, Iden, Query};
use sea_query_hyperdb::HyperQueryBuilder;

#[derive(Iden)]
enum Users {
    Table,
    Name,
    Age,
}

let sql = Query::select()
    .column(Users::Name)
    .from(Users::Table)
    .and_where(Expr::col(Users::Age).gt(18))
    .to_string(HyperQueryBuilder);

let result = conn.fetch_all(&sql)?;

hyperdb-api-salesforce

Salesforce Data Cloud OAuth authentication — JWT Bearer Token, Username-Password, and Refresh Token flows.

[dependencies]
hyperdb-api-salesforce = "1.0"

See hyperdb-api-salesforce/README.md for setup instructions.

Acknowledgments

This crate depends on hyperdb-api-core, which includes code adapted from sfackler/rust-postgres (the postgres-protocol, tokio-postgres, and postgres-types crates by Steven Fackler, MIT or Apache-2.0). See the NOTICE file at the workspace root for the full third-party attribution list. (Relative links to NOTICE work on GitHub but not on crates.io; the absolute URL above is crates.io-friendly.)

License

Licensed under either of Apache License, Version 2.0 or MIT license, at your option.

Navigation Menu

Repository navigation

README.md

README.md

hyperdb-api

Installation

Semver and re-exported crates

Runtime Requirements

Quick Start

Async Quick Start

Connection Pooling

gRPC Transport

Query Execution

Convenience Methods

Parameterized Queries

Known limitation: `OR` / `IN` lists over parameters

Compile-time SQL validation (opt-in)

Streaming Results

Streaming typed structs (`stream_as`)

Bulk Data Insertion

Inserter (COPY Protocol)

Column Mappings

Arrow Format

Multi-threaded Insertion

Catalog Operations

SQL-Safe Names

Transactions

Key-Value Store

Connection Features

TLS

Query Cancellation

Notice Receiver

Query Statistics

Logging

Examples

Companion Crates

sea-query-hyperdb

hyperdb-api-salesforce

Acknowledgments

License

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

hyperdb-api

Installation

Semver and re-exported crates

Runtime Requirements

Quick Start

Async Quick Start

Connection Pooling

gRPC Transport

Query Execution

Convenience Methods

Parameterized Queries

Known limitation: OR / IN lists over parameters

Compile-time SQL validation (opt-in)

Streaming Results

Streaming typed structs (stream_as)

Bulk Data Insertion

Inserter (COPY Protocol)

Column Mappings

Arrow Format

Multi-threaded Insertion

Catalog Operations

SQL-Safe Names

Transactions

Key-Value Store

Connection Features

TLS

Query Cancellation

Notice Receiver

Query Statistics

Logging

Examples

Companion Crates

sea-query-hyperdb

hyperdb-api-salesforce

Acknowledgments

License

Known limitation: `OR` / `IN` lists over parameters

Streaming typed structs (`stream_as`)