Skip to content

Add ch-core-wasm: core behind a versioned C ABI - #34

Open
ccoltx wants to merge 3 commits into
mainfrom
ccoltx/wasm-for-clickhouse-js
Open

ccoltx wants to merge 3 commits into
mainfrom
ccoltx/wasm-for-clickhouse-js

Conversation

@ccoltx

@ccoltx ccoltx commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Ports the WASM proof of concept from #18 into crates/ch-core-wasm and acts on Joe's review there.

  • Hand-written C ABI and JS glue instead of wasm-bindgen. No dependencies beyond core and no imports in the .wasm. ABI.md is the contract, and the glue checks its version at load.
  • Copy out. Results are a JSON descriptor pointing at Rust-owned buffers. The glue copies each buffer into a JS TypedArray once and frees the result, so memory growth can't touch anything a caller holds.
  • Columnar only. Row building from the POC is not ported. It moves to clickhouse-js with the client methods.
  • zstd is now an optional default feature in core, so the wasm build is pure Rust and needs no clang. Default builds are unchanged.
  • Fixes an encoder bug the 32-bit target exposed: a huge row count for zero-width columns wrapped the output size and returned a truncated block.
  • A new CI job builds the .wasm, checks its size, and runs the JS tests.

Decode speed matches the POC. Encode is about 2 ms slower per 1M rows because input is copied twice, which can be fixed later. The final crate and npm name is left for the repo rename.

Refs #18


Note

Medium Risk
Introduces a large new public WASM ABI surface and changes core’s default feature layout; encoder/compression behavior changes are bounded but touch hot codec paths and 32-bit sizing.

Overview
Adds ch-core-wasm, a new workspace crate that compiles core to WebAssembly behind a versioned C ABI (ABI.md) with hand-written JS glue (Node + web). It exposes columnar decode, stream decode, encode, type describe, and binary-value APIs; results are JSON descriptors with buffers that JS copies out of linear memory (never views).

Makes ZSTD optional via a default zstd feature on ch-core-rs so wasm builds are pure Rust; without it, ZSTD frames return CompressionError::UnsupportedCodec. Adds wasm-release profile, CI (MSRV wasm32 check, wasm build, 1 MiB size budget, JS tests), and docs/changelog updates.

Fixes encode_block on 32-bit targets: huge row counts for zero-width columns could wrap output size; encoding now uses checked reserves and returns EncodeError::OutputTooLarge instead of truncating or panicking.

Reviewed by Cursor Bugbot for commit 085715d. Bugbot is set up for automated code reviews on this repo. Configure here.

ccoltx and others added 3 commits November 20, 2026 18:19
libzstd is C, so building core for wasm32 needs clang and llvm-ar. With
default-features = false the crate is pure Rust. ZSTD frames then return
CompressionError::UnsupportedCodec, from frame_header as well as from the
decompress paths. Default builds are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing and Tuple() rows cost no input memory, so the row count alone sizes
the output. On 32-bit targets buf.len() + rows wrapped in release builds and
encode_block returned a truncated block. Bool, null maps and QBit expand
their input up to 8x and could overflow the same way. These paths now
reserve with try_reserve and return the new EncodeError::OutputTooLarge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ports the WASM proof of concept from #18 and applies the review there:

- A hand-written C ABI and JS glue replace wasm-bindgen. The crate has no
  dependencies beyond core and the .wasm has no imports. ABI.md documents
  the contract, and the glue checks chw_abi_version at load.
- Copy out. Results are a JSON descriptor whose buffers point at
  Rust-owned memory behind a result handle. The glue copies each buffer
  into a JS TypedArray once, then frees the result, so memory growth
  cannot empty or corrupt anything a caller holds.
- Columnar only. The POC's row building is not ported; it belongs in
  clickhouse-js with the client methods.
- Handles carry a generation, so a stale handle is an error, not a
  pointer into someone else's result.

A new CI job builds the .wasm for wasm32 with a size budget, runs the JS
tests against committed fixtures, and checks core without zstd.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ccoltx
ccoltx requested a review from a team as a code owner November 20, 2026 17:20

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 085715d. Configure here.

ChType::Decimal { bits, .. } => n("data", *bits as usize / 8),
ChType::BFloat16 => n("values", 2),
other => primitive_width(other).and_then(|w| n("values", w)),
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bool dictionary size cannot be inferred

Medium Severity

dictionary_len infers a LowCardinality dictionary's row count from buffer byte lengths, but a Bool dictionary stores that count in length and packs bits into bitmap, not a values array. primitive_width also omits Bool, so LowCardinality(Bool) fails encode with cannot determine dictionary size even after a successful decode of a valid type.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 085715d. Configure here.

@github-actions

Copy link
Copy Markdown

AI review gate — Cursor Bugbot found issues

  • Cursor Bugbot found issues that are not resolved (check concluded 'neutral').

Unresolved findings (1):

What to do:

  • Fix the findings and push. The reviewer re-reviews the new commit and the gate re-runs by itself.
  • Finding wrong, or reviewer stuck? Ask a human with write access who is not the PR author to approve this exact commit, then re-run the gate. Their approval stands in for the AI review (a new push needs a new approval).

Re-run the gate with “Re-run jobs” after an approval or a finished review. A push re-runs it automatically.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant