Skip to content

Simplify execution and add upstream SQLLogicTest coverage - #28

Merged
jakthom merged 11 commits into
mainfrom
refactor/execution-and-correctness
Sep 10, 2026
Merged

jakthom merged 11 commits into
mainfrom
refactor/execution-and-correctness

Conversation

@jakthom

@jakthom jakthom commented Sep 9, 2026 •

Copy link
Copy Markdown
Collaborator

Go execution duplicated statement ownership and cleanup, and the bridge had limited SQL coverage. This change consolidates execution, splits Rust by responsibility, and runs the pinned DataFusion SQLLogicTest corpus through the public Go Arrow API.

  • Inventory all 504 upstream SQL files and seven driver fixtures. Hash manifests protect the reviewed fixtures and documentation; pinned archives and reproducible TPC-H generation supply datasets. The 146 comment-only Spark stubs and eight postgres-only records have explicit counts.
  • Use upstream fixtures, parsing, type checks, and value comparisons. Go executes each query and consumes its Arrow batches. Test-only callbacks and dependencies are absent from production libraries. Harness tests reject wrong rows/types, missing includes, unsupported directives, and adapter failures that resemble expected SQL errors.
  • Inventory 327 documented functions and 188 SQL documentation sections, with passing examples for every entry in the default DataFusion test setup. Reports retain assertion outcomes, source locations, SQL hashes, reviewed mappings, and the dependency and native library used.
  • Honor the configured SQL dialect during preparation, including changes through SET. Keep SQL PREPARE/CREATE FUNCTION parameters in their SQL scope, including under EXPLAIN. Upgrade Arrow Go to v18.8.0 for correct fractional-hour timezone offsets.
  • Retain the shared Go/Rust query corpus, SQLite oracle, lifecycle sequences, failure injection, installation subprocess tests, fuzz targets, native sanitizers, and separate Go/Rust/C coverage. Generate ABI declarations and layout checks from the C header. Run the required SQL corpus in CI on Linux, macOS, and Windows.

The reviewed corpus has four temporary exclusions. Two FixedSizeList(0, Null) assertions are commented out until the Arrow Go dependency supports zero-length lists; the proposed fix is tracked in apache/arrow-go PR #1308. Case G's result and EXPLAIN ANALYZE assertions in aggregate_memory_spill.slt run separately because four-partition aggregation can exhaust its 1 MiB pool or time out. Cases A–F and the ordinary single-partition Go spill regression remain required. The exclusions and restoration conditions are documented, and their checksums are updated.

The Spark csc and sec fixtures retain ordinary-input and special-value checks. Platform-dependent snapshots near trigonometric poles are replaced with known values away from the poles, checked with an absolute tolerance of 1e-12. The suite's comparison rules are unchanged.

make test.sqllogic.spill attempts 25 repetitions through Go and in each of two native Rust execution modes. It checks the result values and a nonzero spill count in the final aggregation. CI runs only these diagnostic steps with continue-on-error; logs and exit codes are retained under reports/current/spill/ without changing required corpus outcomes. CI also replays failed required SQL files through native DataFusion and retains separate comparison logs.

Validation on macOS arm64 with released Arrow Go v18.8.0 and no module replacement:

make lint
make test
make test.source
make rust.test
make sqllogic.check sqllogic.tools.test
TOKIO_WORKER_THREADS=4 make test.sqllogic

All passed, along with workflow lint. The required corpus passed all 24,956 active assertions across 365 executable files, with all 511 files reported and no documentation gaps. The separate spill diagnostic returned failure as intended when it reproduced Go memory exhaustion, a native blocking-read timeout, and native async memory exhaustion; all three outcomes were retained. The underlying parallel-spill issue remains unresolved. Linux and Windows verification of these changes awaits CI.

Function evidence excludes expected-error, failed, empty-result, EXPLAIN-only, and Spark-only queries. Reviewed section mappings record discrepancies in the pinned documentation, including obsolete compression syntax. These example counts do not prove every overload, input, branch, or possible SQL program correct. Required corpus success and documentation witnesses remain separate metrics.

@jakthom
jakthom marked this pull request as draft September 9, 2026 20:08
@jakthom jakthom changed the title Simplify execution and add cross-language correctness checks Simplify execution and add upstream SQLLogicTest coverage Sep 9, 2026
@jakthom
jakthom marked this pull request as ready for review September 10, 2026 00:03
@jakthom

jakthom commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator Author

Weeee lfg

@jakthom
jakthom merged commit 36fc634 into main Sep 10, 2026
27 checks passed
@jakthom
jakthom deleted the refactor/execution-and-correctness branch September 10, 2026 00:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant