Repository navigation
Conversation
Phase 0 - correctness & robustness in soatable.hpp: - 0.1 Replace signed sparse index (int32 + tombstone=-1) with an unsigned size_type alias and npos sentinel; remove all size_t round-trip casts. - 0.2 Drop the unused <iostream> include from the public header. - 0.3 Document per-method exception guarantees; make emplace (new value) and insert (grow path) strong by reserving/committing so a throw cannot desync the parallel arrays; reorder rebuilds indices then swaps. - 0.4 Retire a slot instead of recycling it when its generation saturates, so a stale handle can never alias a freshly inserted row (ABA wraparound). - 0.5 Gate the sstd namespace alias behind SOATABLE_ENABLE_SSTD_ALIAS. Phase 1 - testing & quality gates: - Restructure tests into tests/unittests/ (7 -> 47 runtime tests). - Add move_only and throws_on_nth column types; prove strong guarantee. - Free-list churn, ABA stale-handle, large-table, select edge cases, const-correctness, row_handle and compression round-trip coverage. - Fixed-seed invariant/property test over 3000 randomized ops. - Compile-fail try_compile contracts and an include-hygiene TU test. - Strict-warning gate target and an ASan+UBSan CI leg. Verified: 50/50 ctest green; header clean under clang -Wconversion -Wsign-conversion -Werror and MSVC /W4 /WX; ASan+UBSan clean under churn. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move build options and tooling helpers (strict warnings target, clang-tidy, coverage, Doxygen, clang-format, AddressSanitizer) into cmake/Helpers.cmake, and pin the GoogleTest version in cmake/Versions.cmake. CMakeLists.txt now includes both. This captures a build-system refactor present in the working tree; it is independent of the library API work. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
IMPROVEMENTS Phase 2: - 2.1 Add row_view, a reference-semantic row proxy modelling the tuple protocol, plus view<>()/const view<>() so structured bindings yield real references (no reference_wrapper .get()); select<>() stays as the lower-level escape hatch. - 2.2 Const-correctness: compile-time static_assert tests proving get/try_get/ get_expected/select yield const references through a const table. - 2.3 Document the throw vs nullptr vs expected accessor contract; add non-throwing get_expected<T>() returning std::expected with an access_error enum (mutable and const overloads). - 2.4 Rename public types to std-style lowercase (soa_table, row_handle, column_vector, delta_value, dirty_mask) with [[deprecated]] PascalCase aliases for one migration window. Update tests, example, benchmark, README. New tests: view, expected, const-correctness, deprecated-alias (60 total). Verified: 60/60 ctest green; header clean under clang -Wconversion -Wsign-conversion -Werror and MSVC /W4 /WX; view/get_expected paths ASan+UBSan clean under churn. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- 3.1 Add CMakePresets.json (debug/release, gcc/clang/msvc, asan-ubsan, bench); raise the CMake minimum to 3.25 and bump the project version to 0.3.0. CI now configures, builds, and tests through these presets so local and CI builds cannot drift. - 3.2 Add CHANGELOG.md (Keep a Changelog; records the 0.1.0 -> 0.2.0 -> 0.3.0 history including the breaking snake_case renames) and CONTRIBUTING.md. - 3.3 Add a Conan recipe, a vcpkg overlay port, and scripts/amalgamate.py that emits a stamped single-header dist/soatable.hpp. - 3.4 Add docs/Doxyfile.in (wired through the build_documenation helper), a GitHub Pages docs workflow, and scripts/run_benchmarks.sh that records the toolchain and emits a JSON results file. - Update README installation section (presets, Conan, vcpkg overlay, amalgamation, docs) and the CMake version badge. Verified locally: 60/60 ctest green via the msvc preset; the bench preset builds the benchmark; the docs target generates HTML; the amalgamated header compiles clean under clang -Wconversion -Wsign-conversion -Werror. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- 4.1 Broaden the benchmark suite beyond the single select scenario: insertion, erase-churn, single-column and parallel sort, a select density sweep, and Array-of-Structures plus hand-rolled columnar scan baselines. - 4.2 Back sort_by_column_parallel with a serial size threshold (parallel_sort_threshold) and let the calling thread reorder the last column so it overlaps the async tasks instead of idling; document the threading model and why a persistent thread pool is intentionally avoided. Add a large-table test that exercises the concurrent path above the threshold. - 4.3 Add a driver-heuristic validation benchmark comparing the automatic smallest-column driver against a forced largest-column driver. Measured (250k rows, Release): smallest-driver select ~168us vs forced largest-driver ~1547us (~9x), confirming the heuristic. 61/61 tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…n (FEATURES Phase A.1/A.3/A.4) - A.1 Zero-copy column spans: column<T>() and a const overload return std::span over a column's dense values; row_indices<T>() and make_row_id() map dense positions back to stable handles. This is the keystone the compute layer, bindings, and serialization build on. - A.3 Validity bitmaps: a packed Arrow-style bitmap type plus validity<T>() reporting per-row column presence over the row-slot space, with for_each_set for branchless masked iteration and words() for interchange. - A.4 Serialization: opt-in <soatable/serialize.hpp> with save()/load(), a versioned, schema-fingerprinted binary snapshot for trivially-copyable columns that round-trips handles, generations, the free list, and physical column order. A friend table_access keeps the core's public surface unchanged. New tests: span, validity, serialize (78 total). Verified: 78/78 ctest green; headers clean under clang -Wconversion -Wsign-conversion -Werror and MSVC /W4 /WX; save/load round-trip ASan+UBSan clean under churn + sort. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Over-align dense column storage to simd_alignment (64 bytes, matching Apache Arrow) through an internal aligned_allocator, so column<T>() spans are ready for aligned SIMD loads and are cache-line friendly. column_vector now stores its data in an aligned_vector; raw_data() and reorder use the same aligned type. The optional AoSoA tiled-layout policy (the other half of A.2) is intentionally deferred as a separate, larger change. Added a runtime alignment assertion. Verified: 79/79 ctest green; clean under clang -Wconversion -Wsign-conversion -Werror and MSVC /W4 /WX; ASan+UBSan clean under churn + sort (no aligned new/delete mismatch). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Introduce a column storage policy so the table's layout is selectable without changing the row-facing API: - Refactor soa_table into basic_soa_table<Storage, Columns...>; soa_table is now an alias for the contiguous SoA default (non-breaking) and aosoa_table<TileSize, Columns...> selects tiled storage. column_vector gains a Storage parameter. - Add soa_layout / aosoa_layout<TileSize> policies and detail::tiled_vector, a sequence of fixed-size, individually 64-byte-aligned tiles whose growth appends a tile and never copies existing tiles (bounded reallocation, stable element addresses). - column<T>() is constrained to contiguous storage; column_tiles<T>() gives a uniform per-tile aligned-span view across both policies. serialize.hpp handles both layouts (raw fast path for SoA, per-element for tiled). New aosoa tests (86 total). Verified: 86/86 ctest green; headers clean under clang -Wconversion -Wsign-conversion -Werror and MSVC /W4 /WX; AoSoA churn + sort + tile traversal + save/load ASan+UBSan clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add opt-in <soatable/compute.hpp>, the start of the compute layer over the A.1 column spans: - Single-column span ufuncs (transform, reduce, inclusive_scan, count_if) that rely on compiler auto-vectorization over the 64-byte-aligned storage. - Table column helpers (transform_column, reduce_column, count_column_if) applied through column_tiles<T>(), so they work uniformly for SoA and AoSoA storage. - Cross-column row-wise assign_from<Out, In...>() that computes an output column from inputs over the select/view join (e.g. pnl = price * qty), in two passes so assigning the output never invalidates the in-progress join. New compute tests (91 total). Verified: 91/91 ctest green; clean under clang -Wconversion -Wsign-conversion -Werror and MSVC /W4 /WX; compute over AoSoA under churn ASan+UBSan clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…se B.2) Add opt-in <soatable/query.hpp>: - select_where<Cols...>(table, pred): lazily composes view<Cols...>() with a row predicate (the smallest-driver scan plus a filter), yielding row_view proxies. - group_reduce<Key, Value>(table, key_of, init, op) and the conveniences group_sum / group_count: group rows by a projected key and fold a value column, building analytics aggregates on top of select<>(). New query tests (96 total). Verified: 96/96 ctest green; clean under clang -Wconversion -Wsign-conversion -Werror and MSVC /W4 /WX; query under churn ASan+UBSan clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extend <soatable/compute.hpp> with the subset/broadcast adapters that feed the B.1 ufuncs: - broadcast / add_scalar / multiply_scalar: apply a scalar across a column span. - transform_if (value-flagged), transform_masked (parallel byte mask), and transform_strided (every n-th element) for operating on a subset of a column. - Table-level broadcast_column and transform_column_if, applied through column_tiles<T>() so they work for both SoA and AoSoA storage. This completes FEATURES Phase B (compute layer). New tests (102 total). Verified: 102/102 ctest green; clean under clang -Wconversion -Wsign-conversion -Werror and MSVC /W4 /WX; B.3 ops over AoSoA under churn ASan+UBSan clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add the soa_layout_with<Allocator> storage policy and the custom_soa_table< Allocator, Columns...> alias, so column storage can use any caller-supplied allocator template. The default soa_layout now derives from soa_layout_with<aligned_allocator>, keeping the 64-byte over-alignment and the existing API unchanged. - Add opt-in <soatable/pmr.hpp> with pmr_allocator and pmr_soa_table, routing column allocations through the active std::pmr memory resource (arena / monotonic buffer / pool) for deterministic, fragmentation-free behaviour. New allocator tests (105 total). Verified: 105/105 ctest green; clean under clang -Wconversion -Wsign-conversion -Werror and MSVC /W4 /WX. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add SOATABLE_NO_EXCEPTIONS: an internal SOATABLE_THROW macro redirects every throw site to a terminating handler (detail::terminate_with) when the macro is defined, and to the original exception type otherwise, preserving the public exception contract in normal builds. Callers use the existing non-throwing get_expected<T>() / try_get<T>() accessors; the library uses no RTTI. The macro is #undef'd at end of header for hygiene; the no-exceptions includes (<cstdio>/<cstdlib>) are pulled in only in that mode. Add a soatable_no_exceptions_check CTest target built with the macro (and -fno-exceptions on GCC/Clang) that runs the non-throwing accessor paths. Verified: 106/106 ctest green; normal mode preserves throws; no-exceptions mode compiles under -fno-exceptions and behaves correctly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add <soatable/reflect.hpp>: table_for<Struct> resolves a struct to its soa_table type via a columns_of<Struct> mapping. Today, a one-line column_list specialization declares a schema's columns portably; the SOATABLE_HAS_REFLECTION feature macro (0 until a toolchain implements C++26 P2996 reflection) gates the future automatic member-enumeration path, so the explicit type list remains the portable fallback. New reflect tests (109 total). Verified: 109/109 ctest green; clean under clang -Wconversion -Wsign-conversion -Werror and MSVC /W4 /WX. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add the chunked_soa_table<ChunkSize, Columns...> alias, framing the tiled storage policy as Arrow-style record batches: each column is a sequence of fixed-size over-aligned chunks, so growth never copies existing chunks and chunks are the unit for streaming and parallelism. - Add compute::for_each_chunk (per-chunk kernel hook) and compute::transform_column_parallel, which dispatches a column's chunks across threads above a size threshold (chunks are disjoint, so no data race) and falls back to serial otherwise. New tests (111 total). Verified: 111/111 ctest green; clean under clang -Wconversion -Wsign-conversion -Werror and MSVC /W4 /WX. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add opt-in <soatable/concurrent.hpp> synchronized_table: a std::shared_mutex wrapper around any table giving many concurrent readers and exclusive writers through scoped read()/write() callbacks that hold the lock for the callback's duration. This covers the "many strategy threads read while one ingests" case. Documented as reader-writer synchronization (not lock-free MVCC or sharding, which remain future enhancements). New concurrent tests, including a 4-reader / 1-writer stress test asserting a consistent final state (113 total). Verified: 113/113 ctest green (concurrency test stable across repeated runs); clean under clang -Wconversion -Wsign-conversion -Werror and MSVC /W4 /WX. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add opt-in <soatable/mmap.hpp> with mmap_allocator and mmap_soa_table: column storage is served from page-granular, demand-paged virtual memory (mmap with MAP_ANONYMOUS on POSIX, VirtualAlloc on Windows), so a column can be larger than the resident set and only touched pages occupy physical RAM. Mappings are page-aligned, so columns remain SIMD-friendly. Composes with the C.2 allocator support; reserve() maps the region once. Backing is anonymous demand-paged virtual memory; persistent named-file mapping is documented as a future extension. New mmap tests (116 total). Verified: 116/116 ctest green on the Windows VirtualAlloc path; clean under clang -Wconversion -Wsign-conversion -Werror and MSVC /W4 /WX. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add four worked, runnable examples under example/cookbook/, one per target domain, each built as a soatable_cookbook_* target and linked from the README: - finance_tick_store: ingest trades, derive notional per row with assign_from, aggregate volume per symbol with group_sum, filter with select_where. - game_ecs: a sparse-component world with a motion system over view<Position, Velocity>() and a data-less Frozen tag. - aerospace_telemetry: delta_value altitude tracking, dirty_mask per-frame subsystem flags, and a validity bitmap count. - scientific_table: zero-copy span reduction and quantized_float confidence. A shared portable OUT_PRINTLN helper keeps the examples building on toolchains without <print>. All four compile and run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add opt-in <soatable/timeseries.hpp> built on the column spans and the delta_value / dirty_mask helpers: - rolling_sum, rolling_mean, and deltas over a column's ordered values (running accumulators, O(n)). - rolling_mean_column projecting a struct column to double in dense order. - for_each_dirty<MaskColumn>: scan only the rows whose dirty-mask column reports a dirty flag and hand the mask to the callback to clear, enabling incremental recompute. New time-series tests (121 total). Verified: 121/121 ctest green; clean under clang -Wconversion -Wsign-conversion -Werror and MSVC /W4 /WX. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add tools/visualizers/: an MSVC soatable.natvis and a GDB pretty-printer module (soatable_gdb.py) that render the key types (row_id, delta_value, dirty_mask, quantized_float, bitmap, column_vector, basic_soa_table, row_handle) as compact summaries; a table expands into its columns and row metadata. A README documents loading them in Visual Studio, GDB, and LLDB, and the package install now ships them under <datadir>/soatable/visualizers. These are best-effort DX aids (a debugger session is required, so they are not CI-tested); the .natvis XML is validated as well-formed and the GDB module byte-compiles. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Complete Phase E with the two schema/metadata pieces: - <soatable/units.hpp>: a compile-time dimensional-analysis quantity<T, Dimension>. Add/subtract require matching dimensions (a mismatch is a compile error, with a WILL_FAIL contract test); multiply/divide combine dimensions. Quantities work directly as column types, giving engineering dimensional safety at op boundaries. Common aliases (length, duration, velocity, force, ...) provided. - <soatable/dynamic.hpp>: a type-erased dynamic_table that adds/removes typed columns by name at runtime, type-checks set<T>/get<T> without RTTI (per-type key addresses), stores cells sparsely per column, and carries per-column string metadata (e.g. unit labels). New units and dynamic tests plus a dimension-mismatch compile-fail (133 total). Verified: 133/133 ctest green; clean under clang -Wconversion -Wsign-conversion -Werror and MSVC /W4 /WX; dynamic-table churn ASan+UBSan clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… Doxygen file headers - README: restructured into a practical what/why/how tour of every feature area, with explicit "when SoaTable is the right tool" guidance, an opt-in headers table, and updated install/benchmark/build sections. - example/soatable_example.cpp: rewritten as a guided feature tour exercising the container, views, compute, query, serialization, AoSoA storage, units, time-series, and the dynamic table in one runnable program. - benchmarks: add SoA-vs-AoSoA dense selection and compute throughput (transform_column / reduce_column) benchmarks. - scripts/amalgamate.py: read the version from VERSION.txt and add --with to inline chosen add-on headers into the single-header bundle; run_benchmarks.sh binary lookup made portable. - Add Doxygen file-level headers (/// @file, @brief, @author Bertin Balouki SIMYELI) to every .cpp/.hpp across include, tests, example, and benchmarks; the @file block now precedes #pragma once. Verified: 133/133 ctest green; the feature-tour example and both amalgamation variants (core and core+compute+query) compile and run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
synchronized_table::write uses std::unique_lock, which lives in <mutex>. Only <shared_mutex> and <utility> were included; std::shared_lock came in transitively and masked the gap, but GCC 13 does not transitively provide std::unique_lock, breaking the concurrent_test build. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
libstdc++'s <expected> only declares std::expected under __cplusplus > 202002L, so on toolchains where the C++23 library feature is inactive the get_expected() declarations were preprocessed away, leaving stray std::expected/std::unexpected references that failed to compile. Detect the feature (mirroring the existing SOATABLE_HAS_PRINT pattern) and declare both get_expected() overloads only when available. try_get<T>() remains the always-available non-throwing accessor. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The tests exercise get_expected(), which is now only declared where std::expected is available. Wrap them in the same feature guard so the translation unit compiles to no tests on toolchains without it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The negative compile-time-contract tests passed build/target/config without the leading -- option prefixes, so CMake ran a configure (succeeding) instead of building the target that must fail to compile. With WILL_FAIL TRUE the success was reported as a test failure on MSVC multi-config. Use --build/--target/--config so the failing target is actually built, making the contract deterministic across generators. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Grow the buffer then std::memcpy into it rather than vector::insert over a reinterpreted byte range; GCC mis-analyzes the latter and warns when it cannot prove the source object's size. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Discard the returned row_id explicitly via std::ignore to clear the -Wunused-result warning. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Set the existing SOATABLE_CXX_STANDARD project variable (consumed by cmake/Helpers.cmake) to 23 in the base configure preset, making the C++23 requirement explicit and visible in the CI configuration. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Allow forcing the non-std::expected path even on a capable toolchain. This documents the feature seam and makes the fallback branch testable on any compiler (e.g. clang++ -DSOATABLE_DISABLE_EXPECTED). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The const-overload return-type assertion references std::expected directly, which is absent on toolchains lacking the C++23 library feature (e.g. the Clang CI leg). Wrap it in SOATABLE_HAS_EXPECTED. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The strict-warning compile gate calls get_expected(), which only exists where std::expected is available. Wrap it in SOATABLE_HAS_EXPECTED so the gate builds on the Clang CI leg. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The freestanding gate exercised get_expected() unconditionally. Guard those checks behind SOATABLE_HAS_EXPECTED and provide an equivalent try_get()-based fallback so the no-exceptions contract is still verified where std::expected is unavailable (e.g. the Clang CI leg). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A table with a fully-empty column serializes that column with zero dense entries. On a successful load(), byte_reader::read_into() then called std::memcpy(dst, src, 0) where dst is the null data() pointer of an empty destination vector. glibc declares memcpy with __nonnull__, and the asan-ubsan CI leg (-fsanitize=undefined includes nonnull-attribute on Linux/libstdc++) aborts on the null argument. This was only reachable on a successful load with an empty column (e.g. RoundTrip*AfterErase / AfterSort); the reject-* tests bail before the column loop, which is why they passed. Make the zero-length read a no-op, and symmetrically guard append_array against forming pointer arithmetic on a null data() pointer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.