Exploring columnar simplification#685
Draft
frankmcsherry wants to merge 3 commits intoTimelyDataflow:master-nextfrom
Draft
Exploring columnar simplification#685frankmcsherry wants to merge 3 commits intoTimelyDataflow:master-nextfrom
columnar simplification#685frankmcsherry wants to merge 3 commits intoTimelyDataflow:master-nextfrom
Conversation
Remove the Layout/WithLayout/LayoutExt trait tower and replace with direct Columnar-based type constraints. All batch containers are now Coltainer<C> (wrapping C::Container from the columnar crate). Key changes: - Add Data trait: Columnar + Ord + Clone + 'static with ordered refs - Add Coltainer<C> as the sole BatchContainer for data columns - Remove Layout, WithLayout, LayoutExt traits and Vector/TStack structs - Remove BuilderInput trait (Columnar provides owned<->ref relationship) - Simplify TraceReader/BatchReader/Cursor to use type Key/Val/Time/Diff directly instead of 12-line LayoutExt bound blocks - Add Columnar bound to crate-level Data trait - Update all operators, wrappers, algorithms for columnar Ref types 28 files changed, ~980 insertions, ~1610 deletions (net -630 lines). Known TODOs: - Enter/enter_at cursor map_times needs container for synthesized times - join_map/reduce convenience APIs convert Refs to owned (temporary workaround until push-into-container API redesign) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…orts Implement the container approach for enter/enter_at/frontier cursor wrappers: synthesized times are pushed into a TInner::Container, then replayed as columnar Refs. Diffs use a temporary container similarly. Also removes unused imports from cargo fix. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Update dogsdogsdogs, experiments, server, and example files: - Replace KeyOwn/ValOwn/TimeGat/DiffGat with Key/Val/Time/Diff - Add Data bounds on Timestamp where arrangements are created - Convert join_core/as_collection closures for columnar Ref types - Add columnar::Columnar derives for custom types (AltNeu, Pair) - Exclude interactive crate (recursive Value enum needs manual Columnar impl) - Add explicit Semigroup<&Present> impl to prevent recursion overflow Remaining: a few examples (progress.rs, pagerank.rs) and tests need similar mechanical updates for Ref types with isize keys. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An exploration of removing DD generality in favor of
columnarstorage, with an eye towards removing towers of abstractions to support hypothetical generality. AndVec; it would remove support forVecas well.A work in progress for now, as we see how far Claude was able to get, and with which unacceptable compromises along the way.