Repository navigation
Conversation
Plan one fixed task per data manifest and dispatch independent readers through bounded TaskGroup execution. Add versioned JSON descriptors with worker-injected FileIO, real manifest round-trip/concurrency/error tests, and documentation of the prototype boundaries. Co-authored-by: Codex <codex@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Executable PoC for the parallel task execution and serialization discussion in apache#1120 (discussion_r4638022509).
This draft targets
codex/metadata-scan-support-20261115in this fork, ate5c816e6, so the diff contains only the prototype. It does not modify the upstream PR branch.What it demonstrates
MetadataScanTaskReaderwith independent read state.TaskGroup/Executorin bounded windows, preserve planning order, and drain active work before propagating failures.The experimental helper API is internal and is not installed. Usage and design details are in
docs/parallel-metadata-poc.md.Deliberate PoC limits
The output is four data-file metadata columns, and only unencrypted data manifests are supported. Execution eagerly buffers all result batches; the task limit bounds concurrency, not output memory. A remote scheduler, streaming/backpressure, cancellation, the complete FilesTable schema, and partition-summary reduction are follow-up work. No throughput improvement is claimed by these correctness tests.
Validation
Focused local validation rebuilt the relevant implementations and tests against this branch using unchanged cached static dependencies.