Skip to content

feat(inspect): PoC for parallel and serializable metadata tasks - #22

Draft
manuzhang wants to merge 1 commit into
codex/metadata-scan-support-20261115from
codex/poc-parallel-metadata-tasks-20261120
Draft

manuzhang wants to merge 1 commit into
codex/metadata-scan-support-20261115from
codex/poc-parallel-metadata-tasks-20261120

Conversation

@manuzhang

Copy link
Copy Markdown
Owner

Purpose

Executable PoC for the parallel task execution and serialization discussion in apache#1120 (discussion_r4638022509).

This draft targets codex/metadata-scan-support-20261115 in this fork, at e5c816e6, so the diff contains only the prototype. It does not modify the upstream PR branch.

What it demonstrates

  • Resolve the snapshot once and plan one distinct work unit per data manifest.
  • Open each task through MetadataScanTaskReader with independent read state.
  • Execute through the existing TaskGroup/Executor in bounded windows, preserve planning order, and drain active work before propagating failures.
  • Serialize a versioned descriptor containing the fixed snapshot, assigned manifest, source schema/spec, and projection. Workers inject their own FileIO; no callbacks, pointer addresses, or driver table objects cross the wire.
  • Compare sequential execution with deserialized parallel execution over real Avro manifests. The concurrency test destroys driver resources, uses separate worker storage with no manifest list, and makes two worker opens rendezvous to prove overlap without timing-based speed assertions.

The experimental helper API is internal and is not installed. Usage and design details are in docs/parallel-metadata-poc.md.

Deliberate PoC limits

The output is four data-file metadata columns, and only unencrypted data manifests are supported. Execution eagerly buffers all result batches; the task limit bounds concurrency, not output memory. A remote scheduler, streaming/backpressure, cancellation, the complete FilesTable schema, and partition-summary reduction are follow-up work. No throughput improvement is claimed by these correctness tests.

Validation

  • All 70 metadata tests passed: 62 existing tests and 8 PoC tests.
  • C++20 public-header compatibility passed.
  • Required pre-commit and clang-tidy 22.1.8 passed.

Focused local validation rebuilt the relevant implementations and tests against this branch using unchanged cached static dependencies.

Plan one fixed task per data manifest and dispatch independent readers through bounded TaskGroup execution. Add versioned JSON descriptors with worker-injected FileIO, real manifest round-trip/concurrency/error tests, and documentation of the prototype boundaries.

Co-authored-by: Codex <codex@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant