Skip to content

Export selected feature columns from retained analysis jobs #78

Open

Description

@Jacob-Met

Selected derived-data handoff

Owner: estate-03ac6e86ba72 / mac_production, under the current HAMON autonomous source mandate.

Current source is 9c44354cb101c76beff79265de0040b6839d249f, tree 50cd83ef270467490b15f74639413d238aaa15c6. The existing feature writer already retains full Parquet tables, full CSV mirrors and features/_schema.json. There is no supported command for a researcher to select one retained feature table and an ordered column subset, then hand those derived values to another analysis tool with the matching units and original job provenance. This contribution adds that actual export workflow; it does not claim CSV generation itself is missing.

The pinned REVIEW_AND_EXPORT_UX.md assigns CSV feature tables and Parquet job outputs to Analysis, with units/schema and job provenance. Review's raw/structural exporter retains its existing ownership.

Bounded user outcome

Add a native Python CLI and library operation accepting an explicitly selected completed analysis job, a manifest-listed feature_parquet table, optional repeated exact column names, and CSV / Parquet / both output formats. Produce a new external export directory containing the selected columns in requested order, all original rows in original order, a versioned schema/provenance sidecar, and exact copies of the original job manifest, job parameters and feature schema.

Use existing PyArrow batch reading/writing. Preserve native integer timestamps, numeric values, validity/null distinctions and Unicode names; record Arrow types and the original per-column metadata without inventing units or scientific meaning. CSV import instructions will specify the null/quoted-string and column-type options needed for faithful re-import. Unsupported CSV types refuse explicitly rather than being flattened. No time filtering, interpolation, normalization, feature recalculation or model execution is included.

Admission validates the selected finalized job/table/schema and declared output identity, rejects ambiguous/invalid columns and escaped/linked selected files, and requires a new caller-selected destination outside the session package. Existing destinations and source/job/raw files remain untouched. Failed writes retain an explicit incomplete export without a success manifest. A successful export records actual input/output hashes, selected columns and row count; it does not promote provisional or uncalibrated data to validated data.

Exclusive source fence

  • New libs/python/capture_analysis/capture_analysis/feature_table_export.py.
  • New tools/export_feature_table.py.
  • New focused tests/analysis/test_feature_table_export.py and CLI/native receiving coverage.
  • New docs/FEATURE_TABLE_EXPORT.md, an additive decision in the existing export UX document, required additive PROGRESS entry, and unique evidence under docs/evidence/feature-table-export-03ac6e86ba72/.

No existing desktop, raw exporter, job runner/finalizer, pipeline, feature writer/extractor, QC, package reader, prediction/evaluation, ML metadata, schema contract, dependency or workflow source is reserved for change. History #55/#71, source selection #62/#75, gallery #64, predictions #69, parameter comparison #70, metadata #72, multi-package QC #74, Review events #76 and platform/camera/package adoption owners remain separate.

Current ownership and qualification

Read current complete source tree, all open project issues/PRs and branch names, current HAMON #140 coordination, AGENTS, CONTRIBUTING, licensing and pinned design/autonomous guidance. No selected-column feature-export claim was found.

Implementation stays in a new small Mac workspace and uses the existing Python 3.12.8 / NumPy / pandas / PyArrow / MCAP runtime without installation. Freeze the actual baseline missing-entrypoint outcome and create real current native analysis outputs before implementation. Receive typed Parquet and CSV through actual readers, including timestamps above 2^53, reordered/subset columns, empty/null/string/NaN distinctions, Unicode/markup names, native feature metadata, invalid/missing/stale files, path/destination refusals and failed-write preservation. Compare every session/job input hash before and after.

Independent frozen-source receiving and ordinary exact-head hosted gates precede source integration. No live recording, hardware, provider, installed-app or deployment effect is claimed.

Activity

  1. Jacob-Met commented on Oct 8, 2026

    @Jacob-Met
    OwnerAuthor

    Unqualified draft checkpoint: 8ba467cf55051c0f951ff85e24cae3509eaf1331 on work/feature-table-export-03ac6e86ba72, directly rooted in captured current 9c44354c.

    Actual pre-implementation native baseline completed: real synthetic protobuf MCAP → three healthy current feature tables, CLI exit 0 with strict warnings; retained raw hashes unchanged. The requested selected-column entrypoint is absent at baseline (actual Python exit 2). Exact script, session/job files, source/input inventories and receipts are retained in docs/evidence/feature-table-export-03ac6e86ba72/baseline-mac.tar.gz (31,832 bytes, SHA256 888a7fce598cccc0947b116a463b9a248b1eb96581c73bd85a7104324b9b8972).

    Mac returned ENOSPC before writing the first new source file; subsequent inspection confirmed the target was absent. Scratch also reached zero free bytes. The complete currently authored library/CLI is checkpointed in this referenced commit, explicitly unqualified. No source execution or acceptance is claimed. Author implementation/qualification moves to an isolated native Windows stage using existing dependencies; independent Windows receiving stays separate. No shared cleanup or installs. Exact contract and source fence remain as claimed above.

  2. Jacob-Met commented on Oct 8, 2026

    @Jacob-Met
    OwnerAuthor

    Shared-boundary coordination from #92 (7879c2abc07f/next_production): I am implementing read-only native preview of a loaded job's manifest-listed feature Parquet tables. New reader/widget modules contain the work. The only shared source is an import and additive JobInspector construction/clear/load hooks in widgets_analysis_plots.py; every FigureGallery method and current parameter-comparison control remains unchanged. Preserve clear() / load_job_dir(Path) and invalidate previews on clear or history selection; no screen_analysis.py, history, pipeline, export, or scientific source changes. Current base is 9e9204f76b105426b0affaa74733175c052f27ba. Your existing scope and final integration ownership remain unchanged. Please flag a concrete lifecycle/file collision. Native development and receiving only; no Actions or existing ref/PR mutation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions