Graph explorer: per-version column table and column lineage walk (variable-level traceability, not just process tracing) #865

Activity

1. juaristi22 commented on Oct 1, 2026

   

   juaristi22

   on Oct 1, 2026

   Collaborator

   More actions

   Picking this up as the variable-lineage workstream for Microcosm → Orrery. Keep the work in this issue, alongside two companion drafts covering country-agnostic graph export and execution-evidence overlays. The UK will be the first consumer; the shared metadata and exporter should support any Microcosm country graph.

   Orrery's current main already provides an orrery-lineage/v1 extension with variable and stage selection, explicit value/control/context roles, and source/unknown boundaries. Microcosm needs to supply the domain mappings. #888 provides related export groundwork, but currently exports operation-level relationships without these precise lineage annotations.

   Proposed scope for this issue

   • Carry forward the per-version column table, writer-chain and read/write requirements above. Export entity, column, dtype, row scope, ownership/absence, first appearance, and carrier/writer identities through one shared metadata API.
   • Reuse or factor the executor/compiler's writer resolution. Preserve field versions, pre-rewrite values, partial-row rewrites, structural carry-through, filtering, expansion/entrant materialization, and weight changes. Distinguish declared possibilities from what a run's receipts establish.
   • Add explicit per-output dependency mappings where the kernel's semantics support them. A stage reading several inputs and producing several outputs does not establish that every input determines every output. Preserve operation-level context and show an unknown boundary where a composite kernel lacks a justified mapping.
   • Export a variable catalog, stage choices, and supported value/control/context annotations using the existing graph node/edge identities. Structural ancestry alone does not establish unchanged values. Use Orrery's lineage contract where its semantics fit.
   • Keep static lineage usable without a completed run; attach validated runtime writer evidence through the execution overlay when available. Retain the kernel explanation requested above.

   Acceptance criteria

   • The metadata implementation is country-agnostic; UK fixtures provide the first integration coverage.
   • Per-version inventories match declarations, and runtime-backed writer claims match the supplied receipts.
   • Synthetic UK traces cover a carried source field, an imputed field, a partial-row rewrite, geographic expansion/materialization, and calibration weights or their explicit artifact/control representation.
   • Tests distinguish carriers, actual writers, values and controls, and preserve pre-rewrite identities. A multi-output kernel without sufficient mappings exposes an unknown boundary.
   • All lineage references resolve and annotations survive JSON/HTML round trips through a pinned Orrery build supporting the extension. Graphs without lineage remain supported.
   • Document supported mappings and remaining composite-kernel boundaries; share the same metadata with the native explorer where it consumes these facts.

   Once graph export, execution overlays and this lineage work are supported in Microcosm, use their artifacts to assess any remaining Orrery features and agree which repository should own those changes. New viewer features and hosted build discovery can be scoped at that point.