Skip to content

fix: two column lineage correctness bugs (nondeterministic ordering, fabricated origins) - #12

Open
eitsupi wants to merge 2 commits into
funcpp:mainfrom
eitsupi:fix/lineage-correctness
Open

fix: two column lineage correctness bugs (nondeterministic ordering, fabricated origins)#12
eitsupi wants to merge 2 commits into
funcpp:mainfrom
eitsupi:fix/lineage-correctness

Conversation

@eitsupi

@eitsupi eitsupi commented Aug 22, 2026

Copy link
Copy Markdown

Note

This change was written by Claude Code and Codex under my review and direction.
I have verified the diff, run the test suite, and confirmed
the behavior change against a probe crate. Please review it as you would any other
patch. Happy to split this into two PRs if you prefer one fix per PR — the two
commits are independent and can be cherry-picked separately.

Two independent column lineage correctness bugs, one commit each, both against the 0.2.0 public API (no type or signature changes).

1. Nondeterministic column mapping order (56677a0)

resolve iterated final_ids as a HashSet, so the order of the resulting
ColumnMappings varied between runs. The later sort_by_key keys on the output
column name, so it does not stabilize outputs that share a name:

SELECT a.id, b.id FROM a JOIN b ON a.id = b.bid

Here both mappings target id, and which one came first was decided by hash
iteration order. Callers that match mappings positionally saw them swap.

The fix walks the scope's ordered_cols, which already carries projection order,
instead of collecting into a set first. Verified by re-running the new test 12 times
against the unfixed resolver (12/12 failures) and 50 times against the fixed one
(stable).

2. Fabricated column origins for unresolved columns (e8e4790)

When a column could not be resolved, the resolver returned a fully resolved
ColumnOrigin::Concrete carrying an invented table name:

  • ?unknown? when no binding was visible in scope
  • ?cte? when the target CTE or derived table had no output column of that name
SELECT bare_col
-- before: bare_col <- ?unknown?.bare_col   (Concrete)

WITH cte AS (SELECT present FROM source) SELECT missing FROM cte
-- before: missing <- ?cte?.missing         (Concrete)

Concrete means "this column comes from this table". These origins violated that
contract, and a consumer had no way to tell fabricated lineage from real lineage
short of string-matching the sentinel table names. For a lineage tool, a confident
wrong answer is worse than an honest unknown.

Both sites now return ColumnOrigin::Ambiguous with an empty candidates list,
which already carries the meaning "not resolved to a table; a catalog is needed".

Returning None instead was considered and rejected: collect_leaf_origins maps
None to an empty vector, so the unresolved source would silently disappear from an
otherwise valid mapping (SELECT missing + known FROM t would keep only known),
and an empty sources list already has a legitimate meaning for constant
expressions.

apply_catalog guard

This part is load-bearing rather than incidental. apply_catalog rewrote every
Ambiguous into Concrete whenever CatalogProvider::resolve_column returned
Some. Without a guard, a provider that resolves a column by name alone would turn
the new empty-candidate Ambiguous straight back into fabricated Concrete
lineage, undoing the fix. apply_catalog now attempts catalog refinement only for
non-empty candidate sets.

The cost is that a catalog can no longer do a global by-name lookup for a column
with no query-visible table. That seems right: a column existing somewhere in the
catalog is not evidence that its table participates in this query, and in the
missing-CTE-column case it is especially likely to be wrong. This boundary is now
pinned from both sides by tests — genuine ambiguity is still resolved by a catalog,
an unresolved origin is not.

ColumnOrigin::Ambiguous and CatalogProvider::resolve_column are documented
accordingly, and the Python stub notes that candidates may be empty.

Compatibility

No public type, variant, or signature changes, so this fits a 0.2.x patch release.
Observable results do change for previously unresolved columns: a Concrete with a
sentinel table becomes an Ambiguous with no candidates, and the CLI prints ?x?
where it printed ?unknown?.x. That is the point of the fix, but if you would
rather ship it as 0.3.0 that is entirely reasonable.

Tests

93 pass (was 90). New:

  • unresolved_column_has_empty_ambiguous_candidates
  • missing_column_from_cte_has_empty_ambiguous_candidates
  • catalog_does_not_fabricate_unresolved_column_owner
  • two tests covering deterministic ordering with duplicate output names

cargo clippy --all-targets reports one unneeded_struct_pattern warning in
build/statement.rs, which is pre-existing on main and untouched here.

cargo fmt was deliberately not run: main is not rustfmt-clean, and running it
would bury these changes in unrelated reformatting.

eitsupi and others added 2 commits August 22, 2026 01:24
The order of `ColumnLineage.mappings` varied between runs of the same
binary whenever a query had duplicate output column names:

    SELECT a.id, b.id FROM a JOIN b ON a.id = b.bid
      sometimes [ id <- a.id , id <- b.id ]
      sometimes [ id <- b.id , id <- a.id ]

With three duplicates, four distinct orderings showed up across six runs.
Callers that cache or diff results see the same input produce different
output.

`resolve` collected the output nodes into a `HashSet<NodeId>` and built
the mappings by iterating it, so construction order followed hash order
with a per-process random seed. The sort afterwards could not undo this:
it keyed on a `HashMap<String, usize>` of output names, so duplicate
names collided and one index won, leaving same-named mappings in
whatever order the set had produced.

`ordered_cols` already holds the projection order, so build the mappings
straight from it. That makes the sort redundant — it can only reproduce
the order the loop now has, and it is the reason duplicates were
reordered in the first place — so it goes too.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
The resolver returned ColumnOrigin::Concrete with invented table names
("?unknown?", "?cte?") when a column could not be resolved, so callers
could not tell fabricated lineage from real lineage.

Both sites now return ColumnOrigin::Ambiguous with an empty candidate
list, which already means "catalog needed to resolve" and keeps the
0.2.x public API unchanged. Returning None instead would drop the source
from the mapping entirely, and an empty source list already has a valid
meaning (constant expressions).

apply_catalog now skips empty candidate lists. Without that guard a
CatalogProvider resolving a column by name alone would turn an
unresolved origin straight back into a fabricated Concrete one.
@eitsupi
eitsupi marked this pull request as ready for review August 22, 2026 05:04
eitsupi added a commit to eitsupi/dlin that referenced this pull request Aug 22, 2026
The pinned revision is the head of funcpp/sqllineage#12, and GitHub
resolves a pull request's commits from the upstream URL, so the fork URL
does not need to be baked into the manifest. When the fixes land, only the
rev goes away.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AxLtpvd6PeK8gbgcMzVh4n
eitsupi added a commit to eitsupi/dlin that referenced this pull request Aug 22, 2026
Adds a `sqllineage`-backed column lineage backend behind the backend boundary from
#135 and #136. Production still runs polyglot; the new backend is reachable only from
tests.

## Why sqllineage

polyglot-sql returns lineage it cannot justify, and dlin has spent a lot of code
compensating for that from the outside. sqllineage's result model maps almost one to
one onto the backend contract: `Concrete`, `Ambiguous` with candidates, `Wildcard`,
`Recursive`. That correspondence is what makes this worth doing. The adapter is about
four hundred lines, and none of polyglot's correction layer had to come with it.

Two correctness bugs in sqllineage itself were fixed first and are pending upstream
(funcpp/sqllineage#12): mapping order was nondeterministic for duplicate output names,
and unresolved columns came back as `Concrete` origins carrying invented table names,
which no caller could tell apart from real lineage. The dependency is pinned to that
revision until they are released.

## Why it is not switched on

The two engines disagree, and the disagreements have to be adjudicated case by case
against a fixture matrix before either is called correct. Landing the backend first
keeps that comparison reviewable on its own, and keeps this change from being both a
new engine and a behavior change at once.

## The principles the adapter follows

**Nothing is decided by a sentinel string.** An `Ambiguous` origin with candidates is
genuine ambiguity; the same origin with an empty candidate list is an unresolved
column. Both fail the output rather than emitting a plausible edge. Origin and
transform translation are exhaustive matches with no wildcard arm, so a new sqllineage
variant fails to compile until someone decides what it means.

**The catalog refuses when it is not certain.** Relation matching requires full arity,
with no suffix or truncation fallback, and unknown, empty, and conflicting entries all
resolve to nothing rather than a guess. A schema that answers confidently when it
should not is a source of fabricated lineage, not a convenience.

**Where the engine cannot be trusted, dlin declines rather than repairs.** sqllineage
joins set-operation branches before a leading `SELECT *` has a known width, so another
branch's contribution can be dropped while the result still looks successful. dlin
detects that shape and reports the statement indeterminate. Reconstructing the intended
lineage from the outside is exactly the kind of compensation this migration exists to
stop.

**Backend capability stays inside the backend.** `DlinDialect` is dlin's domain, not a
statement about what any engine implements, so the conversion into sqllineage's
dialects is fallible and refuses what it cannot serve. What the CLI should do when the
selected backend cannot serve the requested dialect is a policy question that belongs
to the change that switches backends.

## User-visible change

One: MCP's `--dialect` is no longer required, and is inferred from
`manifest.metadata.adapter_type` the way the `column` subcommands already do. A
missing, empty, or unrecognized adapter type still fails at startup, because falling
back to generic silently produces wrong lineage for warehouse-specific SQL.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant