fix: two column lineage correctness bugs (nondeterministic ordering, fabricated origins) - #12
Open
eitsupi wants to merge 2 commits into
Open
fix: two column lineage correctness bugs (nondeterministic ordering, fabricated origins)#12eitsupi wants to merge 2 commits into
eitsupi wants to merge 2 commits into
Conversation
The order of `ColumnLineage.mappings` varied between runs of the same
binary whenever a query had duplicate output column names:
SELECT a.id, b.id FROM a JOIN b ON a.id = b.bid
sometimes [ id <- a.id , id <- b.id ]
sometimes [ id <- b.id , id <- a.id ]
With three duplicates, four distinct orderings showed up across six runs.
Callers that cache or diff results see the same input produce different
output.
`resolve` collected the output nodes into a `HashSet<NodeId>` and built
the mappings by iterating it, so construction order followed hash order
with a per-process random seed. The sort afterwards could not undo this:
it keyed on a `HashMap<String, usize>` of output names, so duplicate
names collided and one index won, leaving same-named mappings in
whatever order the set had produced.
`ordered_cols` already holds the projection order, so build the mappings
straight from it. That makes the sort redundant — it can only reproduce
the order the loop now has, and it is the reason duplicates were
reordered in the first place — so it goes too.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
The resolver returned ColumnOrigin::Concrete with invented table names
("?unknown?", "?cte?") when a column could not be resolved, so callers
could not tell fabricated lineage from real lineage.
Both sites now return ColumnOrigin::Ambiguous with an empty candidate
list, which already means "catalog needed to resolve" and keeps the
0.2.x public API unchanged. Returning None instead would drop the source
from the mapping entirely, and an empty source list already has a valid
meaning (constant expressions).
apply_catalog now skips empty candidate lists. Without that guard a
CatalogProvider resolving a column by name alone would turn an
unresolved origin straight back into a fabricated Concrete one.
eitsupi
marked this pull request as ready for review
August 22, 2026 05:04
eitsupi
added a commit
to eitsupi/dlin
that referenced
this pull request
Aug 22, 2026
The pinned revision is the head of funcpp/sqllineage#12, and GitHub resolves a pull request's commits from the upstream URL, so the fork URL does not need to be baked into the manifest. When the fixes land, only the rev goes away. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AxLtpvd6PeK8gbgcMzVh4n
eitsupi
added a commit
to eitsupi/dlin
that referenced
this pull request
Aug 22, 2026
Adds a `sqllineage`-backed column lineage backend behind the backend boundary from #135 and #136. Production still runs polyglot; the new backend is reachable only from tests. ## Why sqllineage polyglot-sql returns lineage it cannot justify, and dlin has spent a lot of code compensating for that from the outside. sqllineage's result model maps almost one to one onto the backend contract: `Concrete`, `Ambiguous` with candidates, `Wildcard`, `Recursive`. That correspondence is what makes this worth doing. The adapter is about four hundred lines, and none of polyglot's correction layer had to come with it. Two correctness bugs in sqllineage itself were fixed first and are pending upstream (funcpp/sqllineage#12): mapping order was nondeterministic for duplicate output names, and unresolved columns came back as `Concrete` origins carrying invented table names, which no caller could tell apart from real lineage. The dependency is pinned to that revision until they are released. ## Why it is not switched on The two engines disagree, and the disagreements have to be adjudicated case by case against a fixture matrix before either is called correct. Landing the backend first keeps that comparison reviewable on its own, and keeps this change from being both a new engine and a behavior change at once. ## The principles the adapter follows **Nothing is decided by a sentinel string.** An `Ambiguous` origin with candidates is genuine ambiguity; the same origin with an empty candidate list is an unresolved column. Both fail the output rather than emitting a plausible edge. Origin and transform translation are exhaustive matches with no wildcard arm, so a new sqllineage variant fails to compile until someone decides what it means. **The catalog refuses when it is not certain.** Relation matching requires full arity, with no suffix or truncation fallback, and unknown, empty, and conflicting entries all resolve to nothing rather than a guess. A schema that answers confidently when it should not is a source of fabricated lineage, not a convenience. **Where the engine cannot be trusted, dlin declines rather than repairs.** sqllineage joins set-operation branches before a leading `SELECT *` has a known width, so another branch's contribution can be dropped while the result still looks successful. dlin detects that shape and reports the statement indeterminate. Reconstructing the intended lineage from the outside is exactly the kind of compensation this migration exists to stop. **Backend capability stays inside the backend.** `DlinDialect` is dlin's domain, not a statement about what any engine implements, so the conversion into sqllineage's dialects is fallible and refuses what it cannot serve. What the CLI should do when the selected backend cannot serve the requested dialect is a policy question that belongs to the change that switches backends. ## User-visible change One: MCP's `--dialect` is no longer required, and is inferred from `manifest.metadata.adapter_type` the way the `column` subcommands already do. A missing, empty, or unrecognized adapter type still fails at startup, because falling back to generic silently produces wrong lineage for warehouse-specific SQL.
This was referenced Aug 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
This change was written by Claude Code and Codex under my review and direction.
I have verified the diff, run the test suite, and confirmed
the behavior change against a probe crate. Please review it as you would any other
patch. Happy to split this into two PRs if you prefer one fix per PR — the two
commits are independent and can be cherry-picked separately.
Two independent column lineage correctness bugs, one commit each, both against the 0.2.0 public API (no type or signature changes).
1. Nondeterministic column mapping order (56677a0)
resolveiteratedfinal_idsas aHashSet, so the order of the resultingColumnMappings varied between runs. The latersort_by_keykeys on the outputcolumn name, so it does not stabilize outputs that share a name:
Here both mappings target
id, and which one came first was decided by hashiteration order. Callers that match mappings positionally saw them swap.
The fix walks the scope's
ordered_cols, which already carries projection order,instead of collecting into a set first. Verified by re-running the new test 12 times
against the unfixed resolver (12/12 failures) and 50 times against the fixed one
(stable).
2. Fabricated column origins for unresolved columns (e8e4790)
When a column could not be resolved, the resolver returned a fully resolved
ColumnOrigin::Concretecarrying an invented table name:?unknown?when no binding was visible in scope?cte?when the target CTE or derived table had no output column of that nameConcretemeans "this column comes from this table". These origins violated thatcontract, and a consumer had no way to tell fabricated lineage from real lineage
short of string-matching the sentinel table names. For a lineage tool, a confident
wrong answer is worse than an honest unknown.
Both sites now return
ColumnOrigin::Ambiguouswith an emptycandidateslist,which already carries the meaning "not resolved to a table; a catalog is needed".
Returning
Noneinstead was considered and rejected:collect_leaf_originsmapsNoneto an empty vector, so the unresolved source would silently disappear from anotherwise valid mapping (
SELECT missing + known FROM twould keep onlyknown),and an empty
sourceslist already has a legitimate meaning for constantexpressions.
apply_catalogguardThis part is load-bearing rather than incidental.
apply_catalogrewrote everyAmbiguousintoConcretewheneverCatalogProvider::resolve_columnreturnedSome. Without a guard, a provider that resolves a column by name alone would turnthe new empty-candidate
Ambiguousstraight back into fabricatedConcretelineage, undoing the fix.
apply_catalognow attempts catalog refinement only fornon-empty candidate sets.
The cost is that a catalog can no longer do a global by-name lookup for a column
with no query-visible table. That seems right: a column existing somewhere in the
catalog is not evidence that its table participates in this query, and in the
missing-CTE-column case it is especially likely to be wrong. This boundary is now
pinned from both sides by tests — genuine ambiguity is still resolved by a catalog,
an unresolved origin is not.
ColumnOrigin::AmbiguousandCatalogProvider::resolve_columnare documentedaccordingly, and the Python stub notes that
candidatesmay be empty.Compatibility
No public type, variant, or signature changes, so this fits a 0.2.x patch release.
Observable results do change for previously unresolved columns: a
Concretewith asentinel table becomes an
Ambiguouswith no candidates, and the CLI prints?x?where it printed
?unknown?.x. That is the point of the fix, but if you wouldrather ship it as 0.3.0 that is entirely reasonable.
Tests
93 pass (was 90). New:
unresolved_column_has_empty_ambiguous_candidatesmissing_column_from_cte_has_empty_ambiguous_candidatescatalog_does_not_fabricate_unresolved_column_ownercargo clippy --all-targetsreports oneunneeded_struct_patternwarning inbuild/statement.rs, which is pre-existing onmainand untouched here.cargo fmtwas deliberately not run:mainis not rustfmt-clean, and running itwould bury these changes in unrelated reformatting.