Hi there! First of all, thank you for the amazing project—it's a very helpful tool.
I am experiencing an issue where column-level lineage is lost during the transition between the layers of my ETL pipeline. I am using dbt-spark, and I've noticed that the lineage "gap" occurs specifically between two models that utilize a ephemeral model as intermediate.
Context: I have 3 models, A, B and C, A and C are tables, but B is ephemeral. This models persist the same column id, where B consume id from A, and C consume the id from B.
The issue is: The column lineage is interrupted at Model B. The UI displays the column from Model A in isolation and the column from Model C in isolation, failing to establish the connection between them. Model B does not appear in the column lineage graph at all.
Question: Is this a known limitation of the parser regarding ephemeral models within the Spark dialect, or is there a specific configuration I should adjust to ensure the lineage persists across these models?
Hi there! First of all, thank you for the amazing project—it's a very helpful tool.
I am experiencing an issue where column-level lineage is lost during the transition between the layers of my ETL pipeline. I am using dbt-spark, and I've noticed that the lineage "gap" occurs specifically between two models that utilize a ephemeral model as intermediate.
Context: I have 3 models, A, B and C, A and C are tables, but B is ephemeral. This models persist the same column id, where B consume id from A, and C consume the id from B.
The issue is: The column lineage is interrupted at Model B. The UI displays the column from Model A in isolation and the column from Model C in isolation, failing to establish the connection between them. Model B does not appear in the column lineage graph at all.
Question: Is this a known limitation of the parser regarding ephemeral models within the Spark dialect, or is there a specific configuration I should adjust to ensure the lineage persists across these models?