You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, in Dataform, the name parameter in a type: "declaration" block serves a dual, tightly-coupled purpose: it is both the logical identifier used in ${ref()} across the project and the physical table name in the data warehouse.
This creates significant architectural friction for Analytics Engineering teams when dealing with external data sources.
flowchart TB
%% Estilos visuais
classDef coupled fill:#fce4e4,stroke:#cc0000,stroke-width:2px,color:#000,rx:5,ry:5;
classDef decoupled fill:#e2f0d9,stroke:#385723,stroke-width:2px,color:#000,rx:5,ry:5;
classDef ideal fill:#deebf7,stroke:#1f497d,stroke-width:2px,color:#000,rx:5,ry:5;
subgraph Current ["❌ 1. Current Scenario (Tight Coupling Problem)"]
direction LR
DB1[("📦 External Source\n(Fivetran, Airbyte, Kondado...)\n'table_src_f896ae...'")] --> Decl1["📄 Declaration\nname: 'table_src_f896ae...'"]:::coupled
Decl1 --> Down1["🔄 Downstream Model\nref('table_src_f896ae...')"]:::coupled
end
subgraph Proposed ["✅ 2. Proposed Scenario (Semantic Abstraction)"]
direction LR
DB2[("📦 External Source\n(Fivetran, Airbyte, Kondado...)\n'table_src_f896ae...'")] --> Decl2["✨ Declaration\nname: 'raw_meta_ads'\nphysical_name: 'table_src_f896ae...'"]:::ideal
Decl2 --> Down2["🔄 Downstream Model\nref('raw_meta_ads')"]:::decoupled
end
%% O SEGREDO: Link invisível forçando a ordem (Current vem antes de Proposed)
Current ~~~ Proposed
Loading
The Problem
When ingesting data from external third-party tools or APIs, tables often land with non-semantic, volatile, or auto-generated names (e.g., table_src_f896ae219530b91f56b3cd19).
Because Dataform forces the declaration name to match the physical table name exactly, downstream layers are forced to inherit this unreadable nomenclature: SELECT * FROM ${ref("table_src_f896ae219530b91f56b3cd19")}
If the physical table name changes at the source, developers must update the declaration's nameand find/update every downstream ${ref()} that depends on it. This breaks the concept of a stable data contract at the raw layer.
Current Workarounds & Limitations
To create a semantic abstraction layer, teams currently have to choose between two sub-optimal paths:
The "View Wrapper" (Redundant maintenance)
Create a declaration using the raw physical name.
Create a secondary type: "view" (e.g., raw_meta_ads) that does a SELECT * FROM ${ref("ugly_physical_name")}.
Drawback: If the physical table changes, developers still have to update two files (the declaration block and the view wrapper's reference).
The "Hardcoded View" (Losing DAG and Assertions)
Abandon type: "declaration" entirely.
Create a type: "view" and hardcode the physical path directly in the SQL: FROM project.dataset.ugly_physical_name.
Drawback: This solves the routing issue but removes the source node from the dependency graph (DAG) and removes the ability to run Dataform assertions directly against the source table.
Proposed Solution
Allow the separation of the logical reference from the physical table name by introducing a new parameter within the declaration config block, such as physical_name (or conversely, an alias).
Example of the ideal state:
config{type: "declaration",database: "raw-project",schema: "external_source_ab32adf",name: "raw_meta_ads",// The logical, semantic name used in ${ref("raw_meta_ads")}physical_name: "table_src_f896ae219530b91f56b3cd19",// The actual, physical table name in the warehousedescription: "Meta Ads data contract"}
Benefits:
True Abstraction: The declaration acts as a proper interface/contract.
Maintainability: If the external physical table name changes, we only need to update the physical_name parameter in one single file. All downstream models using ${ref("raw_meta_ads")} remain perfectly intact.
Cleaner Codebase: Eliminates the need for redundant "pass-through" views just to rename external tables.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Description
Currently, in Dataform, the
nameparameter in atype: "declaration"block serves a dual, tightly-coupled purpose: it is both the logical identifier used in${ref()}across the project and the physical table name in the data warehouse.This creates significant architectural friction for Analytics Engineering teams when dealing with external data sources.
flowchart TB %% Estilos visuais classDef coupled fill:#fce4e4,stroke:#cc0000,stroke-width:2px,color:#000,rx:5,ry:5; classDef decoupled fill:#e2f0d9,stroke:#385723,stroke-width:2px,color:#000,rx:5,ry:5; classDef ideal fill:#deebf7,stroke:#1f497d,stroke-width:2px,color:#000,rx:5,ry:5; subgraph Current ["❌ 1. Current Scenario (Tight Coupling Problem)"] direction LR DB1[("📦 External Source\n(Fivetran, Airbyte, Kondado...)\n'table_src_f896ae...'")] --> Decl1["📄 Declaration\nname: 'table_src_f896ae...'"]:::coupled Decl1 --> Down1["🔄 Downstream Model\nref('table_src_f896ae...')"]:::coupled end subgraph Proposed ["✅ 2. Proposed Scenario (Semantic Abstraction)"] direction LR DB2[("📦 External Source\n(Fivetran, Airbyte, Kondado...)\n'table_src_f896ae...'")] --> Decl2["✨ Declaration\nname: 'raw_meta_ads'\nphysical_name: 'table_src_f896ae...'"]:::ideal Decl2 --> Down2["🔄 Downstream Model\nref('raw_meta_ads')"]:::decoupled end %% O SEGREDO: Link invisível forçando a ordem (Current vem antes de Proposed) Current ~~~ ProposedThe Problem
When ingesting data from external third-party tools or APIs, tables often land with non-semantic, volatile, or auto-generated names (e.g.,
table_src_f896ae219530b91f56b3cd19).Because Dataform forces the declaration
nameto match the physical table name exactly, downstream layers are forced to inherit this unreadable nomenclature:SELECT * FROM ${ref("table_src_f896ae219530b91f56b3cd19")}If the physical table name changes at the source, developers must update the declaration's
nameand find/update every downstream${ref()}that depends on it. This breaks the concept of a stable data contract at therawlayer.Current Workarounds & Limitations
To create a semantic abstraction layer, teams currently have to choose between two sub-optimal paths:
type: "view"(e.g.,raw_meta_ads) that does aSELECT * FROM ${ref("ugly_physical_name")}.type: "declaration"entirely.type: "view"and hardcode the physical path directly in the SQL:FROM project.dataset.ugly_physical_name.Proposed Solution
Allow the separation of the logical reference from the physical table name by introducing a new parameter within the
declarationconfig block, such asphysical_name(or conversely, analias).Example of the ideal state:
Benefits:
physical_nameparameter in one single file. All downstream models using${ref("raw_meta_ads")}remain perfectly intact.All reactions