fix(ol-dbt-cli): stop registering dbt's __dbt_tmp/__dbt_backup shadow tables

[quazi-h]

What are the relevant tickets?

N/A — found while validating #2403. Partial mitigation of a larger ol-dbt local register issue tracked internally; see Scope below for what this deliberately does not fix.

Description (What does it do?)

ol-dbt local register enumerated Glue tables and filtered on two conditions only:

if table_type.upper() == "ICEBERG" and metadata_location:

dbt's create-temp-then-swap build artifacts satisfy both. <name>__dbt_tmp and <name>__dbt_backup are genuine Iceberg tables and Glue reports them with a real metadata_location, so they were registered as first-class sources alongside real ones.

They are not sources, and they do not survive. dbt deletes them on its next run, so a DuckDB view over one begins failing with HTTP 404 reading its metadata JSON — and the table can disappear between get_tables and CREATE VIEW within a single register run, which is exactly what an error during registration looks like.

Measured against ol_warehouse_production_staging on 2026-09-09, diffing Glue's live table list against _glue_source_registry:

Glue staging tables: 249    registered: 256
in Glue but NOT registered: 0
registered but NO LONGER in Glue: 7
    stg__edxorg__bigquery__mitx_user_email_opt_in__dbt_backup
    stg__edxorg__bigquery__mitx_user_email_opt_in__dbt_tmp
    stg__edxorg__bigquery__mitx_user_info_combo__dbt_backup
    stg__edxorg__bigquery__mitx_user_info_combo__dbt_tmp
    stg__mitxpro__app__postgres__courses_coursetopic__dbt_tmp
    stg__mitxpro__app__postgres__courses_platform__dbt_tmp
    stg__mitxpro__app__postgres__courses_program__dbt_tmp

All seven are shadow tables. Nothing real was missing. That same register run reported ✗ Errors: 6.

One claim I want to label honestly: attributing those six errors specifically to these seven tables is inferred, not proven — the per-table ✗ lines were not captured, and a --dry-run re-check cannot reproduce them because it never issues the CREATE VIEW. The mechanism matches exactly and no non-shadow table was absent from the registry, so it is the strongly likely cause, but the 249/256/7 counts are the measured part.

Implementation notes

• Filters on the table name, anchored to the end: __dbt_(?:tmp|backup)\d*$.
• Checked before the ICEBERG test, so a shadow table lacking Iceberg parameters isn't reported twice (once as an artifact, once as "not Iceberg").
• Numbered variants (__dbt_tmp1, produced when a build is interrupted and retried) are covered. Registered views with __dbt_tmp1 in their name have been observed in this catalog.
• Anchoring matters: a real table whose name merely contains the token — int__dbt_tmp_usage_metrics — must still register, and there's a test for it.

Scope — what this does not fix

This is not the fix for the bigger problem: Glue points the canonical name of a dbt-built Iceberg table at a __dbt_tmp-<uuid> metadata location, because Iceberg RENAME does not move data. Re-measured immediately after a fresh register of three layers on 2026-09-09, 624 of 640 non-raw registered views (97.5%) sit on such a location, and they either 404 or silently return the temp table's accumulated snapshots — duplicated and partially missing rows, measured at 31x duplication on int__mitxonline__proctored_exam_grades.

Fixing that means resolving the table's current snapshot from the Iceberg catalog rather than trusting Glue's metadata_location, or refusing to register such a view at all — a design decision with its own tradeoffs, tracked separately. This PR removes only the junk views and the registration errors they cause. It does not make local row counts or fill rates trustworthy.

How can this be tested?

cd src/ol_dbt_cli
uv run pytest tests/test_local_dev.py -k 'ShadowTable'   # 20 new tests
uv run pytest                                            # full suite: 639 passed
uv run ruff check . && uv run ruff format --check .

To confirm the behaviour end to end against real Glue (needs AWS creds):

uv run ol-dbt local register --database ol_warehouse_production_staging

Shadow tables now print as ⊘ <name> (dbt build artifact, not a real source) and are excluded from the registry. Re-run the count that surfaced the problem and expect no orphans:

-- against ~/.ol-dbt/local.duckdb, after registering
select glue_table from _glue_source_registry
where glue_table like '%__dbt_tmp%' or glue_table like '%__dbt_backup%';

Existing views registered before this change are not retro-actively removed. ol-dbt local cleanup-local drops locally-registered views if you want a clean slate; otherwise the next --force register leaves the seven stale ones in place, since the code no longer enumerates them to update.

Additional Context

Verified by negative control, not just by the tests passing: stubbing _is_dbt_shadow_table to return False fails 10 of the 20 new tests. The parametrised cases use the seven real table names measured above rather than invented ones, so the test suite documents the actual catalog state that motivated the change.

Pre-existing mypy errors. mypy surfaces 6 errors in ol_dbt_cli/lib/sql_parser.py from this package. They are untouched here — the file is byte-identical to main, and mypy --no-incremental reproduces them on main. They're invisible in a warm-cache run, which is why the pre-commit hook passes.

Behaviour change worth a reviewer's eye: anyone who has come to rely on a __dbt_tmp view being present locally will find it gone after the next register. That should be nobody — they 404 unpredictably — but it is a change in what register produces, not only in what it errors on.

🤖 Generated with Claude Code

[github-actions]

🔎 ol-dbt impact — column-level blast radius

✅ No column-level downstream impact detected for the changed models.

_{Posted by ol-dbt impact (annotate-only — does not block merge).}

[Copilot]

🟢 Approval recommended

The focused filtering logic matches the documented scope and is adequately covered by tests.

Pull request overview

Filters transient dbt Iceberg shadow tables from local Glue registration.

Changes:

• Detects __dbt_tmp and __dbt_backup suffixes, including numbered variants.
• Adds focused positive, negative, and integration-style tests.

File summaries

File	Description
src/ol_dbt_cli/ol_dbt_cli/commands/local_dev.py	Excludes dbt shadow tables during Glue enumeration.
src/ol_dbt_cli/tests/test_local_dev.py	Tests shadow-table detection and filtering.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.