fix(bigquery): widen auto-detected partition window to avoid silent empty samples

[zerafachris]

Why

Fixes #33084

BigQuery TIME_UNIT and INGESTION_TIME partitions were auto-detected with a partitionInterval=1 (1 day for DAY granularity, 1 hour for HOUR granularity). Any table whose most-recent partition is older than that window is silently sampled as an empty result — the profiler reports success, so tables that were never actually examined are indistinguishable from tables that were examined and found clean.

Common sources of this silent failure:

• GA4 export tables (~2-day ingestion lag)
• Weekly or monthly aggregate tables
• Any source with a slow or batched load cadence

What changed

ingestion/src/metadata/sampler/partition.py

• TIME_UNIT auto-detection: partitionInterval=1 → 24 (HOUR) or 3 (DAY/MONTH/YEAR)
• INGESTION_TIME auto-detection: same change
• Per-table profiler config overrides are fully preserved — any explicitly set partitionInterval in tableProfilerConfig.partitioning continues to take precedence via the existing get_partition_details early-return path

ingestion/src/metadata/sampler/sqlalchemy/sampler.py

• _get_asset_row_count(): emit logger.warning when a partitioned-table COUNT(*) returns 0 rows, naming the table, current window, and the docs link to override it. This makes any remaining narrow-window case visible in run logs rather than silent.

ingestion/tests/unit/test_partition.py

• Updated existing INGESTION_TIME assertions from 1 → 24 (HOUR) and 3 (DAY)
• Added test_bigquery_time_unit_partition_default_intervals covering both DAY (3-day window) and HOUR (24-hour window) for TIME_UNIT partitions

Test evidence

The OpenMetadata ingestion package requires generated schema files not present in a shallow clone, which prevented running pytest in isolation. Logic was verified via a standalone script that mocks the pydantic models and calls the same branching logic directly — all 5 cases pass:

PASS TIME_UNIT DAY: interval=3 unit=DAY
PASS TIME_UNIT HOUR: interval=24 unit=HOUR
PASS INGESTION_TIME DAY: interval=3 col=_PARTITIONDATE
PASS INGESTION_TIME HOUR: interval=24 col=_PARTITIONTIME
PASS INTEGER_RANGE: interval=None rangeStart=1

---

Prepared with AI assistance (Claude Code, Anthropic), reviewed for correctness before submission.

[github-actions]

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

• Linked issue BigQuery auto-detected partition window is 1 day, so tables not updated daily sample zero rows and classify as clean #33084 is not added to the Shipping project.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[gitar-bot]

Code Review ✅ Approved 1 resolved / 1 findings

Widens auto-detected partition windows for BigQuery TIME_UNIT and INGESTION_TIME partitions to prevent silent empty samples on tables with slow ingestion cadence. Scales partitionInterval from 1 to 24 (HOUR) or 3 (DAY/MONTH/YEAR), adds warning logging when partitioned-table row counts return zero, and preserves explicit tableProfilerConfig overrides. The 3-day window for MONTH/YEAR partitions may still silently empty on slower aggregate tables and is addressed in a follow-up commit.

✅ 1 resolved

✅ Edge Case: 3-day window still silently empty for MONTH/YEAR partitions

📄 ingestion/src/metadata/sampler/partition.py:124 📄 ingestion/src/metadata/sampler/partition.py:136
For TIME_UNIT/INGESTION_TIME partitions with MONTH or YEAR granularity, is_hourly is False so the config gets partitionIntervalUnit=DAY, partitionInterval=3, which build_partition_predicate turns into partition_column >= CURRENT_DATE - INTERVAL 3 DAY. A monthly/yearly table whose freshest data is older than 3 days is still sampled as empty — the exact silent-empty failure this PR targets. The comment claims 3 days covers "daily/monthly/yearly", but 3 days is far narrower than a month or year; consider scaling the interval to the granularity (e.g. ~35 days for MONTH, ~370 for YEAR). The new 0-row warning does at least surface the case now.

Options

Display: compact → Counting what did not apply, without listing it.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source}