feat(eap-items): add version column to eap_items_1 - #8424
Merged
Conversation
Adds a UInt64 `version` column defaulting to the current time in milliseconds to eap_items_1_local and eap_items_1_dist. This is a schema-only step toward moving eap_items to a ReplacingMergeTree keyed on `version`. The column needs to exist on the current table before old parts can be attached to the new table.
|
This PR has a migration; here is the generated SQL for -- start migrations
-- forward migration events_analytics_platform : 0064_add_version_column
Local op: ALTER TABLE eap_items_1_local ON CLUSTER 'cluster_one_sh' ADD COLUMN IF NOT EXISTS version UInt64 DEFAULT 0 CODEC (ZSTD(1));
Distributed op: ALTER TABLE eap_items_1_dist ON CLUSTER 'cluster_one_sh' ADD COLUMN IF NOT EXISTS version UInt64 DEFAULT toUnixTimestamp64Milli(now64(3)) CODEC (ZSTD(1));
-- end forward migration events_analytics_platform : 0064_add_version_column
-- backward migration events_analytics_platform : 0064_add_version_column
Distributed op: ALTER TABLE eap_items_1_dist ON CLUSTER 'cluster_one_sh' DROP COLUMN IF EXISTS version;
Local op: ALTER TABLE eap_items_1_local ON CLUSTER 'cluster_one_sh' DROP COLUMN IF EXISTS version;
-- end backward migration events_analytics_platform : 0064_add_version_column |
ADD COLUMN does not rewrite existing parts, so rows predating this migration have no `version` on disk and the local DEFAULT is evaluated at read/merge time. Under ReplacingMergeTree(version) a now64() default on the local table would give those legacy rows a version of whenever the merge ran -- always newer than a real write -- letting a stale row silently beat a genuine update. The local table now defaults to 0, the sentinel for rows written before versioning existed, which deterministically loses to every real write. Inserts go through the distributed table, which evaluates its own DEFAULT on the initiator and ships the materialized value to the shard, so the dist table keeps the now64() default and live writes still get a real millisecond timestamp.
phacops
approved these changes
Sep 1, 2026
This was referenced Sep 3, 2026
onewland
added a commit
that referenced
this pull request
Sep 3, 2026
…#8428) ### Stack 1. ~#8424 — `eap_items_1` (merged)~ 2. #8428 — `eap_items_1_downsample_8_local` 3. #8426 — `eap_items_1_downsample_64_local` 4. #8427 — `eap_items_1_downsample_512_local` 5. #8430 — downsample MVs carry `version` Merge in order; each targets the one above it. --- Adds the `version` column to the `downsample_8` tier. ```sql ALTER TABLE eap_items_1_downsample_8_local ADD COLUMN IF NOT EXISTS version UInt64 DEFAULT 0 CODEC (ZSTD(1)); ``` ### Why Part of getting the `eap_items` schemas aligned with the target `ReplacingMergeTree` (keyed on `version`) so that existing parts can be attached onto the new tables. This mirrors the column added to `eap_items_1` in migration 0064. Schema-only — no engine change, so no dedup behaviour changes yet. ### Why `DEFAULT 0` `ADD COLUMN` does not rewrite existing parts, so rows written before this migration have no `version` on disk and the default is evaluated at read/merge time. That makes a non-deterministic default actively dangerous here: under `ReplacingMergeTree(version)` a `now64()` default gives legacy rows a version of "whenever the merge ran", which always beats a genuine write. Verified on CH 25.3 — a real update was silently discarded in favour of the stale row against a `DEFAULT now64()` destination, and won as expected against a `DEFAULT 0` one. `0` is the sentinel for "written before versioning existed" and deterministically loses to every real write, matching the `received_at` precedent in `snuba/manual_jobs/create_eap_received_at_version_test.py`. ### Why local only Unlike 0064, this does not touch `eap_items_1_downsample_8_dist` (or `_dist_ro`): - We never query `version` — it is internal bookkeeping for the cutover and is absent from the storage YAML, so the query layer cannot generate SQL naming it. - The downsample tiers are fed by materialized views that write straight to `*_local` (`swap_downsample_materialized_views` uses `destination_table_name={prefix}_local`, `target=OperationTarget.LOCAL`), so the distributed table is never an insert target. A Distributed table declaring a subset of its local table's columns behaves normally — verified that reads and `SELECT *` through the narrower dist table work, and that an insert routed through it still lets the local default fire. Only an explicit `SELECT version` through the dist table errors, and nothing does that. ### Follow-up The downsample MVs do not yet carry `version` through from the source, so rows they insert will read `0` rather than the source row's version. Fixing that needs an MV rebuild and is deliberately out of scope here. ### Testing - `tests/migrations/test_runner.py::test_no_schema_differences` - `tests/migrations/test_runner.py::test_run_and_reverse_all`
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
version UInt64column toeap_items_1_localandeap_items_1_dist.Why
We intend to move
eap_itemsto aReplacingMergeTreethat usesversionas its version column. Before we can attach the existing parts onto the new table, the current table's schema has to line up with the target schema — so the column needs to land here first, and subsequently be copied to the downsampled tables.This PR is schema-only. It does not change the engine, so no dedup behavior changes yet.
Why the local and dist defaults differ
This asymmetry is deliberate. Both halves were verified empirically against ClickHouse 25.3.
Local is
DEFAULT 0.ADD COLUMNdoes not rewrite existing parts — the column is simply absent from them (confirmed viasystem.parts_columns), so reads synthesize it from the default expression at read time. With anow64()default that is non-deterministic; the same row returns a different value on consecutive reads:Under
ReplacingMergeTree(version)this is actively harmful. The legacy row's version gets computed at merge time, which is always later than any real write, so the stale row wins. Inserting a genuine update against aDEFAULT now64()destination and runningOPTIMIZE ... FINALsilently discarded the update in favour of the pre-existing row. Against aDEFAULT 0destination the real update won and the untouched legacy row stayed a stable0.So
0is the sentinel for "written before versioning existed", and it deterministically loses to every real write. This matches thereceived_atprecedent insnuba/manual_jobs/create_eap_received_at_version_test.py, which uses a bareUInt64and filtersWHERE received_at != 0.Dist is
DEFAULT toUnixTimestamp64Milli(now64(3)). A Distributed table evaluates its ownDEFAULTexpressions on the initiator node and ships the materialized value to the shard, so for inserts routed through_distthe local default never fires. Measured, with deliberately distinguishable values:DEFAULT 111DEFAULT 999DEFAULT 111DEFAULT 999Note the middle row: declaring the column on
_distwithout a default does not fall through to the local default, it ships an explicit0. So the dist default is what actually assigns a version to live writes, and it is required if we want inserts to carry a real timestamp before the consumer is taught to populate the column.End-to-end, the pair gives the semantics we want:
Scope
Limited to
_local+_dist. The downsample tiers (eap_items_1_downsample_{8,64,512}) are a follow-up, along with the corresponding materialized view rebuild.versionis intentionally not added toeap_items_1_dist_ro. It is internal bookkeeping for theReplacingMergeTreecutover and is absent from the storage YAML, so nothing the query layer generates can reference it. Migration 0056 creates the_dist_rotables before this one runs, so fresh and existing deployments both end up without the column there — there is no environment drift. If we ever want to readversionthrough the read-only routing path it can be added in its own migration.The column is likewise not added to
eap_items.yaml, so it stays invisible to the query layer for now (same asclient_sample_rate).Testing
tests/migrations/test_runner.py::test_no_schema_differencestests/migrations/test_runner.py::test_run_and_reverse_all