Repository navigation
replica-identity warns when a captured Postgres table has no replica identity - #29
Merged
Merged
Conversation
… a column Postgres needs no space between a constraint keyword and its parenthesis, and Mattermost's migrations use that form throughout: UNIQUE(name, deleteat), PRIMARY KEY(id, channelid). The word splitter keeps a parenthesised group glued to the word before it, so the reader saw a column named "UNIQUE(name, deleteat)" (an info line for a column that does not exist) and a word "KEY(id, channelid)" that hid the primary key. The same held for CHECK( on a column and for ALTER TABLE ... ADD UNIQUE( / ADD CHECK(, which added a column. Keywords in the Postgres reader are now compared up to a parenthesis. The helper stays in this reader: the shared ddl.HasPrefixFold is also used by the ClickHouse reader, where a word like Kafka(...) is compared whole. Measured on all 280 of Mattermost v10.11.0's Postgres migration files: nine made-up columns before (channels, emoji, PropertyGroups, roles, schemes, sharedchannelremotes, teams, usergroups twice), none after, and no real column lost. RefuseRadar unchanged at 162 info. Not done here: Postgres folds unquoted identifiers to lower case, and the reader keeps CREATE TABLE PropertyGroups as written. Its own change.
…identity A table on REPLICA IDENTITY NOTHING, or on DEFAULT with no primary key, has no replica identity. Both consequences are documented. Postgres (logical-replication-publication): such a table "cannot support UPDATE or DELETE operations when included in a publication replicating these actions", which is every pgoutput connector, so the application's own writes fail. Debezium (PostgreSQL connector, Replica identity): with DEFAULT and no primary key "the connector does not emit UPDATE or DELETE events for that table", and with NOTHING its events carry no previous values. The finding says which one applies, from plugin.name (decoderbufs when unset, the documented default). The connector's replica.identity.autoset.values overrides the database on the tables it captures, so it is read and wins; when it decides, the finding points at the connector file. Two pairs that disagree about one table, or a pair that cannot be read, leave the identity unknown and the table unjudged: the documentation does not say which pair wins. A warning, not an error: a table that only takes inserts loses nothing, and the fix says how to record that with cdclint:ignore replica-identity. To keep it from judging keys it cannot see, the Postgres reader now reads ALTER TABLE ... ADD [CONSTRAINT x] PRIMARY KEY (a key added later), and marks a table's key unknown when a DO block alters it or the key is PRIMARY KEY USING INDEX. Mattermost adds reactions' key in a DO block; running DO bodies is its own change. USING INDEX identities are not judged: whether the index still exists is not in the files. Two older corpus entries had been reduced without the keys their sources have; they now carry them, and their expectations do not change. golang-migrate-down-files gets Mattermost's real teammembers key and reactions' DO block. flatten-bare-columns gets REPLICA IDENTITY FULL: the discussion's sample message has a whole before row on a keyless table, which only FULL produces. Rejected: judging what the sink reads from the before image (Flatten on the envelope reading before.<column>). It is the README's longer promise for this rule, needs the transform model to say which before fields a sink reads, and is its own change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #28. This branch includes #28's commit (glued
PRIMARY KEY(id, ...)), which the rule needs to read keys correctly. Merge #28 first; this diff then shrinks to its own commit.What changes
A new warning,
replica-identity, for a captured Postgres table with no replica identity:REPLICA IDENTITY NOTHING, orDEFAULTwith no primary key.Both consequences are documented, and the finding says which one applies, based on
plugin.name:pgoutput. Postgres, logical-replication-publication: "Tables with a replica identity defined as NOTHING, DEFAULT without a primary key, or USING INDEX with a dropped index, cannot support UPDATE or DELETE operations when included in a publication replicating these actions." The application's own writes fail.decoderbufs(Debezium's documented default). Debezium, PostgreSQL connector, Replica identity: with DEFAULT and no primary key "the connector does not emit UPDATE or DELETE events for that table". With NOTHING, its events carry no previous values.Other behaviour:
replica.identity.autoset.valuesoverrides the database for captured tables, so cdclint reads it. When the override decides, the finding points at the connector file. If two pairs disagree about one table, or a pair can't be read, the table is not judged, because the documentation doesn't say which wins.-- cdclint:ignore replica-identity: append-onlyon theCREATE TABLEline, and I checked that this marker acknowledges the finding.USING INDEX, because whether the index still exists isn't in the files.Reader changes, so the rule never judges a key it cannot see
ALTER TABLE ... ADD [CONSTRAINT x] PRIMARY KEY (...)now sets the key.DOblock alters, or one withPRIMARY KEY USING INDEX, is markedKeyUnknown, and the rule stays quiet for it. Mattermost addsreactions' key inside aDOblock. Actually runningDObodies is its own change.Corpus first
There are six new entries. I wrote their expectations by hand, and the three that expect a finding failed before the rule existed:
no-primary-keynothing-pgoutputautoset-nothingautoset-fullkey-added-laterkey-in-do-block(Mattermost's real000016)Two older entries had been cut down without the keys their sources have. They now carry them, and their
expected.txtis unchanged:golang-migrate-down-files: Mattermost's realteammembersprimary key, andreactions'DOblock.flatten-bare-columns:REPLICA IDENTITY FULL. The source discussion's sample message has a wholebeforerow on a keyless table, which only FULL produces.Verified
gofmt -l .is clean;go vet,go buildandgo test ./...pass. There are new unit tests for the override (including the conflicting and unreadable cases) and for the reader's key handling.0 error(s), 0 warning(s), 162 info, so no new warning, which matters because its PR gate fails on warnings.reactionsis covered by theDO-block guard.Rejected
Judging the before image a sink reads (Flatten on the envelope reading
before.<column>). That is the longer promise in the README for this rule. It needs the transform model to know whichbeforefields a sink reads, so it is its own change. The README row now describes what this rule actually does.After merge
The README status line ("Ten rules") changes at the release that ships this, as with earlier rules.