Postgres table connector gets NullPointerException when scan.incremental.snapshot.enabled true

[meng-miao-wu]

Hi team,

I attempt to use the postgres table connector to listen multiple tables of the same db, settings as follow

      'database-name' = '${DB_NAME}',
      'schema-name' = 'public',
      'table-name' = 'table1,public.table2',

I am getting the following error:

2025-03-18 15:41:27,708 ERROR org.apache.flink.cdc.connectors.base.source.reader.external.IncrementalSourceScanFetcher [] - Execute snapshot read task for snapshot split SnapshotSplit{tableId=[database_name].[schema_name].[table2_name], splitId='[database_name].[schema_name].[table2_name]:0', splitKeyType=[`id` BIGINT NOT NULL], splitStart=null, splitEnd=[9780], highWatermark=null} fail
io.debezium.DebeziumException: java.lang.NullPointerException
	at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:85) ~[blob_p-ad0e0411efb43f0201b8a2ddbef309e911be2d18-105bc3fd0be6883d753134636ce8991a:?]
	at org.apache.flink.cdc.connectors.postgres.source.fetch.PostgresScanFetchTask.executeDataSnapshot(PostgresScanFetchTask.java:112) ~[blob_p-ad0e0411efb43f0201b8a2ddbef309e911be2d18-105bc3fd0be6883d753134636ce8991a:?]
	at org.apache.flink.cdc.connectors.base.source.reader.external.AbstractScanFetchTask.execute(AbstractScanFetchTask.java:71) ~[blob_p-ad0e0411efb43f0201b8a2ddbef309e911be2d18-105bc3fd0be6883d753134636ce8991a:?]
	at org.apache.flink.cdc.connectors.postgres.source.fetch.PostgresScanFetchTask.execute(PostgresScanFetchTask.java:86) ~[blob_p-ad0e0411efb43f0201b8a2ddbef309e911be2d18-105bc3fd0be6883d753134636ce8991a:?]
	at org.apache.flink.cdc.connectors.base.source.reader.external.IncrementalSourceScanFetcher.lambda$submitTask$1(IncrementalSourceScanFetcher.java:99) ~[blob_p-ad0e0411efb43f0201b8a2ddbef309e911be2d18-105bc3fd0be6883d753134636ce8991a:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.lang.NullPointerException
	at java.util.Objects.requireNonNull(Unknown Source) ~[?:?]
	at org.apache.flink.cdc.connectors.postgres.source.fetch.PostgresScanFetchTask$PostgresSnapshotSplitReadTask.createDataEvents(PostgresScanFetchTask.java:268) ~[blob_p-ad0e0411efb43f0201b8a2ddbef309e911be2d18-105bc3fd0be6883d753134636ce8991a:?]
	at org.apache.flink.cdc.connectors.postgres.source.fetch.PostgresScanFetchTask$PostgresSnapshotSplitReadTask.doExecute(PostgresScanFetchTask.java:255) ~[blob_p-ad0e0411efb43f0201b8a2ddbef309e911be2d18-105bc3fd0be6883d753134636ce8991a:?]
	at org.apache.flink.cdc.connectors.postgres.source.fetch.PostgresScanFetchTask$PostgresSnapshotSplitReadTask.doExecute(PostgresScanFetchTask.java:211) ~[blob_p-ad0e0411efb43f0201b8a2ddbef309e911be2d18-105bc3fd0be6883d753134636ce8991a:?]
	at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:76) ~[blob_p-ad0e0411efb43f0201b8a2ddbef309e911be2d18-105bc3fd0be6883d753134636ce8991a:?]
	... 7 more

OFC, when scan.incremental.snapshot.enabled is flipped to false, it works.

I've also attempted to set

  'debezium.table.include.list' = 'public.[table1],public.table2'

no luck

🙏 any guidance is appreciated

[yuxiqian]

Hi @meng-miao-wu, maybe passing an RegEx pattern to match the table name part (like table(1|2)) and omit the schema name prefix would work?

[meng-miao-wu]

Hey @yuxiqian, thanks for the prompt reply

I've tried the proposed way with regex, the tables can be properly identified, but the error persists.

[meng-miao-wu]

Closing the loop:

'database-name' = '${DB_NAME}',
      'schema-name' = 'public',
      'table-name' = '(table1|table2)',
      'debezium.table.include.list'='public.table1,public.table2,'
      

^ worked with above