Skip to content

fix: SQL export/import round-trip for dropped statements and non-table objects#1264

Merged
datlechin merged 2 commits into
mainfrom
fix/sql-import-export-roundtrip
May 14, 2026
Merged

fix: SQL export/import round-trip for dropped statements and non-table objects#1264
datlechin merged 2 commits into
mainfrom
fix/sql-import-export-roundtrip

Conversation

@datlechin

Copy link
Copy Markdown
Member

Fixes two bugs that broke exporting a database to SQL and importing it back. Both come from #1114.

1. SQL import silently dropped statements

SQLFileParser.parseFile() fed its statement stream through AsyncThrowingStream(bufferingPolicy: .bufferingNewest(8)). That is a dropping buffer, not backpressure. A detached parser task raced through the file while the consumer did one DB round-trip per statement, so whenever the 8-slot buffer filled, the oldest statement was discarded. The import ran a sparse, timing-dependent subset of the file. A re-imported export failed with errors like relation "public.ses_email_events" does not exist because CREATE TABLE was dropped while a later CREATE INDEX on it survived. It also explains wrong "N statements" counts in the import preview.

Rewrote parseFile to use the pull-based AsyncThrowingStream(unfolding:). A ParseSession holds the parse state and produces one statement per next() call, so the consumer drives production: natural backpressure, no detached task racing ahead, bounded memory (~one 64 KB chunk), zero drops. The tokenizer logic is unchanged.

Also fixed a second bug from the same area: SqlFileImportSource.statements() never forwarded dialect to the parser, so PostgreSQL dumps were parsed as .generic and dollar-quoted function bodies were split at semicolons inside the body.

2. SQL export emitted DROP TABLE for non-tables

The export drop phase hardcoded DROP TABLE IF EXISTS for every object. Re-importing a dump that contained a materialized view failed with "daily_revenue" is not a table. HINT: Use DROP MATERIALIZED VIEW. Root cause was upstream: ExportService collapsed every object type to "table" or "view" before the export plugin saw it, so materialized views and foreign tables arrived labeled "table".

ExportService now passes the real object type through. SQLExportPlugin.writeDropPhase picks the matching keyword: DROP VIEW, DROP MATERIALIZED VIEW, DROP FOREIGN TABLE, or DROP TABLE. The data and finalization phases are unchanged.

Known limitation, not addressed here: the create phase still emits CREATE TABLE for views and materialized views (pre-existing behavior), so a materialized view round-trips back as a regular table holding the snapshot data. Emitting real CREATE [MATERIALIZED] VIEW ... AS <query> is a larger change for a follow-up.

Tests

  • slow_consumer_no_dropped_statements - 200 statements with a slow consumer; fails on the old dropping buffer, deterministic on the fix.
  • count_statements_across_chunk_boundary - 4,000 statements spanning multiple read chunks.
  • Fixed a pre-existing data bug in large_multi_row_insert_correctness (literal $0 where \($0) was meant for the id column) - it fails on main whenever it actually runs in the suite.
  • Full SQLFileParserTests suite green across 3 consecutive runs; app builds; swiftlint --strict clean.

The export plugin builds as a separate .tableplugin bundle and the repo has no unit-test harness for plugin targets, so that change is build-verified.

@datlechin datlechin merged commit 12bf8db into main May 14, 2026
2 checks passed
@datlechin datlechin deleted the fix/sql-import-export-roundtrip branch May 14, 2026 07:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant