feat(migrations): add onDuplicate option to CSV/JSON/Appwrite imports#11910
Merged
Conversation
Exposes two new optional boolean params on the three migration
creation endpoints so CSV / JSON / appwrite-to-appwrite imports can
choose how to handle rows whose IDs already exist at the destination.
Endpoints updated (app/controllers/api/migrations.php):
- POST /v1/migrations/appwrite
- POST /v1/migrations/csv/imports
- POST /v1/migrations/json/imports
Parameter semantics:
- overwrite=true -> destination uses upsertDocuments instead of
createDocuments; existing rows are replaced
with imported values
- skip=true -> destination wraps createDocuments in
skipDuplicates; existing rows are preserved
unchanged, duplicate-id rows silently no-op
- both false -> default; fails fast on DuplicateException
(original behavior, unchanged)
- both true -> overwrite wins (upsert subsumes skip)
Both params are stored in the migration Document's options array
(matches the existing pattern for destination behavior config like
path, size, delimiter, bucketId, etc.) and read back in the worker's
processDestination() to instantiate DestinationAppwrite with the
new constructor params.
Feature-branch note: depends on utopia-php/migration#feat/skip-duplicates
(DestinationAppwrite constructor params) which in turn depends on
utopia-php/database#852 (skipDuplicates scope guard). composer.json is
temporarily pinned to dev-feat/skip-duplicates and
dev-csv-import-upsert-v2 respectively; both must be reset to proper
release versions once the upstream PRs merge.
Three new test methods in MigrationsBase, following the existing testCreateCSVImport setup pattern: - testCreateCSVImportSkipDuplicates Seeds documents.csv, mutates one row, re-imports with skip=true. Asserts the mutated row keeps its mutated value (not overwritten by the CSV's original value) and the row count stays at 100. - testCreateCSVImportOverwrite Seeds documents.csv, mutates one row, re-imports with overwrite=true. Asserts the mutated row is restored to the CSV's original value (proving upsertDocuments actually replaced the row) and the row count stays at 100. - testCreateCSVImportDefaultFailsOnDuplicate Regression guard: re-imports documents.csv with no flags. Asserts the migration goes to status=failed with errors populated, proving the default duplicate-throws behavior is preserved. All three share a prepareCsvImportFixture() helper that sets up database + table (name, age columns) + bucket + documents.csv upload. Returns the known first-row id + original name/age so tests can mutate and assert on a predictable row. Reuses the existing documents.csv fixture (100 rows with \$id as the first column). No new fixture files needed.
🔄 PHP-Retry SummaryFlaky tests detected across commits: Commit
|
| Test | Retries | Total Time | Details |
|---|---|---|---|
WebhooksCustomServerTest::testDeleteDeployment |
1 | 26ms | Logs |
WebhooksCustomServerTest::testDeleteFunction |
1 | 12ms | Logs |
WebhooksCustomServerTest::testCreateCollection |
1 | 11ms | Logs |
WebhooksCustomServerTest::testCreateAttributes |
1 | 21ms | Logs |
WebhooksCustomServerTest::testCreateDocument |
1 | 17ms | Logs |
WebhooksCustomServerTest::testUpdateDocument |
1 | 123ms | Logs |
WebhooksCustomServerTest::testDeleteDocument |
1 | 13ms | Logs |
WebhooksCustomServerTest::testCreateTable |
1 | 21ms | Logs |
WebhooksCustomServerTest::testCreateColumns |
1 | 30ms | Logs |
WebhooksCustomServerTest::testCreateRow |
1 | 29ms | Logs |
WebhooksCustomServerTest::testUpdateRow |
1 | 17ms | Logs |
WebhooksCustomServerTest::testDeleteRow |
1 | 23ms | Logs |
WebhooksCustomServerTest::testCreateStorageBucket |
1 | 15ms | Logs |
WebhooksCustomServerTest::testUpdateStorageBucket |
1 | 14ms | Logs |
WebhooksCustomServerTest::testCreateBucketFile |
1 | 12ms | Logs |
WebhooksCustomServerTest::testUpdateBucketFile |
1 | 6ms | Logs |
WebhooksCustomServerTest::testDeleteBucketFile |
1 | 6ms | Logs |
WebhooksCustomServerTest::testDeleteStorageBucket |
1 | 15ms | Logs |
WebhooksCustomServerTest::testCreateTeam |
1 | 7ms | Logs |
WebhooksCustomServerTest::testUpdateTeam |
1 | 12ms | Logs |
WebhooksCustomServerTest::testUpdateTeamPrefs |
1 | 16ms | Logs |
WebhooksCustomServerTest::testDeleteTeam |
1 | 6ms | Logs |
WebhooksCustomServerTest::testCreateTeamMembership |
1 | 16ms | Logs |
WebhooksCustomServerTest::testDeleteTeamMembership |
1 | 11ms | Logs |
WebhooksCustomServerTest::testWebhookAutoDisable |
1 | 31ms | Logs |
UsageTest::testPrepareDatabaseStatsTablesAPI |
1 | 32.30s | Logs |
Commit e63f9fd - 4 flaky tests
| Test | Retries | Total Time | Details |
|---|---|---|---|
UsageTest::testFunctionsStats |
1 | 10.15s | Logs |
UsageTest::testPrepareSitesStats |
1 | 6ms | Logs |
UsageTest::testEmbeddingsTextUsageDoesNotBreakProjectUsage |
1 | 4ms | Logs |
WebhooksCustomServerTest::testExecutions |
1 | 2.88s | Logs |
Commit f9c5f41 - 5 flaky tests
| Test | Retries | Total Time | Details |
|---|---|---|---|
UsageTest::testFunctionsStats |
1 | 10.17s | Logs |
UsageTest::testPrepareSitesStats |
1 | 7ms | Logs |
UsageTest::testEmbeddingsTextUsageDoesNotBreakProjectUsage |
1 | 4ms | Logs |
DatabasesStringTypesTest::testCreateTable |
1 | 242.15s | Logs |
TablesDBTransactionsCustomServerTest::testMixedSingleOperations |
1 | 240.53s | Logs |
Commit 2e9841c - 28 flaky tests
| Test | Retries | Total Time | Details |
|---|---|---|---|
UsageTest::testFunctionsStats |
1 | 10.20s | Logs |
UsageTest::testPrepareSitesStats |
1 | 6ms | Logs |
UsageTest::testEmbeddingsTextUsageDoesNotBreakProjectUsage |
1 | 5ms | Logs |
WebhooksCustomServerTest::testDeleteDeployment |
1 | 14ms | Logs |
WebhooksCustomServerTest::testDeleteFunction |
1 | 10ms | Logs |
WebhooksCustomServerTest::testCreateCollection |
1 | 13ms | Logs |
WebhooksCustomServerTest::testCreateAttributes |
1 | 45ms | Logs |
WebhooksCustomServerTest::testCreateDocument |
1 | 17ms | Logs |
WebhooksCustomServerTest::testUpdateDocument |
1 | 23ms | Logs |
WebhooksCustomServerTest::testDeleteDocument |
1 | 119ms | Logs |
WebhooksCustomServerTest::testCreateTable |
1 | 11ms | Logs |
WebhooksCustomServerTest::testCreateColumns |
1 | 9ms | Logs |
WebhooksCustomServerTest::testCreateRow |
1 | 12ms | Logs |
WebhooksCustomServerTest::testUpdateRow |
1 | 13ms | Logs |
WebhooksCustomServerTest::testDeleteRow |
1 | 17ms | Logs |
WebhooksCustomServerTest::testCreateStorageBucket |
1 | 7ms | Logs |
WebhooksCustomServerTest::testUpdateStorageBucket |
1 | 12ms | Logs |
WebhooksCustomServerTest::testCreateBucketFile |
1 | 11ms | Logs |
WebhooksCustomServerTest::testUpdateBucketFile |
1 | 7ms | Logs |
WebhooksCustomServerTest::testDeleteBucketFile |
1 | 8ms | Logs |
WebhooksCustomServerTest::testDeleteStorageBucket |
1 | 17ms | Logs |
WebhooksCustomServerTest::testCreateTeam |
1 | 6ms | Logs |
WebhooksCustomServerTest::testUpdateTeam |
1 | 11ms | Logs |
WebhooksCustomServerTest::testUpdateTeamPrefs |
1 | 13ms | Logs |
WebhooksCustomServerTest::testDeleteTeam |
1 | 7ms | Logs |
WebhooksCustomServerTest::testCreateTeamMembership |
1 | 20ms | Logs |
WebhooksCustomServerTest::testDeleteTeamMembership |
1 | 11ms | Logs |
WebhooksCustomServerTest::testWebhookAutoDisable |
1 | 32ms | Logs |
Commit e621701 - 5 flaky tests
| Test | Retries | Total Time | Details |
|---|---|---|---|
VectorsDBConsoleClientTest::testGetCollectionLogs |
1 | 6ms | Logs |
DatabasesConsoleClientTest::testGetCollectionLogs |
1 | 6ms | Logs |
UsageTest::testFunctionsStats |
1 | 10.17s | Logs |
UsageTest::testPrepareSitesStats |
1 | 7ms | Logs |
UsageTest::testEmbeddingsTextUsageDoesNotBreakProjectUsage |
1 | 5ms | Logs |
Note: Flaky test results are tracked for the last 5 commits
✨ Benchmark resultsComparing 1.9.x (before) to feat/skip-duplicates (after). Before
After
Delta
Top API waits
|
This comment has been minimized.
This comment has been minimized.
… database to 5.3.22
4 tasks
premtsd-code
changed the title
Contributor
Greptile SummaryThis PR adds an
Confidence Score: 4/5The Appwrite-to-Appwrite migration path works correctly, but the CSV and JSON import paths silently ignore the new The worker passes src/Appwrite/Platform/Workers/Migrations.php — Important Files Changed
Reviews (45): Last reviewed commit: "Merge branch '1.9.x' into feat/skip-dupl..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
premtsd-code
force-pushed
the
feat/skip-duplicates
branch
from
8a00770 to
f3c2502
Compare
This comment has been minimized.
This comment has been minimized.
premtsd-code
force-pushed
the
feat/skip-duplicates
branch
from
7057189 to
d0603c4
Compare
…lerance utopia-php/migration's DestinationAppwrite now handles schema tolerance on re-migration (PR #171 on feat/skip-duplicates): it pre-checks destination `_metadata` for each database / table / column / index and tolerates in Skip/Upsert mode. Re-runs no longer produce schema-level errors, so the E2E tests can drop the status-tolerant workaround and assert strict 'completed' outcomes. Changes: - composer.json: pin utopia-php/migration to dev-feat/skip-duplicates (aliased to 1.9.99 for stability resolution). Will be replaced with a fixed 1.10.0 tag once the migration PR lands. - testAppwriteMigrationRowsOnDuplicate: replace the tolerant runMigrationAssertingRowSuccess helper with performMigrationSync on the Skip and Upsert re-runs. Asserts 'completed' status on every run, destination row content matches the expected value per mode (Mutated preserved on Skip, Original restored on Upsert). Helper method removed. - testAppwriteMigrationReRunIsIdempotent (new): seeds two rows on source, runs the migration three times back-to-back (fresh, Skip re-run, Upsert re-run) against unchanged source data, asserts strict 'completed' on every run and row content is stable across all three. Exercises the schema-tolerance path end-to-end: every database/table/column on destination already exists with a matching spec, so DestinationAppwrite's pre-check returns Tolerate for every resource.
Branch is iterating on utopia-php/migration's re-migration tolerance. Other test matrices (unit, general, abuse, screenshots, benchmark, and every other e2e service) add ~30+ minutes to CI without exercising code this PR touches. Restrict the matrix to the Migrations service and skip the unrelated test jobs until the migration work is ready to merge. All jobs marked with 'TEMP:' comments + 'if: false' — revert to the full matrix before merging to main. Static analysis (lint, phpstan, composer audit, specs, locale, security) still runs on every PR push.
This comment has been minimized.
This comment has been minimized.
Picks up the PR #171 refactor + unit tests: - resolveSchemaAction decision point consolidation - deleteAttributeCompletely primitive (two-way cleanup in one place) - Hardened sourceIsNewer against MySQL zero-date sentinel - 14 unit tests locking the decision matrix
This comment has been minimized.
This comment has been minimized.
Picks up the UpdateInPlace branch — database/table metadata drift is now reconciled on Upsert-newer (renames, enable toggles, table permissions / documentSecurity) via updateDocument, without touching child rows.
… dest drift
Two new E2E tests exercising the schema-tolerance UpdateInPlace path
added in utopia-php/migration's DestinationAppwrite.
testAppwriteMigrationUpsertUpdatesContainerMetadata (positive):
- Fresh migration copies source database + table + column + row to dest.
- Mutates source database name (PUT /databases/:id) and table
name/permissions/rowSecurity/enabled (PUT /tablesdb/:db/tables/:id).
- One-second sleep before mutation ensures source's $updatedAt is
strictly greater than dest's at second granularity (strtotime
comparison).
- Upsert re-migration asserts:
- 'completed' status.
- dest database name matches source's new name.
- dest table name / enabled / rowSecurity match source's new values.
- child row's 'name' attribute is untouched — UpdateInPlace only
rewrites container metadata, not rows.
testAppwriteMigrationSkipPreservesContainerDrift (negative):
- Fresh migration, then mutate BOTH dest (simulating ops tightening
permissions post-migration) and source (divergence).
- Skip re-migration asserts dest kept its tightened values — Skip's
strict "don't touch" contract protects dev→prod cutover workflows
from accidentally wiping ops-side drift on schema re-sync.
Both tests use performMigrationSync for strict 'completed' assertions.
Runtime ~18s combined. Existing testAppwriteMigrationRowsOnDuplicate
and testAppwriteMigrationReRunIsIdempotent regression-tested locally.
Two coverage gaps closed: - testAppwriteMigrationUpsertOneWayRelationshipDropAndRecreate exercises the path that updateRelationshipInPlace gates off: one-way + onDelete change → returns false → falls through to DropAndRecreate via deleteRelationship. Coverage was lost when testAppwriteMigrationUpsertUpdatesRelationshipOnDeleteInPlace was converted to two-way to actually hit the in-place path. - testAppwriteMigrationUpsertAttributeRecreateDropsAndRecreates pins the createdAt-different leaf path: source drops + recreates the attribute (createdAt advances), re-migration must DropAndRecreate on dest and re-flow the row data through the row pass. Companion to testAppwriteMigrationUpsertUpdatesAttributeInPlace which covers the same-createdAt + newer-updatedAt path. Migration package already at 09c1b21 (the maintainability commit) from the previous lock bump — no further composer.lock change needed.
testAppwriteMigrationUpsertSameSpecRecreateTolerates exercises the new spec-match guard added in utopia-php/migration c8d1789. Source drops + recreates a column with the EXACT same spec as before; createdAt advances but specs match → action is forced to Tolerate. Asserts dest column's $createdAt stays at first-migration value (proving Tolerate, not DropAndRecreate). Row pass under Upsert still propagates source's new row value. Companion to testAppwriteMigrationUpsertAttributeRecreateDropsAndRecreates which exercises the spec-DIFFERS path: same precondition (drop + recreate), different outcome (DropAndRecreate vs Tolerate) gated on spec equality. composer.lock: utopia-php/migration 24fd23b -> c8d1789 (spec-match guard).
After dropping createdAt from resolveSchemaAction, source-side recreate no longer routes through DropAndRecreate via the outer decision. The inner fallthrough still drops+recreates when the spec diff is a non-SDK change, so this test now toggles 'array' (a non-SDK field) on recreate to actually exercise the drop+recreate path it pins. Also clarifies the two-way recreate test's docblock — with createdAt gone and identical spec on recreate, it exercises spec-match + pair-key dedup (both tolerate paths) rather than parent-side drop. End-state assertions unchanged.
… before count/validator)
Maintainer review on utopia-php/migration#171 renamed OnDuplicate::Upsert -> OnDuplicate::Overwrite (value 'upsert' -> 'overwrite') to align with Appwrite terms (skip / overwrite / fail). Applying the cross-repo ripple here: - app/controllers/api/migrations.php: 3 endpoint param descriptions updated ('upsert' -> 'overwrite' in the help text). The validator still uses OnDuplicate::values() so it auto-picks up the new value. - tests/e2e/Services/Migrations/MigrationsBase.php: all 'onDuplicate' => 'upsert' -> 'overwrite'; method names testAppwriteMigrationUpsert* -> testAppwriteMigrationOverwrite*; comments / assertion messages / local var names switched. - Left untouched: utopia's upsertDocuments operation, transaction TransactionState 'upsert' action, Operation validator — those refer to the database-level upsert primitive, not the OnDuplicate enum. composer.lock: utopia-php/migration 7d71505 -> b8ae7bc.
# Conflicts: # app/controllers/api/migrations.php # composer.lock
1.9.x reorganized app/controllers/api/migrations.php into the
Platform/Modules/Migrations structure but dropped the onDuplicate
param. After the merge, every e2e migration test that passed
'onDuplicate' => 'overwrite' would 400 since the param wasn't in the
allowlist anymore.
Restoring it on the three endpoints that take row-level conflict
behavior: Appwrite/Create, CSV/Imports/Create, JSON/Imports/Create.
Each:
- Imports OnDuplicate + WhiteList.
- Adds optional ->param('onDuplicate', OnDuplicate::Fail->value,
new WhiteList(OnDuplicate::values()), …).
- Threads $onDuplicate through the action signature.
- Stores it on the migration document's 'options' attribute so
Workers/Migrations.php can pick it up via
OnDuplicate::tryFrom($options['onDuplicate'] ?? '')
?? OnDuplicate::Fail.
Worker code already reads options['onDuplicate'] (unchanged) — no
edits needed there.
server-ce 1.9.x's tablesdb POST /rows tightened input validation: the modular Documents/Create.php rejects `data => []` with a 400 "missing data" because TablesDB's Rows/Create.php inherits the strict default of getSupportForEmptyDocument() = false (only DocumentsDB overrides it to true). The test was relying on the older permissive behavior to seed an empty parent row before the relationship cascade links it. Add a non-relationship `label` string column on the parents table and populate it with `data => ['label' => 'p1']` so the POST passes the empty-data guard. The test's actual assertion target — partner-side pair-key dedup on DropAndRecreate — is unchanged. Cascade fixes: testAppwriteMigrationOverwriteAttributeRecreate and testAppwriteMigrationOverwriteSameSpecRecreate were failing in the retry pass because TwoWayRecreate's bail at L1616 left source/dest state uncleaned. Once TwoWayRecreate completes, those tests see a clean project again. Caught in CI run 25419479164 / job 74562934987 on the MongoDB (dedicated) Migrations matrix.
# Conflicts: # composer.lock
abnegate
approved these changes
May 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a single
onDuplicatestring param to the three migration creation endpoints so CSV / JSON / Appwrite-to-Appwrite imports can choose how to react to any resource (database, table, column, index, row) that already exists at the destination.Each accepts optional
onDuplicate: "fail" | "skip" | "overwrite"(default"fail").Parameter semantics
"fail"(default)status=failed."skip""overwrite"The string is validated by
WhiteList(OnDuplicate::values())at the REST boundary — same convention as other enum-style params (priority,encryption,period,grant_type). Allowed values are owned byUtopia\Migration\Destinations\OnDuplicate.Scope
onDuplicatedrives all resource handling in the destination — not just rows. The migration package's destination resolves aSchemaAction(Create / Skip / Overwrite) per resource based ononDuplicate:overwrite, preserved onskipoverwrite, kept onskipoverwriteOn
overwrite, a destination-newer guard ($updatedAt > source) tolerates the local copy so a fresh edit on dest isn't clobbered by a stale source.The param is persisted on the migration document under
options.onDuplicate; the worker reads it back viaOnDuplicate::tryFrom($options['onDuplicate'] ?? '') ?? OnDuplicate::Failand threads it into theDestinationAppwriteconstructor.Changes
src/Appwrite/Platform/Modules/Migrations/Http/Migrations/Appwrite/Create.php— addsonDuplicateparam + persists inoptionssrc/Appwrite/Platform/Modules/Migrations/Http/Migrations/CSV/Imports/Create.php— same wiringsrc/Appwrite/Platform/Modules/Migrations/Http/Migrations/JSON/Imports/Create.php— same wiringsrc/Appwrite/Platform/Workers/Migrations.php—processDestination()readsoptions.onDuplicateand constructsDestinationAppwritewith the resolved enumcomposer.json—utopia-php/migrationconstraint set to1.*(lock pins to1.10.0);utopia-php/databaseis5.*(lock pins to5.4.2)tests/e2e/Services/Migrations/MigrationsBase.php— 19 new E2E test methods + fixtures + helpersE2E test coverage
Container metadata (skip + overwrite, schema-level)
testAppwriteMigrationOverwriteUpdatesContainerMetadata— db/table name+enabled drift restoredtestAppwriteMigrationSkipPreservesContainerDrift— drift preservedOrphan columns (skip + overwrite)
testAppwriteMigrationOverwriteDropsOrphanColumntestAppwriteMigrationSkipKeepsOrphanColumnAttribute drift (skip + overwrite)
testAppwriteMigrationOverwriteUpdatesAttributeInPlace— SDK-mutable field changes propagate via UpdateInPlacetestAppwriteMigrationSkipPreservesAttributeDrift— destination drift preservedRelationship paths (overwrite-driven)
testAppwriteMigrationOverwriteUpdatesRelationshipOnDeleteInPlace— onDelete updated in place on both sidestestAppwriteMigrationOverwriteOneWayRelationshipDropAndRecreate— structural change forces drop+recreatetestAppwriteMigrationOverwriteTwoWayRecreateSkipsPartnerSide— partner-side pair-key dedup pins to existing datatestAppwriteMigrationOverwriteAttributeRecreateDropsAndRecreates— non-SDK column change forces drop+recreatetestAppwriteMigrationOverwriteSameSpecRecreateTolerates— spec-match guard avoids no-op churn after source-side recreateRows
testAppwriteMigrationRowsOnDuplicate— fail / skip / overwrite for rowstestAppwriteMigrationReRunIsIdempotent— clean re-run with onDuplicate=skipCSV imports
testCreateCSVImportSkipDuplicatestestCreateCSVImportOverwritetestCreateCSVImportDefaultFailsOnDuplicateJSON imports
testCreateJSONImportSkipDuplicatestestCreateJSONImportOverwritetestCreateJSONImportDefaultFailsOnDuplicate19 new test methods covering every code path in the destination.
Dependencies
Both upstream deps are tagged releases — no dev-branch pins:
utopia-php/database@5.4.2— provides theskipDuplicates()scope guard (utopia-php/database#852)utopia-php/migration@1.10.0— provides theOnDuplicateenum, full schema-tolerance refactor, partner-side dedup, spec-match guard (utopia-php/migration#171)Test plan
composer validatecleanRelated