Repository navigation
perf(migrations): Keep pending deletions unrendered until their DELETE runs - #130863
Closed
sentry-junior[bot] wants to merge 3 commits into
Closed
sentry-junior[bot] wants to merge 3 commits into
sentry-junior[bot] wants to merge 3 commits into
Conversation
…E runs SafeDeleteModel and SafeRemoveField with MOVE_TO_PENDING looked the model up through state.apps, which rendered every model each time. Replaying the history to build the migration state did that for each of the past pending deletions, which cost ~20s on every `sentry django migrate`, even with nothing to apply. Store the pending ModelState and field as they are, and render them only when a DELETE step needs the model. Models whose DELETE ran stay available so an M2M `through` deleted before its owner still resolves. This also replaces the CheckedMigration.mutate_state workaround that dropped the rendered apps. Co-Authored-By: Ming Chen <ming.chen@sentry.io>
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 2c14d75. Configure here.
…endering it A pending FK or M2M field whose target or through model has since gone pending or been deleted couldn't resolve when its DELETE ran.
…reated live model When a model is moved to pending and then re-created under the same name, the pending ModelState now replaces the live one in the throwaway render, so the DELETE drops the pending model's table instead of the live model's. Co-Authored-By: Ming Chen <ming.chen@sentry.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

sentry django migratespends ~20s rebuilding migration state before it does anything, even with nothing to apply.SafeDeleteModel/SafeRemoveFieldwithMOVE_TO_PENDINGlooked the model up throughstate.apps, andSentryProjectState.remove_model/remove_fielddid the same to stash a rendered model or field. Since #127458 drops the rendered registry at each migration boundary, every migration with a pending deletion rendered every model from scratch. Replaying the history does that once per past pending deletion, andmigratereplays it up to three times (pre-migrate state, executor state, autodetector check).SentryProjectStatenow keeps the pendingModelStateand the unbound field as they are.get_pending_deletion_model/get_pending_deletion_fieldrender them only when a DELETE step (or its backwards) needs them, in a throwaway clone of the state, so the replay never renders. Thestate_forwardssafety checks read theModelStatefields directly. Because nothing renders during replay any more, theCheckedMigration.mutate_stateworkaround from #127458 is removed.Rendering later means a pending model's relations have to resolve against a later state.
investigations/0008and0017DELETE an M2Mthroughmodel before the model that declares it, so models whose DELETE already ran are kept indeleted_modelsand added back when a pending deletion points at them. All 61 DELETE operations in the current history resolve to the same tables, columns, and through tables as before; I compared them againstmaster.tools/migrations/drop_pending_deletions.pyuses the newrender_pending_deletions(), so it renders everything once instead of reading stored model classes.Timings on a full dev environment, two runs each. "2 pending" is
sentry.1254(a realSafeRemoveFieldDELETE ofgrouprelease.new_release_id) plussentry.1255:sentry django migrateUnder cProfile, with nothing to apply: 76 full
StateAppsrenders (21,228ModelState.rendercalls, 29.6s) before, and 1 render (336 calls, 0.8s) after.MigrationLoader.project_state()alone drops from 9.8s to 0.2s.Changed test expectation: moving a model to pending twice now raises Django's usual
LookupError: App '…' doesn't have a 'TestTable' model.instead ofNo installed app with label …, which came from rendering a fresh registry.The full migration suite passes with
--create-db, which applies every historical DELETE against a real database. So dotests/sentry/new_migrations, the safe-migrations integration tests, andtests/sentry/runner/commands/test_migrations.py. Unapplyingsentry.1254adds the column back.via Ming Chen.
--
View Session [Sentry]