Repository navigation
Remove interval content query experiment #1701
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
This file was deleted.
This file was deleted.
This file was deleted.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| from django.db import migrations | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Medium · Cleanup release note is missing When this non-trivial change is merged, no new Add a towncrier fragment in Prompt for AI agentsIn `pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py` at line 1:
**Cleanup release note is missing**
When this non-trivial change is merged, no new `CHANGES/` fragment records the experiment and index cleanup, so towncrier release notes omit this non-trivial change.
Add a towncrier fragment in `CHANGES/` describing the cleanup.From Uses **towncrier**. For any non-trivial change, create a file in `CHANGES/` named `{issue_number}.{category}` where category is one of: `feature`, `bugfix`, `doc`, `removal`, `deprecation`, `misc`. |
||
|
|
||
| _INDEX_NAME = "core_ca_relpath_pattern_idx" | ||
| _INDEX_MARKER = "pulp-service:0024:core_ca_relpath_pattern_idx" | ||
| _DROP_INDEX_SQL = f'DROP INDEX CONCURRENTLY IF EXISTS "{_INDEX_NAME}"' | ||
|
|
||
|
|
||
| def remove_experiment_index(apps, schema_editor): # noqa: ARG001 | ||
| """Remove only the index owned by the temporary pulp-service migration.""" | ||
| with schema_editor.connection.cursor() as cursor: | ||
| cursor.execute( | ||
| """ | ||
| SELECT obj_description(indexrel.oid, 'pg_class') | ||
| FROM pg_class AS indexrel | ||
| JOIN pg_namespace AS namespace ON namespace.oid = indexrel.relnamespace | ||
| WHERE indexrel.relname = %s AND namespace.nspname = current_schema() | ||
| """, | ||
| [_INDEX_NAME], | ||
| ) | ||
| row = cursor.fetchone() | ||
| if row and row[0] == _INDEX_MARKER: | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Medium · Unmarked index survives cleanup When migration 0024 encounters or leaves a valid index without its ownership comment, Ensure retries mark an existing service-owned index before migration 0025 checks its comment. Also at Prompt for AI agentsIn `pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py` at line 21:
**Unmarked index survives cleanup**
When migration 0024 encounters or leaves a valid index without its ownership comment, `remove_experiment_index` drops the index only when its comment matches `_INDEX_MARKER`, while migration 0024 accepts a valid existing index without adding the comment, so migration 0025 leaves the service-owned index in the database.
Ensure retries mark an existing service-owned index before migration 0025 checks its comment.
Also at `pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py:22-23`.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Medium · Cleanup leaves the index behind When migration 0024 creates the index but does not successfully add its ownership comment, Handle the existing index by name even when its ownership comment is missing, so the cleanup removes it. Prompt for AI agentsIn `pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py` at line 21:
**Cleanup leaves the index behind**
When migration 0024 creates the index but does not successfully add its ownership comment, `remove_experiment_index` skips the index when its ownership marker is absent, so migration 0025 completes while the service-owned index remains in the database.
Handle the existing index by name even when its ownership comment is missing, so the cleanup removes it. |
||
| cursor.execute(_DROP_INDEX_SQL) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. security (python.sqlalchemy.security.sqlalchemy-execute-raw-query): Avoiding SQL string concatenation: untrusted input concatenated with raw SQL query can result in SQL Injection. In order to execute raw query safely, prepared statement should be used. SQLAlchemy provides TextualSQL to easily used prepared statement with named parameters. For complex SQL composition, use SQL Expression Language or Schema Definition Language. In most cases, SQLAlchemy ORM will be a better option. Source: opengrep
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. security (python.sqlalchemy.security.sqlalchemy-execute-raw-query): Avoiding SQL string concatenation: untrusted input concatenated with raw SQL query can result in SQL Injection. In order to execute raw query safely, prepared statement should be used. SQLAlchemy provides TextualSQL to easily used prepared statement with named parameters. For complex SQL composition, use SQL Expression Language or Schema Definition Language. In most cases, SQLAlchemy ORM will be a better option. Source: opengrep
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Medium · Content lookups lose index performance When the service migration runs before pulpcore’s replacement query and index are active, Keep the index until pulpcore’s replacement query and index are active. Also at Prompt for AI agentsIn `pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py` at line 22:
**Content lookups lose index performance**
When the service migration runs before pulpcore’s replacement query and index are active, `remove_experiment_index` drops the index while the retained content-query patch still filters `relative_path` by prefix, so PostgreSQL scans `core_contentartifact` and content requests become slower on large repositories.
Keep the index until pulpcore’s replacement query and index are active.
Also at `pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py:23`.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. security (python.sqlalchemy.security.sqlalchemy-execute-raw-query): Avoiding SQL string concatenation: untrusted input concatenated with raw SQL query can result in SQL Injection. In order to execute raw query safely, prepared statement should be used. SQLAlchemy provides TextualSQL to easily used prepared statement with named parameters. For complex SQL composition, use SQL Expression Language or Schema Definition Language. In most cases, SQLAlchemy ORM will be a better option. Source: opengrep
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. security (python.sqlalchemy.security.sqlalchemy-execute-raw-query): Avoiding SQL string concatenation: untrusted input concatenated with raw SQL query can result in SQL Injection. In order to execute raw query safely, prepared statement should be used. SQLAlchemy provides TextualSQL to easily used prepared statement with named parameters. For complex SQL composition, use SQL Expression Language or Schema Definition Language. In most cases, SQLAlchemy ORM will be a better option. Source: opengrep |
||
|
|
||
|
Comment on lines
+11
to
+23
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Medium · Replacement index gets dropped When the pulpcore replacement migration operates on the same index concurrently with this migration, Serialize the migrations or make the drop conditional on ownership at the time the index is dropped. Prompt for AI agentsIn `pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py` at lines 11-23:
**Replacement index gets dropped**
When the pulpcore replacement migration operates on the same index concurrently with this migration, `remove_experiment_index` checks the ownership comment and later drops by name, with no lock spanning those operations. A concurrent pulpcore migration that replaces or adopts the same-named index between them can have its replacement dropped, leaving the replacement implementation without its index.
Serialize the migrations or make the drop conditional on ownership at the time the index is dropped. |
||
|
|
||
| class Migration(migrations.Migration): | ||
| """Remove the temporary index before pulpcore owns the replacement.""" | ||
|
|
||
| atomic = False # DROP INDEX CONCURRENTLY cannot run in a transaction. | ||
|
|
||
| dependencies = [ | ||
| ("service", "0024_contentartifact_relative_path_pattern_index"), | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔴 Critical · Migration commands report conflicting branches When django loads the service migration graph for migration generation or deployment, this migration and Join the migration branches by making the subsequent migration depend on this migration as well as Prompt for AI agentsIn `pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py` at line 31:
**Migration commands report conflicting branches**
When django loads the service migration graph for migration generation or deployment, this migration and `0025_delete_domainorg` remain separate leaves because `0026_andgatecontentguard` follows only the latter, so Django reports conflicting migrations and blocks migration generation or deployment.
Join the migration branches by making the subsequent migration depend on this migration as well as `0025_delete_domainorg`. |
||
| ] | ||
|
|
||
| operations = [ | ||
| migrations.RunPython(remove_experiment_index, migrations.RunPython.noop), | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Medium · Rollback leaves index missing When an operator reverses migration 0025 without also reversing migration 0024, Provide a reverse operation that restores the index, or otherwise ensure it is present when migration 0025 is reversed. Prompt for AI agentsIn `pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py` at line 35:
**Rollback leaves index missing**
When an operator reverses migration 0025 without also reversing migration 0024, `RunPython.noop` does not restore the index dropped by `remove_experiment_index`, so the prior service version resumes its optimized content queries without the index they expect.
Provide a reverse operation that restores the index, or otherwise ensure it is present when migration 0025 is reversed.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Medium · Rollback leaves the index absent When an operator rolls back across migration 0025 to a release that uses the index, Provide a reverse operation that recreates the index when migration 0025 is reversed. Prompt for AI agentsIn `pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py` at line 35:
**Rollback leaves the index absent**
When an operator rolls back across migration 0025 to a release that uses the index, `migrations.RunPython.noop` does not restore the index, so migration 0024 remains applied while the restored application runs its optimized content queries without the index.
Provide a reverse operation that recreates the index when migration 0025 is reversed. |
||
| ] | ||
This file was deleted.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 Medium · Content requests lose query optimizations
When the image is built and migration 0025 is applied before the replacement is available, the image stops applying patch 0080 and migration 0025 drops the index used by optimized content queries; the installed directory-serving patch also loses its
relative_pathindex, so content and directory requests use slower query paths.Keep applying patch 0080 and retain the index until the replacement serving implementation is available.
Also at
pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py:21-22.Prompt for AI agents