Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion CHANGES/1641.misc

This file was deleted.

1 change: 0 additions & 1 deletion CHANGES/2346.misc

This file was deleted.

3 changes: 0 additions & 3 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -185,9 +185,6 @@ RUN patch -p1 -d /usr/local/lib/pulp/lib/python${PYTHON_VERSION}/site-packages <
COPY images/assets/patches/0079-Log-Maven-path-index-fallback-reasons.patch /tmp/
RUN patch -p1 -d /usr/local/lib/pulp/lib/python${PYTHON_VERSION}/site-packages < /tmp/0079-Log-Maven-path-index-fallback-reasons.patch

COPY images/assets/patches/0080-Optimize-read-only-content-queries.patch /tmp/
RUN patch -p1 -d /usr/local/lib/pulp/lib/python${PYTHON_VERSION}/site-packages < /tmp/0080-Optimize-read-only-content-queries.patch
Comment on lines -188 to -189

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Content requests lose query optimizations

When the image is built and migration 0025 is applied before the replacement is available, the image stops applying patch 0080 and migration 0025 drops the index used by optimized content queries; the installed directory-serving patch also loses its relative_path index, so content and directory requests use slower query paths.

Keep applying patch 0080 and retain the index until the replacement serving implementation is available.

Also at pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py:21-22.

Prompt for AI agents
In `Dockerfile` at lines 188-189:

**Content requests lose query optimizations**

When the image is built and migration 0025 is applied before the replacement is available, the image stops applying patch 0080 and migration 0025 drops the index used by optimized content queries; the installed directory-serving patch also loses its `relative_path` index, so content and directory requests use slower query paths.

Keep applying patch 0080 and retain the index until the replacement serving implementation is available.

Also at `pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py:21-22`.


COPY images/assets/patches/0081-Fix-task-cancellation-during-graceful-shutdown.patch /tmp/
RUN patch -p1 -d /usr/local/lib/pulp/lib/python${PYTHON_VERSION}/site-packages < /tmp/0081-Fix-task-cancellation-during-graceful-shutdown.patch

Expand Down
101 changes: 0 additions & 101 deletions images/assets/patches/0080-Optimize-read-only-content-queries.patch

This file was deleted.

62 changes: 1 addition & 61 deletions pulp_service/pulp_service/app/experiments.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,13 @@
import logging
import random
from time import perf_counter
from uuid import UUID

from django.conf import settings
from django.db import connections
from django.db.models import Exists, OuterRef, Q
from django.db.models import Q
from django_guid import get_guid

logger = logging.getLogger("pulp.experiment")
DIRECTORY_EXPERIMENT = "PULP-2505"
CONTENT_QUERY_EXPERIMENT = "PULP-2346"
CONTENT_QUERY_P_CANDIDATE = 0.5


def run_experiment(exp_id, control, candidate, *, p_candidate, context):
Expand Down Expand Up @@ -133,59 +129,3 @@ def directory_membership_dates(version, content_to_names, sources, *, serving_mo
p_candidate=probability,
context=context,
)


def _content_query_marker(variant, repository_id):
"""Return the PULP-2346 marker for a variant and repository."""
repository = UUID(str(repository_id))
return f"/* pexp={CONTENT_QUERY_EXPERIMENT} v={variant} */ /* repo={repository} */"


def mark_lazy_queryset(queryset, marker):
"""Embed an experiment marker in the SQL, so it also survives later evaluation (e.g. ``.values()``)."""
# The marker is built from constants and a UUID-validated repository id, never from user input.
return queryset.extra(where=[f"1 = 1 {marker}"]) # noqa: S610


def content_query_experiment(query, version, queryset, *, filters, limit=None):
"""Run one content app lookup as PULP-2346 and return ``(rows, queryset used)``.

Variant A is stock pulpcore (``content__in=version.content``). Variant B checks repository
membership per matching row with a correlated EXISTS. Both return the same rows. Always on:
half of the calls run variant B.
"""

def build(variant):
if variant == "A":
selected = queryset.filter(content__in=version.content, **filters)
else:
membership = version._content_relationships().filter(content_id=OuterRef("content_id"))
selected = queryset.filter(Exists(membership), **filters)
return mark_lazy_queryset(selected, _content_query_marker(variant, version.repository_id))

used = {}

def run(variant):
def evaluate():
used["queryset"] = build(variant)
return list(used["queryset"] if limit is None else used["queryset"][:limit])

return evaluate

rows = run_experiment(
CONTENT_QUERY_EXPERIMENT,
run("A"),
run("B"),
p_candidate=CONTENT_QUERY_P_CANDIDATE,
context={"query": query, "repository_id": str(version.repository_id)},
)
return rows, used["queryset"]


def single_or_raise(model, rows):
"""Return the only row, raising what ``QuerySet.get()`` would for none or several."""
if not rows:
raise model.DoesNotExist(f"{model.__name__} matching query does not exist.")
if len(rows) > 1:
raise model.MultipleObjectsReturned(f"get() returned more than one {model.__name__}.")
return rows[0]
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
from django.db import migrations

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Cleanup release note is missing

When this non-trivial change is merged, no new CHANGES/ fragment records the experiment and index cleanup, so towncrier release notes omit this non-trivial change.

Add a towncrier fragment in CHANGES/ describing the cleanup.

Prompt for AI agents
In `pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py` at line 1:

**Cleanup release note is missing**

When this non-trivial change is merged, no new `CHANGES/` fragment records the experiment and index cleanup, so towncrier release notes omit this non-trivial change.

Add a towncrier fragment in `CHANGES/` describing the cleanup.

From CLAUDE.md L76:

Uses **towncrier**. For any non-trivial change, create a file in `CHANGES/` named `{issue_number}.{category}` where category is one of: `feature`, `bugfix`, `doc`, `removal`, `deprecation`, `misc`.


_INDEX_NAME = "core_ca_relpath_pattern_idx"
_INDEX_MARKER = "pulp-service:0024:core_ca_relpath_pattern_idx"
_DROP_INDEX_SQL = f'DROP INDEX CONCURRENTLY IF EXISTS "{_INDEX_NAME}"'


def remove_experiment_index(apps, schema_editor): # noqa: ARG001
"""Remove only the index owned by the temporary pulp-service migration."""
with schema_editor.connection.cursor() as cursor:
cursor.execute(
"""
SELECT obj_description(indexrel.oid, 'pg_class')
FROM pg_class AS indexrel
JOIN pg_namespace AS namespace ON namespace.oid = indexrel.relnamespace
WHERE indexrel.relname = %s AND namespace.nspname = current_schema()
""",
[_INDEX_NAME],
)
row = cursor.fetchone()
if row and row[0] == _INDEX_MARKER:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Unmarked index survives cleanup

When migration 0024 encounters or leaves a valid index without its ownership comment, remove_experiment_index drops the index only when its comment matches _INDEX_MARKER, while migration 0024 accepts a valid existing index without adding the comment, so migration 0025 leaves the service-owned index in the database.

Ensure retries mark an existing service-owned index before migration 0025 checks its comment.

Also at pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py:22-23.

Prompt for AI agents
In `pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py` at line 21:

**Unmarked index survives cleanup**

When migration 0024 encounters or leaves a valid index without its ownership comment, `remove_experiment_index` drops the index only when its comment matches `_INDEX_MARKER`, while migration 0024 accepts a valid existing index without adding the comment, so migration 0025 leaves the service-owned index in the database.

Ensure retries mark an existing service-owned index before migration 0025 checks its comment.

Also at `pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py:22-23`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Cleanup leaves the index behind

When migration 0024 creates the index but does not successfully add its ownership comment, remove_experiment_index skips the index when its ownership marker is absent, so migration 0025 completes while the service-owned index remains in the database.

Handle the existing index by name even when its ownership comment is missing, so the cleanup removes it.

Prompt for AI agents
In `pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py` at line 21:

**Cleanup leaves the index behind**

When migration 0024 creates the index but does not successfully add its ownership comment, `remove_experiment_index` skips the index when its ownership marker is absent, so migration 0025 completes while the service-owned index remains in the database.

Handle the existing index by name even when its ownership comment is missing, so the cleanup removes it.

cursor.execute(_DROP_INDEX_SQL)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security (python.sqlalchemy.security.sqlalchemy-execute-raw-query): Avoiding SQL string concatenation: untrusted input concatenated with raw SQL query can result in SQL Injection. In order to execute raw query safely, prepared statement should be used. SQLAlchemy provides TextualSQL to easily used prepared statement with named parameters. For complex SQL composition, use SQL Expression Language or Schema Definition Language. In most cases, SQLAlchemy ORM will be a better option.

Source: opengrep

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security (python.sqlalchemy.security.sqlalchemy-execute-raw-query): Avoiding SQL string concatenation: untrusted input concatenated with raw SQL query can result in SQL Injection. In order to execute raw query safely, prepared statement should be used. SQLAlchemy provides TextualSQL to easily used prepared statement with named parameters. For complex SQL composition, use SQL Expression Language or Schema Definition Language. In most cases, SQLAlchemy ORM will be a better option.

Source: opengrep

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Content lookups lose index performance

When the service migration runs before pulpcore’s replacement query and index are active, remove_experiment_index drops the index while the retained content-query patch still filters relative_path by prefix, so PostgreSQL scans core_contentartifact and content requests become slower on large repositories.

Keep the index until pulpcore’s replacement query and index are active.

Also at pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py:23.

Prompt for AI agents
In `pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py` at line 22:

**Content lookups lose index performance**

When the service migration runs before pulpcore’s replacement query and index are active, `remove_experiment_index` drops the index while the retained content-query patch still filters `relative_path` by prefix, so PostgreSQL scans `core_contentartifact` and content requests become slower on large repositories.

Keep the index until pulpcore’s replacement query and index are active.

Also at `pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py:23`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security (python.sqlalchemy.security.sqlalchemy-execute-raw-query): Avoiding SQL string concatenation: untrusted input concatenated with raw SQL query can result in SQL Injection. In order to execute raw query safely, prepared statement should be used. SQLAlchemy provides TextualSQL to easily used prepared statement with named parameters. For complex SQL composition, use SQL Expression Language or Schema Definition Language. In most cases, SQLAlchemy ORM will be a better option.

Source: opengrep

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security (python.sqlalchemy.security.sqlalchemy-execute-raw-query): Avoiding SQL string concatenation: untrusted input concatenated with raw SQL query can result in SQL Injection. In order to execute raw query safely, prepared statement should be used. SQLAlchemy provides TextualSQL to easily used prepared statement with named parameters. For complex SQL composition, use SQL Expression Language or Schema Definition Language. In most cases, SQLAlchemy ORM will be a better option.

Source: opengrep


Comment on lines +11 to +23

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Replacement index gets dropped

When the pulpcore replacement migration operates on the same index concurrently with this migration, remove_experiment_index checks the ownership comment and later drops by name, with no lock spanning those operations. A concurrent pulpcore migration that replaces or adopts the same-named index between them can have its replacement dropped, leaving the replacement implementation without its index.

Serialize the migrations or make the drop conditional on ownership at the time the index is dropped.

Prompt for AI agents
In `pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py` at lines 11-23:

**Replacement index gets dropped**

When the pulpcore replacement migration operates on the same index concurrently with this migration, `remove_experiment_index` checks the ownership comment and later drops by name, with no lock spanning those operations. A concurrent pulpcore migration that replaces or adopts the same-named index between them can have its replacement dropped, leaving the replacement implementation without its index.

Serialize the migrations or make the drop conditional on ownership at the time the index is dropped.


class Migration(migrations.Migration):
"""Remove the temporary index before pulpcore owns the replacement."""

atomic = False # DROP INDEX CONCURRENTLY cannot run in a transaction.

dependencies = [
("service", "0024_contentartifact_relative_path_pattern_index"),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Critical · Migration commands report conflicting branches

When django loads the service migration graph for migration generation or deployment, this migration and 0025_delete_domainorg remain separate leaves because 0026_andgatecontentguard follows only the latter, so Django reports conflicting migrations and blocks migration generation or deployment.

Join the migration branches by making the subsequent migration depend on this migration as well as 0025_delete_domainorg.

Prompt for AI agents
In `pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py` at line 31:

**Migration commands report conflicting branches**

When django loads the service migration graph for migration generation or deployment, this migration and `0025_delete_domainorg` remain separate leaves because `0026_andgatecontentguard` follows only the latter, so Django reports conflicting migrations and blocks migration generation or deployment.

Join the migration branches by making the subsequent migration depend on this migration as well as `0025_delete_domainorg`.

]

operations = [
migrations.RunPython(remove_experiment_index, migrations.RunPython.noop),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Rollback leaves index missing

When an operator reverses migration 0025 without also reversing migration 0024, RunPython.noop does not restore the index dropped by remove_experiment_index, so the prior service version resumes its optimized content queries without the index they expect.

Provide a reverse operation that restores the index, or otherwise ensure it is present when migration 0025 is reversed.

Prompt for AI agents
In `pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py` at line 35:

**Rollback leaves index missing**

When an operator reverses migration 0025 without also reversing migration 0024, `RunPython.noop` does not restore the index dropped by `remove_experiment_index`, so the prior service version resumes its optimized content queries without the index they expect.

Provide a reverse operation that restores the index, or otherwise ensure it is present when migration 0025 is reversed.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Rollback leaves the index absent

When an operator rolls back across migration 0025 to a release that uses the index, migrations.RunPython.noop does not restore the index, so migration 0024 remains applied while the restored application runs its optimized content queries without the index.

Provide a reverse operation that recreates the index when migration 0025 is reversed.

Prompt for AI agents
In `pulp_service/pulp_service/app/migrations/0025_remove_experiment_relative_path_index.py` at line 35:

**Rollback leaves the index absent**

When an operator rolls back across migration 0025 to a release that uses the index, `migrations.RunPython.noop` does not restore the index, so migration 0024 remains applied while the restored application runs its optimized content queries without the index.

Provide a reverse operation that recreates the index when migration 0025 is reversed.

]
157 changes: 0 additions & 157 deletions pulp_service/pulp_service/tests/unit/test_content_query_experiment.py

This file was deleted.

Loading