Skip to content

Commit bc2c3d8

Browse files
committed
Add a paginated repository endpoint that lists Python packages.
Clients can walk every package in a repository version. Each row is one package name, and each stored version is one entry. Closes #1358. Assisted-By: Cursor
1 parent 71f42e8 commit bc2c3d8

15 files changed

Lines changed: 807 additions & 25 deletions

File tree

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ __pycache__/
88
# Distribution / packaging
99
.Python
1010
env/
11+
.venv/
1112
build/
1213
develop-eggs/
1314
dist/
@@ -61,3 +62,6 @@ target/
6162

6263
# PyCharm
6364
.idea
65+
66+
# VS Code / Cursor
67+
.vscode/

‎CHANGES/1358.feature‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Added a repository endpoint that lists Python packages with limit/offset pagination.

‎docs/index.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ The REST API documentation for `pulp_python` is available [here](site:pulp_pytho
1212

1313
- [Create local mirrors of PyPI](site:pulp_python/docs/user/guides/sync/) that you have full control over
1414
- [Upload your own Python packages](site:pulp_python/docs/user/guides/upload/)
15+
- [Browse the package catalog](site:pulp_python/docs/user/guides/catalog/) over the REST API
1516
- [Perform pip install](site:pulp_python/docs/user/guides/host/) from your Pulp Python repositories
1617
- Download packages on-demand to reduce disk usage
1718
- Every operation creates a restorable snapshot with Versioned Repositories
@@ -34,5 +35,4 @@ Users may also find pulpcore’s conceptual docs useful.
3435
This documentation falls into two main categories:
3536

3637
1. `How-to Guides` shows the **major features** of the Python plugin, with links to reference docs.
37-
2. The [REST API Docs](site:pulp_python/restapi/) are automatically generated and provide more detailed information for each
38-
minor feature, including all fields and options.
38+
2. The [REST API Docs](site:pulp_python/restapi/) are automatically generated and provide more detailed information for each minor feature, including all fields and options.

‎docs/user/guides/_SUMMARY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
* [Set up your own PyPI](pypi.md)
22
* [Sync from Remote Repositories](sync.md)
33
* [Upload and Manage Content](upload.md)
4+
* [Browse the package catalog](catalog.md)
45
* [Host Python Content](host.md)
56
* [Vulnerability Report](vulnerability_report.md)
67
* [Attestation Hosting](attestation.md)

‎docs/user/guides/catalog.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# Browse the package catalog
2+
3+
The package list returns **one row per package name**. It defaults to the
4+
repository's latest complete version. Pass `repository_version` (HREF or PRN)
5+
to read an older snapshot. `{pulp_id}` is the repository UUID.
6+
7+
## List packages
8+
9+
```bash
10+
http GET "${BASE_ADDR}/pulp/api/v3/repositories/python/python/${REPO_PK}/packages/?limit=10"
11+
```
12+
13+
`count` is the number of distinct packages, not files. Each row is one package
14+
name. `summary`, `description`, `author`, `author_email`, and `license` come
15+
from the newest version. `versions` has one entry per stored version string,
16+
newest first, with that version's `license_expression`. A wheel and an sdist
17+
of the same version are one entry. A rebuild such as `1.0.0+test.1` stays its
18+
own entry because the stored version differs.
19+
20+
```json
21+
{
22+
"count": 1,
23+
"next": null,
24+
"previous": null,
25+
"results": [
26+
{
27+
"name": "shelf-reader",
28+
"name_normalized": "shelf-reader",
29+
"summary": "A small example package",
30+
"description": "",
31+
"author": "",
32+
"author_email": "",
33+
"license": "",
34+
"versions": [
35+
{"version": "0.1", "license_expression": ""}
36+
]
37+
}
38+
]
39+
}
40+
```
41+
42+
Results are ordered by `name_normalized`. Within a package, versions are newest
43+
first. Page with `limit` and `offset` to read the whole catalog. Pass
44+
`repository_version` when the export must stay on one snapshot.

‎pulp_python/app/catalog.py‎

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
"""Helpers for the repository package list."""
2+
3+
from collections import defaultdict
4+
from urllib.parse import urlparse
5+
6+
from django.core.exceptions import FieldError
7+
from django.core.exceptions import ValidationError as DjangoValidationError
8+
from django.db.models import Case, IntegerField, Q, Value, When
9+
from django.urls import Resolver404, resolve
10+
from packaging.version import InvalidVersion, Version
11+
12+
from pulpcore.plugin.models import Repository, RepositoryContent, RepositoryVersion
13+
from pulpcore.plugin.util import resolve_prn
14+
15+
from pulp_python.app.models import PythonPackageContent
16+
17+
18+
def _version_sort_key(version):
19+
"""PEP 440 sort key. Use with ``reverse=True`` for newest first.
20+
21+
Invalid versions sort after all valid ones when ``reverse=True``.
22+
"""
23+
if not version:
24+
return (-1, "")
25+
try:
26+
return (0, Version(version))
27+
except InvalidVersion:
28+
return (-1, str(version))
29+
30+
31+
def _in_version_bounds_q(repository_version, prefix=""):
32+
"""Version-added/removed bounds for rows present in ``repository_version``.
33+
34+
``prefix`` is empty for ``RepositoryContent`` and ``in_repo__`` for the
35+
filtered membership join. Both catalog paths use this helper.
36+
"""
37+
return Q(**{f"{prefix}version_added__number__lte": repository_version.number}) & (
38+
Q(**{f"{prefix}version_removed__isnull": True})
39+
| Q(**{f"{prefix}version_removed__number__gt": repository_version.number})
40+
)
41+
42+
43+
def memberships_in_version(repository_version):
44+
"""RepositoryContent rows contained in ``repository_version``.
45+
46+
A subquery against this queryset keeps the content-id list in the database.
47+
``RepositoryVersion.content`` inlines ``content_ids`` as one bound UUID per
48+
content unit whenever that array is shorter than 65535.
49+
"""
50+
return RepositoryContent.objects.filter(
51+
repository_id=repository_version.repository_id,
52+
).filter(_in_version_bounds_q(repository_version))
53+
54+
55+
def python_packages_in_version(repository_version):
56+
"""Python package content contained in ``repository_version``."""
57+
if repository_version is None:
58+
return PythonPackageContent.objects.none()
59+
content_ids = (
60+
memberships_in_version(repository_version)
61+
.filter(content__pulp_type=PythonPackageContent.get_pulp_type())
62+
.order_by()
63+
.values("content_id")
64+
)
65+
return PythonPackageContent.objects.filter(pk__in=content_ids)
66+
67+
68+
def resolve_repository_version(href):
69+
"""Load a repository version without selecting ``content_ids``.
70+
71+
Accepts a repository-version or repository HREF/PRN. Domain-prefixed hrefs
72+
include ``pulp_domain`` in the resolved kwargs; that field lives on the
73+
repository, not ``RepositoryVersion``.
74+
"""
75+
if not href:
76+
raise ValueError("No value supplied for repository version.")
77+
78+
if href.startswith("prn:"):
79+
model, pk = resolve_prn(href)
80+
found_kwargs = {"pk": pk}
81+
else:
82+
try:
83+
match = resolve(urlparse(href).path)
84+
except Resolver404:
85+
raise ValueError(f"URI not valid: {href}") from None
86+
model = match.func.cls.queryset.model
87+
found_kwargs = match.kwargs
88+
89+
if "pk" in found_kwargs:
90+
lookup = {"pk": found_kwargs["pk"]}
91+
else:
92+
lookup = {}
93+
for key, value in found_kwargs.items():
94+
if key.endswith("_pk"):
95+
lookup[f"{key[:-3]}__pk"] = value
96+
elif key == "pulp_domain":
97+
if hasattr(model, "pulp_domain"):
98+
lookup["pulp_domain__name"] = value
99+
elif key in ("api_root", "version"):
100+
continue
101+
else:
102+
lookup[key] = value
103+
104+
qs = model.objects.all()
105+
if issubclass(model, RepositoryVersion):
106+
qs = qs.defer("content_ids")
107+
try:
108+
obj = qs.get(**lookup)
109+
except model.DoesNotExist:
110+
raise ValueError(f"URI {href} not found.") from None
111+
except (model.MultipleObjectsReturned, DjangoValidationError, FieldError):
112+
raise ValueError(f"URI {href} is not a valid repository version.") from None
113+
114+
if isinstance(obj, RepositoryVersion):
115+
return obj
116+
if isinstance(obj, Repository):
117+
try:
118+
return obj.versions.complete().defer("content_ids").latest()
119+
except RepositoryVersion.DoesNotExist:
120+
return None
121+
raise ValueError("Must be a repository version.")
122+
123+
124+
def distinct_package_names_qs(content_qs):
125+
"""One row per distinct ``name_normalized``, ordered for stable pagination."""
126+
return content_qs.order_by().values("name_normalized").distinct().order_by("name_normalized")
127+
128+
129+
def assemble_package_index(content_qs, name_rows, repository_version):
130+
"""One catalog row per package name, with one content unit per stored version.
131+
132+
Wheel and sdist of the same ``(name_normalized, version)`` collapse to one
133+
row. An sdist is preferred; otherwise the newest ``pulp_created`` is kept.
134+
Version strings are the values stored on the content unit, newest first.
135+
Summary, description, author, author email, and license are taken from the
136+
newest version because they do not vary by version.
137+
"""
138+
if not name_rows or repository_version is None:
139+
return []
140+
141+
names = [row["name_normalized"] for row in name_rows]
142+
by_name = defaultdict(list)
143+
# DISTINCT ON keeps the first row for each name and stored version.
144+
units = (
145+
content_qs.filter(name_normalized__in=names)
146+
.order_by(
147+
"name_normalized",
148+
"version",
149+
Case(
150+
When(packagetype="sdist", then=Value(0)),
151+
default=Value(1),
152+
output_field=IntegerField(),
153+
),
154+
"-pulp_created",
155+
"pk",
156+
)
157+
.distinct("name_normalized", "version")
158+
)
159+
for unit in units:
160+
by_name[unit.name_normalized].append(unit)
161+
162+
result = []
163+
for row in name_rows:
164+
normalized = row["name_normalized"]
165+
versions = sorted(
166+
by_name.get(normalized, []),
167+
key=lambda unit: (_version_sort_key(unit.version), unit.filename),
168+
reverse=True,
169+
)
170+
source = versions[0] if versions else None
171+
result.append(
172+
{
173+
"name": source.name if source else normalized,
174+
"name_normalized": normalized,
175+
"summary": source.summary if source else "",
176+
"description": source.description if source else "",
177+
"author": source.author if source else "",
178+
"author_email": source.author_email if source else "",
179+
"license": source.license if source else "",
180+
"versions": [
181+
{"version": unit.version, "license_expression": unit.license_expression}
182+
for unit in versions
183+
],
184+
}
185+
)
186+
return result

‎pulp_python/app/pypi/views.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -340,7 +340,7 @@ def list(self, request, path):
340340
if self.should_redirect(repo_version=repo_version):
341341
return redirect(urljoin(self.base_content_url, f"{path}/simple/"))
342342
names = (
343-
content.order_by("name_normalized")
343+
content.order_by("name_normalized", "name")
344344
.values_list("name", flat=True)
345345
.distinct("name_normalized")
346346
.iterator()

‎pulp_python/app/serializers.py‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -641,6 +641,57 @@ class Meta:
641641
model = python_models.PythonPackageContent
642642

643643

644+
class PythonRepositoryPackageVersionSerializer(serializers.Serializer):
645+
"""One stored version of a package."""
646+
647+
version = serializers.CharField(help_text=_("The package's version number."))
648+
license_expression = serializers.CharField(
649+
allow_blank=True,
650+
help_text=_("Text string that is a valid SPDX license expression."),
651+
)
652+
653+
654+
class PythonRepositoryPackageSerializer(serializers.Serializer):
655+
"""One distinct package name and its stored versions."""
656+
657+
name = serializers.CharField(
658+
help_text=_("A project name stored on one of this package's files."),
659+
)
660+
name_normalized = serializers.CharField(
661+
help_text=_("PEP 503 normalized package name. Index rows are unique on this field."),
662+
)
663+
summary = serializers.CharField(
664+
allow_blank=True,
665+
help_text=_("A one-line summary of what the package does."),
666+
)
667+
description = serializers.CharField(
668+
allow_blank=True,
669+
help_text=_("A longer description of the package that can run to several paragraphs."),
670+
)
671+
author = serializers.CharField(
672+
allow_blank=True,
673+
help_text=_(
674+
"Text containing the author's name. Contact information can also be added,"
675+
" separated with newlines."
676+
),
677+
)
678+
author_email = serializers.CharField(
679+
allow_blank=True,
680+
help_text=_("The author's e-mail address."),
681+
)
682+
license = serializers.CharField(
683+
allow_blank=True,
684+
help_text=_("Text indicating the license covering the distribution."),
685+
)
686+
versions = PythonRepositoryPackageVersionSerializer(
687+
many=True,
688+
help_text=_(
689+
"One entry per stored version, newest first. A wheel and an sdist of the "
690+
"same version are one entry. Version strings are unchanged."
691+
),
692+
)
693+
694+
644695
class PackageProvenanceSerializer(core_serializers.NoArtifactContentUploadSerializer):
645696
"""
646697
A Serializer for PackageProvenance.

‎pulp_python/app/tasks/publish.py‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -56,11 +56,14 @@ def write_simple_api(publication):
5656
domain = get_domain()
5757
simple_dir = "simple/"
5858
os.mkdir(simple_dir)
59+
# Secondary ORDER BY makes DISTINCT ON pick a stable display name when
60+
# metadata names differ but canonicalize to the same name_normalized
61+
# (e.g. msg-parser vs msg_parser).
5962
project_names = (
6063
python_models.PythonPackageContent.objects.filter(
6164
pk__in=publication.repository_version.content, _pulp_domain=domain
6265
)
63-
.order_by("name_normalized")
66+
.order_by("name_normalized", "name")
6467
.values_list("name", flat=True)
6568
.distinct("name_normalized")
6669
)
@@ -81,7 +84,7 @@ def write_simple_api(publication):
8184
packages = python_models.PythonPackageContent.objects.filter(
8285
pk__in=publication.repository_version.content, _pulp_domain=domain
8386
)
84-
releases = packages.order_by("name_normalized").values("name", "filename", "sha256")
87+
releases = packages.order_by("name_normalized", "filename").values("name", "filename", "sha256")
8588

8689
ind = 0
8790
current_name = canonicalize_name(project_names[ind])

‎pulp_python/app/utils.py‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,6 @@
2626

2727
log = logging.getLogger(__name__)
2828

29-
3029
PYPI_LAST_SERIAL = "X-PYPI-LAST-SERIAL"
3130
"""TODO This serial constant is temporary until Python repositories implements serials"""
3231
PYPI_SERIAL_CONSTANT = 1000000000

0 commit comments

Comments
 (0)