Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/dev/guides/agent-envs.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,6 @@ multi-env flow with `-e` if you prefer hand-written files; see

## When not to use lean agents

Interactive work that needs UI, Kafka, OpenTelemetry, MinIO, or other sidecars should keep using
Interactive work that needs UI, Kafka, OpenTelemetry, RustFS, or other sidecars should keep using
the normal profiles. Add those profiles with `--env COMPOSE_PROFILE=lean:…` only when required —
they increase resource use and reduce how many agents fit on one machine.
43 changes: 0 additions & 43 deletions docs/dev/reference/profiles/pminio.md

This file was deleted.

33 changes: 33 additions & 0 deletions docs/dev/reference/profiles/s3_storage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# s3_storage

## Usage

This profile starts a RustFS S3-compatible storage service and configures Pulp to use it. The S3 API
is available inside the environment at `http://rustfs:9000`. Neither the S3 API nor the RustFS
console is published to the host by default.

If you want to access Pulp distributions from the host, add the `rustfs` alias to `/etc/hosts`:

```
127.0.0.1 localhost localhost4 rustfs
::1 localhost localhost6 rustfs
```

The alias is not needed in the container because the `rustfs` service is addressable on the internal
network. The profile creates the configured `pulp` bucket during startup.

To expose the S3 API on the host, add a port mapping under the `rustfs` service in a Compose
override, for example `127.0.0.1:9100:9000`. Add `127.0.0.1:9101:9001` if you also want to expose
the RustFS console. The internal S3 URL used by Pulp remains `http://rustfs:9000`.

## Extra Variables

- `S3_ENDPOINT_URL`
- Description: The internal URL for the RustFS S3 API.
- Default: http://rustfs:9000
- `S3_ACCESS_KEY`
- Description: The S3 access key used by Pulp and RustFS.
- Default: pulps3storageaccesskey
- `S3_SECRET_KEY`
- Description: The S3 secret key used by Pulp and RustFS.
- Default: pulps3storageinsecuresecretkey
2 changes: 1 addition & 1 deletion profiles/lean/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ coding agents running scoped functional tests).
- Sets `PULP_WORKERS=1` so each environment runs a single task worker
- Caps the `pulp` container with `mem_limit` and `cpus` so parallel stacks do not starve the host

This profile does **not** add sidecars (UI, Kafka, OpenTelemetry, MinIO, etc.). Combine it with
This profile does **not** add sidecars (UI, Kafka, OpenTelemetry, RustFS, etc.). Combine it with
other profiles only when those services are required.

## Usage
Expand Down
42 changes: 0 additions & 42 deletions profiles/pminio/README.md

This file was deleted.

24 changes: 0 additions & 24 deletions profiles/pminio/compose.yaml

This file was deleted.

38 changes: 0 additions & 38 deletions profiles/pminio/init.sh

This file was deleted.

6 changes: 0 additions & 6 deletions profiles/pminio/profile_default_config.env

This file was deleted.

33 changes: 33 additions & 0 deletions profiles/s3_storage/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# s3_storage

## Usage

This profile starts a RustFS S3-compatible storage service and configures Pulp to use it. The S3 API
is available inside the environment at `http://rustfs:9000`. Neither the S3 API nor the RustFS
console is published to the host by default.

To access Pulp distributions from the host, add the `rustfs` alias to `/etc/hosts`:

```
127.0.0.1 localhost localhost4 rustfs
::1 localhost localhost6 rustfs
```

The alias is not needed in the container because the `rustfs` service is addressable on the internal
network. The profile creates the configured `pulp` bucket during startup.

To expose the S3 API on the host, add a port mapping under the `rustfs` service in a Compose
override, for example `127.0.0.1:9100:9000`. Add `127.0.0.1:9101:9001` if you also want to expose
the RustFS console. The internal S3 URL used by Pulp remains `http://rustfs:9000`.

## Extra Variables

- `S3_ENDPOINT_URL`
- Description: The internal URL for the RustFS S3 API.
- Default: http://rustfs:9000
- `S3_ACCESS_KEY`
- Description: The S3 access key used by Pulp and RustFS.
- Default: pulps3storageaccesskey
- `S3_SECRET_KEY`
- Description: The S3 secret key used by Pulp and RustFS.
- Default: pulps3storageinsecuresecretkey
22 changes: 22 additions & 0 deletions profiles/s3_storage/compose.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
services:
rustfs:
image: rustfs/rustfs:latest
environment:
- "RUSTFS_ACCESS_KEY={S3_ACCESS_KEY}"
- "RUSTFS_SECRET_KEY={S3_SECRET_KEY}"
- "RUSTFS_VOLUMES=/data"
- "RUSTFS_ADDRESS=0.0.0.0:9000"
- "RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001"
- "RUSTFS_CONSOLE_ENABLE=true"
volumes:
- "oci_s3_storage:/data"
healthcheck:
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:9000/health"]
interval: 20s
timeout: 5s
retries: 5
start_interval: 5s

volumes:
oci_s3_storage:
name: "{COMPOSE_PROJECT_NAME}_s3_storage"
56 changes: 56 additions & 0 deletions profiles/s3_storage/init.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
#!/bin/bash

set -e

if ! python3 -c 'import boto3, storages' >/dev/null 2>&1; then
echo "Installing S3 backend"
pip3 install --prefix /usr/local/ --no-cache-dir 'django-storages[boto3]'
fi

echo "Setting up RustFS backend"
python3 - <<'PY'
import os
import time

import boto3
from botocore.config import Config
from botocore.exceptions import ClientError

endpoint_url = os.environ["PULP_STORAGES__default__OPTIONS__endpoint_url"]
access_key = os.environ["PULP_STORAGES__default__OPTIONS__access_key"]
secret_key = os.environ["PULP_STORAGES__default__OPTIONS__secret_key"]
bucket_name = os.environ["PULP_STORAGES__default__OPTIONS__bucket_name"]

s3 = boto3.client(
"s3",
endpoint_url=endpoint_url,
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
region_name="us-east-1",
config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)

for attempt in range(1, 7):
try:
s3.create_bucket(Bucket=bucket_name)
print(f"Created RustFS {bucket_name} bucket")
break
except Exception as error:
already_exists = (
isinstance(error, ClientError)
and error.response.get("Error", {}).get("Code")
in {
"BucketAlreadyOwnedByYou",
"BucketAlreadyExists",
}
)
if already_exists:
print(f"RustFS {bucket_name} bucket already created")
break
if attempt == 6:
raise
print(f"RustFS bucket creation attempt {attempt}/6 failed; retrying in 5 seconds")
time.sleep(5)
else:
raise RuntimeError("Failed to create RustFS bucket")
PY
3 changes: 3 additions & 0 deletions profiles/s3_storage/profile_default_config.env
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
S3_ENDPOINT_URL=http://rustfs:9000
S3_ACCESS_KEY=pulps3storageaccesskey
S3_SECRET_KEY=pulps3storageinsecuresecretkey
File renamed without changes.
Loading