Migrating Vroom data into Seaweed should not interrupt service uptime

[afaianswq]

Self-Hosted Version

25.5.1

CPU Architecture

x86_64

Docker Version

29.3.1

Docker Compose Version

5.1.1

Machine Specification

• My system meets the minimum system requirements of Sentry

Installation Type

Upgrade from 25.5.1 to 26.3.1

Steps to Reproduce

1. run install.sh

Expected Result

Sentry is successfully upgraded

Actual Result

Upgrade fails. Last lines of output before the failure:

upload: '/var/vroom/sentry-profiles/1/99/fffe61af14e64b9cb9f61900e3ced12f' -> 's3://profiles/1/99/fffe61af14e64b9cb9f61900e3ced12f' (14659 bytes in 0.0 seconds, 4.71 MB/s) [6097783 of 6097786]
upload: '/var/vroom/sentry-profiles/1/99/fffe61af14e64b9cb9f61900e3ced12f.attrs' -> 's3://profiles/1/99/fffe61af14e64b9cb9f61900e3ced12f.attrs' (211 bytes in 0.0 seconds, 81.24 KB/s) [6097784 of 6097786]
upload: '/var/vroom/sentry-profiles/1/99/fffe9037e581422a9eefc37405254f53' -> 's3://profiles/1/99/fffe9037e581422a9eefc37405254f53' (22012 bytes in 0.0 seconds, 6.63 MB/s) [6097785 of 6097786]
upload: '/var/vroom/sentry-profiles/1/99/fffe9037e581422a9eefc37405254f53.attrs' -> 's3://profiles/1/99/fffe9037e581422a9eefc37405254f53.attrs' (211 bytes in 0.0 seconds, 73.90 KB/s) [6097786 of 6097786]
Done. Uploaded 94406925034 bytes in 29262.6 seconds, 3.08 MB/s.
Migration completed.
WARNING: Retrying failed request: /?lifecycle (timed out)
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: /?lifecycle (timed out)
WARNING: Waiting 6 sec...
WARNING: Retrying failed request: /?lifecycle (timed out)
WARNING: Waiting 9 sec...
WARNING: Retrying failed request: /?lifecycle (timed out)
WARNING: Waiting 12 sec...
WARNING: Retrying failed request: /?lifecycle (timed out)
WARNING: Waiting 15 sec...
ERROR: S3 Temporary Error: Request failed for: /?lifecycle.  Please try again later.
Error in install/bootstrap-s3-profiles.sh:112.
'$s3cmd --access_key=sentry --secret_key=sentry --no-ssl --region=us-east-1 --host=localhost:8333 --host-bucket='localhost:8333/%(bucket)' setlifecycle /tmp/profiles-lifecycle-policy.xml s3://profiles' exited with status 75
-> ./install.sh:main:43
--> install/bootstrap-s3-profiles.sh:source:112

Clearly, trying to migrate ~ 6.1 million files into s3 (took roughly 12 hours) and then expecting to apply lifecycle policy within a few minutes was quite naive.

Install script should probably stop executing and offer a proper solution when so many files found in vroom volume. Or at least continue the upgrade after s3cmd failed to apply lifecycle policy, or even do it in the background without waiting for the result.

Event ID

No response

[aldy505]

I'll rename your issue to fit the problem instead of making it an "upgrade issue".

[afaianswq]

I'll rename your issue to fit the problem instead of making it an "upgrade issue".

Sure, the goal is to make sure the upgrade will not fail because of this problem.

Meanwhile, is it safe to simply delete sentry-self-hosted_sentry-vroom volume before the upgrade to skip this migration entirely? AFAIK, this volume contains not important runtime data.

[aldy505]

Meanwhile, is it safe to simply delete sentry-self-hosted_sentry-vroom volume before the upgrade to skip this migration entirely? AFAIK, this volume contains not important runtime data.

No you cannot, it will result in an invalid Docker volume state.