Skip to content

Cloud cutover: cloud instance and tenancy, freeze, importer --verify, runbook (Workstream O) - #14

Open
roncodes wants to merge 4 commits into
release/v2.0.0from
feat/cloud-cutover
Open

roncodes wants to merge 4 commits into
release/v2.0.0from
feat/cloud-cutover

Conversation

@roncodes

Copy link
Copy Markdown
Member

Phase 5, Workstream O (cloud migration): the service side of the cutover and its runbook. Companion PRs: fleetbase/registry-bridge (cloud auto-registration, per-company requests, console freeze) and fleetbase/registry-infra#2 (verdaccio.desired_count).

Cloud instance identity

  • POST /v1/instances/register/cloud: the cloud console's API registers itself on boot, signed with a shared platform secret (REGISTRY_CLOUD_PLATFORM_SECRET) instead of a registration code. HMAC-SHA256 over {timestamp}.{nonce}.{body}, 5-minute window, single-use nonce (a replay would hand out a fresh token). 403 when no secret is configured, 401 for a bad, expired or replayed signature, checked before the body.
  • One instance per cloud deployment, kind cloud, meta.platform, owned by REGISTRY_CLOUD_ORGANIZATION (created on first use). Re-registration follows D17: previous_token, else same organization and host, else the tokenless instance the importer prepared. Another host is another deployment's instance.
  • Tenancy. The cloud instance serves every cloud company, but licences belong to each company's organization, so instance requests name the company: X-Registry-Tenant: <company uuid> (X-Registry-Tenant-Name names an organization for a company seen the first time). Licences in me, install rows and seats, checkouts and hand-offs then belong to that organization. Only the platform instance can act for anyone.
  • installs.organization_uuid (migration with backfill): one install row and seat per company on the cloud instance.
  • GET /v1/instances/me/tenant-installs: what every company has installed, for the bridge to record locally after the import.
  • Package gate: the cloud instance may download a paid package while one of its companies holds an active seat (the install report claims the seat before the download; Composer and npm cannot name a company).

Importer

  • Cloud installs and paid seats attach to that one cloud instance, each install for its company's organization, instead of a synthetic instance per company. --cloud-host (default REGISTRY_CLOUD_HOST) finds the instance its registration will rotate onto; otherwise a tokenless placeholder is created and adopted. Per-company instances from earlier runs are revoked, their seats moved.
  • registry:import --verify: exits 1 on drift. A dry run against the target must create or update nothing; every extension, release, purchase, licence and install has its row except those the importer skips; a seeded sample (--sample, --seed) matches field by field, tarballs by SHA-256 of the source object, the stored copy and its record.
  • --from=disk:path reads a dump from a filesystem disk (the cloud bucket the importer already reads), so production needs no network path to the cloud database. mysql:// sources take ?unix_socket= and ?ssl_ca=.
  • Fixed, found by the rehearsal: on MySQL every re-import counted extensions, releases and purchases as updated, because MySQL reorders JSON object keys and Eloquent compares them in order, so --verify would have reported drift on every production run. The import also ran out of PHP's default 128 MB at three times today's scale; it now sets 512 MB (--memory-limit) and prints its peak. And a migration index name over MySQL's 64 characters.

Maintenance freeze

php artisan registry:maintenance on|off|status [--message=] (stored in the cache store, so it holds across tasks and deploys) or REGISTRY_MAINTENANCE=true: publisher writes, review approve and publish, and both checkouts answer 503 {errors: [message], maintenance: true} with Retry-After. Catalog, package downloads, installs and Stripe webhooks keep working.

Runbook and rehearsal

  • docs/cutover.md: the five steps with exact commands (dump, dry run, import, verify, service.enabled / service.route_all / verdaccio.desired_count, deploy, second import, install sync, unfreeze), smoke tests, rollback per step, timings, go/no-go checklist.
  • scripts/fake-cloud-dump.php (seeded fake cloud database and objects, every anomaly class), scripts/rehearse-cutover.sh (times the data path end to end), scripts/cloud-registry-dump.sh (dumps exactly the rows the importer reads, in one transaction).

Tests

  • Pest: 527 pass, 39 of them new (cloud registration and its refusals, rotation and placeholder adoption, tenancy for me/installs/seats/checkout, tenant installs listing, the package gate, the freeze; importer: the cloud instance, retiring per-company instances, --verify clean and five drift cases, MySQL key order, disk dumps, socket/TLS DSNs). Pint clean.
  • Rehearsal on MySQL 8.1 (scripts/rehearse-cutover.sh, on a shared laptop): scale 1 (80 extensions, 453 bundles / 134 MB, 600 purchases, 9,000 installs): import 2 min 11 s, verify 31 s with exit 0, second import 1 min 3 s with every table unchanged, peak 72 MB. Scale 3 (27,000 installs): import 9 min 38 s, verify 3 min 11 s, peak 134 MB. Full table in the runbook.
  • Not run: anything against AWS, a real cloud snapshot, or S3 (local folders stood in for the buckets).

Conflicts

app/Services/LicenceGate.php and routes/api.php are also touched by #13 (one added method and condition; different lines in the instance route group). Whichever merges second needs a small rebase.

Decisions needed

See "Decisions this runbook depends on" in docs/cutover.md; the main ones: freezing the running cloud API (relies on a second import instead of a 0.1.x backport), whether to flip route_all while old consoles still read @fleetbase/* through Verdaccio, and service.enabled moving before the window.

Fleetbase Cloud registers one instance per deployment (kind cloud) that acts for each cloud
company's organization, and the service can freeze publishing and checkout during the cutover.

- POST /v1/instances/register/cloud: signed with the shared platform secret (HMAC-SHA256 over
  timestamp, single-use nonce and body; 5-minute window) instead of a registration code. The
  instance belongs to REGISTRY_CLOUD_ORGANIZATION and is marked meta.platform. Re-registration
  follows D17: previous_token, else same organization and host, else the placeholder the
  importer left; a new host is a separate deployment's instance.
- X-Registry-Tenant (company uuid) lets the platform instance act for that company's
  organization on every instance route: licences in /me, installs and seats, checkout and
  hand-offs. A company seen for the first time gets an organization (X-Registry-Tenant-Name).
  Other instances never act for anyone.
- installs.organization_uuid: one install row and seat per company on the cloud instance;
  existing rows take their instance's organization.
- GET /v1/instances/me/tenant-installs: what every company has installed, for the cloud API to
  record locally after the import.
- Package gate: the platform instance may download a paid package while a company holds an
  active seat on it (the install report claims the seat before the download).
- registry:maintenance on|off|status (or REGISTRY_MAINTENANCE=true): publisher writes, review
  approve and publish, and both checkouts answer 503 with the message and Retry-After. Reads,
  installs, package downloads and Stripe webhooks keep working.
- Cloud installs and paid licences' seats attach to the one cloud instance (CloudPlatform),
  each install for its company's organization, instead of a synthetic instance per company.
  The instance is found by --cloud-host (default REGISTRY_CLOUD_HOST) or created as a tokenless
  placeholder that the cloud API's first registration rotates onto. Per-company instances from
  earlier runs are revoked and their seats move to it.
- --verify compares source and target and exits 1 on drift: a dry run against the target must
  create or update nothing, every extension, release, purchase, licence and install has its row
  except the ones the importer skips, and a seeded sample of rows matches field by field,
  tarballs by SHA-256 of the source object, the stored copy and its record.
- Re-runs stay "unchanged" on MySQL, which stores JSON with its keys reordered (found in the
  cutover rehearsal: every extension, release and purchase counted as updated).
- mysql:// sources take ?unix_socket= and ?ssl_ca= (a TLS-only source database).
- IMPORT_SOURCE_DRIVER=local reads the cloud objects from a folder (rehearsals).
…e peak printed

- --from=disk:path downloads a dump from a configured disk (the cloud files bucket the importer
  already reads), so the production service needs no network path to the cloud database.
- The source tables are held in memory: the rehearsal at three times today's scale needed 134 MB,
  over PHP's default 128 MB. --memory-limit (default 512M, never lowered) and the peak in the output
  and the JSON report.
- docs/cutover.md: the five steps (freeze, import, infra, deploy, unfreeze) with exact commands,
  preparation, go/no-go checklist, smoke tests, rollback per step, decisions it depends on, and the
  timings from a local rehearsal.
- scripts/fake-cloud-dump.php: a seeded fake of the cloud registry tables and their objects, at any
  scale, with every anomaly class the importer reports. No real data.
- scripts/rehearse-cutover.sh: times the data path end to end against it (two dry runs, import,
  verify, second import, verify, dump, dry run from the dump) on MySQL and S3 or local folders.
- scripts/cloud-registry-dump.sh: dumps exactly the cloud rows the importer reads, in one
  transaction, for --from=import_source:<path>.
- README: the cloud instance API, maintenance, and the new import options.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant