Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,17 @@ IMPORT_SOURCE_AWS_BUCKET=
# IMPORT_SOURCE_AWS_SECRET_ACCESS_KEY=
# IMPORT_SOURCE_AWS_DEFAULT_REGION=

# Fleetbase Cloud: the cloud API registers itself with a request signed by this secret (the
# same value as REGISTRY_CLOUD_PLATFORM_SECRET in the cloud API, 32+ characters). Empty: off.
REGISTRY_CLOUD_PLATFORM_SECRET=
# Organization (slug) that owns the cloud instance, and the host the cloud API registers with.
REGISTRY_CLOUD_ORGANIZATION=fleetbase
REGISTRY_CLOUD_HOST=

# Cutover freeze: publishing and checkout answer 503 (or `php artisan registry:maintenance on`).
REGISTRY_MAINTENANCE=false
# REGISTRY_MAINTENANCE_MESSAGE=

# Registry
REGISTRY_VERSION=2.0.0-dev
REGISTRY_CATALOG_CACHE_SECONDS=60
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,4 @@ Thumbs.db
/playwright-report
/test-results
/storage/e2e
/storage/rehearsal
28 changes: 26 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ Phase 1, Workstreams A to C, and Phase 2 Workstream E:
| Website (Workstream F): Inertia (React) catalog, account, Studio and review console, light and dark | done |
| Commerce (Phase 4, K to M): Stripe Connect Express, prices mirrored to Stripe, Checkout from instances and the website, webhooks, payouts and billing APIs and pages, staff finance and refunds | done |
| Publisher analytics: listing and organization stats APIs with CSV, Studio Insights | done |
| Cloud cutover (Phase 5, O): the cloud instance (signed registration, per-company tenancy), cutover freeze, importer on one cloud instance with `--verify`, [runbook](docs/cutover.md) and rehearsal scripts | done |
| CLI v2 (G), staff finance view | later workstreams |

## Stack
Expand Down Expand Up @@ -422,6 +423,26 @@ disables a token.
| `GET /v1/instances/me/updates?channel=` | Every extension installed on the instance: the installed release, the newest compatible published release, `update_available`. `channel=beta` includes beta releases |
| `POST /v1/instances/me/installs` | Body: `extension` (slug, `ext_` id or an npm/Composer package name), `version` or `release` (`rel_` id), `status` (`installing`, `installed`, `failed`), optional `error`, `installed_by`. Returns `{install, activation}` |
| `DELETE /v1/instances/me/installs/{extension}` | Reports an uninstall (scoped npm names work in the path). Returns `{install, deactivated}` |
| `POST /v1/instances/register/cloud` | Fleetbase Cloud only. No bearer token and no code: signed with the platform secret (below). Body: `host` (required), optional `name`, `console_version`, `api_version`, `previous_token`. Same response as `register` |
| `GET /v1/instances/me/tenant-installs?page=&per_page=` | Cloud instance only: every installed extension per cloud company (`company_uuid`, `extension`, `packages`, `release`), 500 a page |

**Fleetbase Cloud.** The cloud console's API registers one instance per deployment (kind `cloud`,
`meta.platform`) in the organization `REGISTRY_CLOUD_ORGANIZATION`, on boot and without a code:
`X-Registry-Timestamp` (unix seconds, within 5 minutes), `X-Registry-Nonce` (16 to 64 characters,
single use) and `X-Registry-Signature: v1=<hex HMAC-SHA256 of "{timestamp}.{nonce}.{body}" with
REGISTRY_CLOUD_PLATFORM_SECRET>`. A missing secret answers 403, a bad, expired or replayed
signature 401. Re-registration rotates the same instance by `previous_token`, else by host (D17),
else it adopts the tokenless instance `registry:import` prepared. That instance serves every cloud
company: `X-Registry-Tenant: <company uuid>` on any instance route makes it act for that company's
organization (licences in `me`, install rows and seats, checkouts and hand-offs); an unknown company
gets an organization named from `X-Registry-Tenant-Name`. Other instances never act for anyone. The
package gate lets the cloud instance download a paid package while one of its companies holds an
active seat on it.

**Maintenance.** `php artisan registry:maintenance on [--message=]` (or `REGISTRY_MAINTENANCE=true`)
answers 503 `{errors: [message], maintenance: true}` with `Retry-After` on publishing (publisher
writes, review approve and publish) and on both checkouts, for the cutover window
([docs/cutover.md](docs/cutover.md)). Catalog, packages, installs and Stripe webhooks keep working.

Re-registration:

Expand Down Expand Up @@ -704,11 +725,14 @@ php artisan registry:import --from=mysql://reader:secret@cloud-replica:3306/flee

| Option | Meaning |
| --- | --- |
| `--from` | `mysql://user:pass@host:3306/db` (or `mariadb://`), `sqlite:///path/file.sqlite`, or a path to a `.sql` / `.sql.gz` dump. Dumps are loaded into a temporary SQLite file without a MySQL server; mysqldump's extended inserts and one-tuple-per-line dumps both work, and statements for other tables are skipped |
| `--from` | `mysql://user:pass@host:3306/db` (or `mariadb://`; `?unix_socket=` and `?ssl_ca=` accepted), `sqlite:///path/file.sqlite`, a path to a `.sql` / `.sql.gz` dump, or `disk:path` for a dump on a filesystem disk (`import_source:registry-cutover/cloud.sql.gz`). Dumps are loaded into a temporary SQLite file without a MySQL server; mysqldump's extended inserts and one-tuple-per-line dumps both work, and statements for other tables are skipped. `scripts/cloud-registry-dump.sh` dumps exactly the rows the importer reads |
| `--dry-run` | Runs the whole mapping in a transaction that is rolled back, so the counts and anomalies are exactly what a real run would produce |
| `--source-disk` | Filesystem disk holding the cloud `files` objects. Default `REGISTRY_IMPORT_SOURCE_DISK` (`import_source`, an S3 disk configured by `IMPORT_SOURCE_AWS_*`) |
| `--report` | Also writes the full report, every anomaly included, as JSON |
| `--examples` | Anomaly examples printed per type (default 10) |
| `--memory-limit` | PHP memory limit for the run (default `512M`, never lowered; the source tables are held in memory). The peak is printed and kept in the report |
| `--cloud-host` | Host the cloud API registers with (default `REGISTRY_CLOUD_HOST`); cloud installs attach to that host's cloud instance |
| `--verify` | Compares source and target instead of importing; exits 1 on drift. A dry run against the target must create or update nothing; every extension, release, purchase, licence and install has its row except the ones the importer skips; a seeded sample of rows (`--sample`, default 25; `--seed`, default today) matches field by field, tarballs by SHA-256 of the source object, the copy and its record |

What it writes:

Expand All @@ -721,7 +745,7 @@ What it writes:
| `registry_extensions` | `extensions`, `prices`, `extension_packages` | Package names from the current bundle's `package.json` and `composer.json`, then older bundles, then the Phase 0 `package_name` column; listings with no bundle and no package name become drafts; `self_managed` maps to hosting `self_hosted`; icon and screenshots copied |
| `registry_extension_bundles` | `releases` | Version from `extension.json`; `bundle_number` kept in `manifest`; status pending → uploaded, awaiting_review → in_review, approved, published, rejected; `compat` from `extension.json` when present. The bundle archive is copied to `tarballs/imported/…` as a `source_bundle` file with a SHA-256 checksum; a bundle whose file is missing imports as `yanked` |
| `registry_extension_purchases` | `purchases`, `licences` | One paid licence per purchase with 5 seats; a soft-deleted purchase revokes its licence |
| `registry_extension_installs` | `instances`, `installs`, `licence_activations` | One synthetic cloud instance per company (no token); soft-deleted installs become `uninstalled`; live installs claim a seat on the paid licence or an automatic free licence |
| `registry_extension_installs` | `instances`, `installs`, `licence_activations` | One cloud instance for all companies (the cloud host's, else a tokenless placeholder its first registration adopts); each install is for its company's organization; soft-deleted installs become `uninstalled`; live installs claim a seat on the company's paid licence or an automatic free licence. Per-company instances from earlier runs are revoked and their seats moved |

Idempotency: every imported row stores the uuid of its source row in `source_uuid`, and
re-runs match on it and update in place; an unchanged source produces only "unchanged"
Expand Down
10 changes: 6 additions & 4 deletions app/Actions/ActivateLicence.php
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,9 @@
* extension, creating the automatic free licence on first install of a free extension.
*
* Seat = instance (D4). Re-activating an instance that already holds a seat only moves
* the activation to the new release.
* the activation to the new release. The cloud instance acting for a company uses that
* company's organization's licences (`Instance::licensingOrganizationUuid()`), so each cloud
* company holds its own seat on the one cloud instance.
*/
class ActivateLicence
{
Expand All @@ -25,7 +27,7 @@ public function __invoke(Instance $instance, Extension $extension, ?Release $rel
$licences = Licence::query()
->active()
->where('extension_uuid', $extension->uuid)
->where('organization_uuid', $instance->organization_uuid)
->where('organization_uuid', $instance->licensingOrganizationUuid())
->orderByRaw('case when kind = ? then 0 else 1 end', [Licence::KIND_PAID])
->orderBy('created_at')
->lockForUpdate()
Expand Down Expand Up @@ -82,7 +84,7 @@ protected function freeLicence(Instance $instance, Extension $extension): Licenc

$revoked = Licence::query()
->where('extension_uuid', $extension->uuid)
->where('organization_uuid', $instance->organization_uuid)
->where('organization_uuid', $instance->licensingOrganizationUuid())
->where('status', Licence::STATUS_REVOKED)
->exists();

Expand All @@ -92,7 +94,7 @@ protected function freeLicence(Instance $instance, Extension $extension): Licenc

return Licence::create([
'extension_uuid' => $extension->uuid,
'organization_uuid' => $instance->organization_uuid,
'organization_uuid' => $instance->licensingOrganizationUuid(),
'kind' => Licence::KIND_FREE,
'max_activations' => null,
'status' => Licence::STATUS_ACTIVE,
Expand Down
8 changes: 5 additions & 3 deletions app/Actions/DeactivateLicence.php
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@
use App\Models\LicenceActivation;

/**
* Releases every seat an instance holds on licences for an extension.
* Returns the number of activations closed.
* Releases every seat an instance holds on licences for an extension; for the cloud instance
* acting for a company, only that company's seats. Returns the number of activations closed.
*/
class DeactivateLicence
{
Expand All @@ -17,7 +17,9 @@ public function __invoke(Instance $instance, Extension $extension): int
return LicenceActivation::query()
->where('instance_uuid', $instance->uuid)
->whereNull('deactivated_at')
->whereHas('licence', fn ($q) => $q->where('extension_uuid', $extension->uuid))
->whereHas('licence', fn ($q) => $q
->where('extension_uuid', $extension->uuid)
->when($instance->tenant(), fn ($q, $tenant) => $q->where('organization_uuid', $tenant->uuid)))
->update(['deactivated_at' => now(), 'updated_at' => now()]);
}
}
105 changes: 101 additions & 4 deletions app/Console/Commands/ImportRegistry.php
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
namespace App\Console\Commands;

use App\Import\ImportReport;
use App\Import\ImportVerifier;
use App\Import\RegistryImporter;
use App\Import\SourceDatabase;
use App\Models\Event;
Expand All @@ -14,23 +15,34 @@
class ImportRegistry extends Command
{
protected $signature = 'registry:import
{--from= : mysql://user:pass@host:3306/db, sqlite:///path/to/file.sqlite, or a path to a .sql / .sql.gz dump}
{--from= : mysql://user:pass@host:3306/db, sqlite:///path/to/file.sqlite, a path to a .sql / .sql.gz dump, or disk:path for a dump on a filesystem disk (import_source:cutover/cloud.sql.gz)}
{--dry-run : Map everything and print counts and anomalies, then roll back; no files are copied}
{--verify : Compare source and target after an import (pending changes, counts, a sample of rows and tarballs); exits 1 on drift}
{--sample=25 : Rows per table compared field by field with --verify}
{--seed= : Sample seed for --verify (default: today\'s date)}
{--cloud-host= : Host the cloud API registers with; cloud installs attach to its instance (default: registry.cloud.host)}
{--source-disk= : Filesystem disk holding the cloud `files` objects (default: registry.import.source_disk)}
{--report= : Also write the full report, every anomaly included, as JSON to this path}
{--examples=10 : Anomaly examples printed per type}';
{--examples=10 : Anomaly examples printed per type}
{--memory-limit=512M : PHP memory limit for the run (the source tables are held in memory); never lowered}';

protected $description = 'Import extensions, releases, accounts, purchases and installs from the cloud registry-bridge tables';

public function handle(): int
{
$this->raiseMemoryLimit((string) $this->option('memory-limit'));

$from = (string) $this->option('from');
if ($from === '') {
$this->error('--from is required: a mysql:// or sqlite:// DSN, or a path to a SQL dump.');

return self::INVALID;
}

if ($this->option('verify')) {
return $this->verify(new SourceDatabase($from));
}

$dryRun = (bool) $this->option('dry-run');
$sourceDisk = $this->option('source-disk') ?: config('registry.import.source_disk');
$targetDisk = config('registry.storage.disk');
Expand All @@ -41,7 +53,7 @@ public function handle(): int

try {
$connection = $source->open();
$report = (new RegistryImporter($connection, $sourceDisk, $targetDisk, $dryRun))->run();
$report = (new RegistryImporter($connection, $sourceDisk, $targetDisk, $dryRun, $this->cloudHost()))->run();
} catch (\Throwable $e) {
$this->error('Import failed: '.$e->getMessage());

Expand All @@ -60,8 +72,11 @@ public function handle(): int
], '127.0.0.1');
}

$peak = (int) round(memory_get_peak_usage(true) / 1048576);
$this->line("Peak memory: {$peak} MB of ".ini_get('memory_limit'));

if ($path = $this->option('report')) {
file_put_contents($path, json_encode($report->toArray(), JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE));
file_put_contents($path, json_encode([...$report->toArray(), 'peak_memory_mb' => $peak], JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE));
$this->line("Full report written to {$path}");
}

Expand All @@ -71,6 +86,88 @@ public function handle(): int
return self::SUCCESS;
}

/**
* `--verify`: exit 0 when the target matches the source, 1 on drift (see ImportVerifier).
*/
protected function verify(SourceDatabase $source): int
{
$this->line('<info>Verifying</info> '.$source->describe().' against this registry');

try {
$verifier = (new ImportVerifier(
$source->open(),
$this->option('source-disk') ?: config('registry.import.source_disk'),
config('registry.storage.disk'),
$this->cloudHost(),
max(0, (int) $this->option('sample')),
$this->option('seed') !== null ? (int) $this->option('seed') : null,
))->run();
} catch (\Throwable $e) {
$this->error('Verify failed: '.$e->getMessage());

return self::FAILURE;
} finally {
$source->close();
}

$this->newLine();
$this->line('<options=bold>Pending changes</> (a dry run against this registry)');
$this->line($verifier->pending === [] ? ' none: the import is current' : '');
if ($verifier->pending !== []) {
$this->table(['Table', 'Would create', 'Would update'], collect($verifier->pending)->map(fn ($c, $t) => [$t, $c['created'], $c['updated']])->values()->all());
}

$this->line('<options=bold>Counts</> (target rows matched on source_uuid)');
$this->table(['Table', 'Source', 'Target', 'Missing', 'Skipped by importer', 'Drift'], collect($verifier->counts)->map(fn ($c) => [
$c['table'], $c['source'], $c['target'], $c['missing'], $c['skipped'], $c['drift'] ? '<error>yes</error>' : 'no',
])->all());

$this->line('<options=bold>Sample</> ('.collect($verifier->sampled)->map(fn ($n, $t) => "{$t} {$n}")->implode(', ').')');
if ($verifier->mismatches === []) {
$this->line(' every sampled row matches');
} else {
$this->table(['Table', 'Source uuid', 'Field', 'Source', 'Target'], collect($verifier->mismatches)->map(fn ($m) => [
$m['table'], $m['uuid'], $m['field'], json_encode($m['source']), json_encode($m['target']),
])->all());
}

$cloud = $verifier->cloudInstance;
$this->line('Cloud instance: '.($cloud ? "{$cloud->public_id} ({$cloud->host}, ".($cloud->token_hash ? 'registered' : 'waiting for the cloud API to register').')' : 'none'));

$this->newLine();
if ($verifier->drift()) {
$this->error('Drift between source and target.');

return self::FAILURE;
}

$this->info('No drift.');

return self::SUCCESS;
}

/**
* Raise PHP's memory limit to `$limit` (e.g. 512M); a higher or unlimited one is kept.
*/
protected function raiseMemoryLimit(string $limit): void
{
$bytes = fn (string $value) => (int) $value * match (strtoupper(substr(trim($value), -1))) {
'G' => 1024 ** 3, 'M' => 1024 ** 2, 'K' => 1024, default => 1,
};

$current = (string) ini_get('memory_limit');
if ($limit !== '' && $current !== '-1' && $bytes($limit) > $bytes($current)) {
ini_set('memory_limit', $limit);
}
}

protected function cloudHost(): ?string
{
$host = $this->option('cloud-host') ?: config('registry.cloud.host');

return is_string($host) && trim($host) !== '' ? trim($host) : null;
}

protected function printReport(ImportReport $report, int $examples): void
{
$this->newLine();
Expand Down
48 changes: 48 additions & 0 deletions app/Console/Commands/ManageMaintenance.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
<?php

namespace App\Console\Commands;

use App\Support\Maintenance;
use Illuminate\Console\Command;

/**
* `php artisan registry:maintenance on` freezes publishing and checkout for the cutover window;
* `off` lifts it; `status` shows it. Catalog, package downloads and Stripe webhooks keep working.
*/
class ManageMaintenance extends Command
{
protected $signature = 'registry:maintenance
{action=status : on, off or status}
{--message= : Message shown to publishers and buyers (default: registry.maintenance.message)}';

protected $description = 'Turn the cutover freeze (publishing and checkout answer 503) on or off';

public function handle(): int
{
$action = strtolower((string) $this->argument('action'));
if (! in_array($action, ['on', 'off', 'status'], true)) {
$this->error("Unknown action [{$action}]: use on, off or status.");

return self::INVALID;
}

if ($action === 'on') {
Maintenance::enable($this->option('message') ?: null);
} elseif ($action === 'off') {
Maintenance::disable();
}

$status = Maintenance::status();
$this->line(sprintf('Maintenance: <%s>%s</>%s', $status['active'] ? 'comment' : 'info', $status['active'] ? 'ON' : 'off', $status['since'] ? " since {$status['since']}" : ''));
if ($status['active']) {
$this->line('Message: '.$status['message']);
}
if ($action === 'off' && $status['forced']) {
$this->warn('REGISTRY_MAINTENANCE=true still forces it on; unset it and restart the tasks.');

return self::FAILURE;
}

return self::SUCCESS;
}
}
Loading
Loading