Skip to content

One-command local development and self-hosting with Docker Compose - #1

Open
nickmarden wants to merge 20 commits into
developfrom
lineage/docker-compose
Open

nickmarden wants to merge 20 commits into
developfrom
lineage/docker-compose

Conversation

@nickmarden

@nickmarden nickmarden commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

Work in progress. This is the self-hosting work from allmaps#672, against upstream develop at f3633bf.

It builds on three small fixes for bugs found while testing it. Each is its own pull request against develop, and this branch merges them first (ae3411f):

What and why

Two things that are hard today:

  1. Local development of the whole stack. Running everything takes several terminals, Node, pnpm, Rust and wasm-pack on your machine, and a local API runner that only talks to localhost.
  2. Running Allmaps on your own server. The apps and workers build for Cloudflare, the REST and Annotations APIs run only as Workers, auth assumes allmaps.org, and a new database comes with allmaps.org's partner organizations in it.

This PR makes both one command:

docker compose up                              # development: mounted source, hot reload, demo data
docker compose -f compose.yaml up -d --build   # deployment: self-contained images, empty database

Every service gets a hostname under one domain behind a Caddy proxy (http://editor.allmaps.localhost:5490 and so on locally), so there is one port to remember, and the same layout works on a server with TLS. docker/README.md covers both modes and every setting.

Those local addresses are the same for everyone, and that's on purpose. On your own machine, every developer gets these:

Service On your machine On a server
Editor http://editor.allmaps.localhost:5490 https://editor.$DOMAIN
Viewer http://viewer.allmaps.localhost:5490 https://viewer.$DOMAIN
Console http://console.allmaps.localhost:5490 https://console.$DOMAIN
REST API http://api.allmaps.localhost:5490 https://api.$DOMAIN
Annotations API http://annotations.allmaps.localhost:5490 https://annotations.$DOMAIN
Live API http://live.allmaps.localhost:5490 https://live.$DOMAIN
Tile Server http://tiles.allmaps.localhost:5490 https://tiles.$DOMAIN
Preview http://preview.allmaps.localhost:5490 https://preview.$DOMAIN
IIIF image server (Cantaloupe) http://iiif.allmaps.localhost:5490 https://iiif.$DOMAIN
IIIF fixtures, Demo Amsterdam Archive http://fixtures-amsterdam.allmaps.localhost:5490 None
IIIF fixtures, Demo European Map Library http://fixtures-europe.allmaps.localhost:5490 None

The fixture hostnames only exist in development mode. Both are the same fixture server; the demo organizations are told apart by domain, so each gets its own. The server column assumes TLS; with TLS=off it's http:// plus :$HTTP_PORT. So, locally:

  • One GitHub OAuth app setup works for everybody, and the README can give the exact callback URL (http://api.allmaps.localhost:5490/auth/callback/github) instead of "whatever your ports are".
  • Links in issues, PRs and docs work on anyone's machine.
  • Allowlists only need one entry: adding *.allmaps.localhost to the Protomaps key would cover every local install.
  • The session cookie is shared across *.allmaps.localhost, so logging in once works for the Console, the Editor and the APIs.
  • Browsers resolve *.localhost to your own machine without touching /etc/hosts (as do macOS and mainstream Linux for other tools).

If port 5490 is taken, or you'd like different hostnames, set HTTP_PORT or DOMAIN. Most people won't need to.

Nothing changes for allmaps.org unless it opts in: every new setting defaults to today's behaviour, and the Cloudflare builds and deploy scripts are untouched.

Commits

  1. Node entry points for the REST and Annotations APIs (5988765). They ran only as Cloudflare Workers; serveNode() in @allmaps/api-shared/node serves the same apps on Node with plain PostgreSQL. Live already ran on Node.
  2. ADAPTER=node builds the Editor, Viewer and Console with adapter-node (8a18091). They already read their settings at runtime, so one image serves every environment.
  3. A migration runner that also sets up an empty database (11fe9a6). migrate.ts works like migrate-ci.ts, but creates PostGIS and the migration history table first, so a fresh install migrates in one step.
  4. The Tile Server and Preview only write to the Cache API when USE_CACHE is on (3d8c1ff). They already only read with it on. workerd without a cache rejects the write, so every response failed.
  5. wrangler is a devDependency of the REST and Annotations APIs (248adcc). It deploys them; as a dependency it added about 200 MB (wrangler and workerd) to every production install.
  6. The auth cookie domain and trusted origins are settings (49eaae2): BETTER_AUTH_COOKIE_DOMAIN and BETTER_AUTH_TRUSTED_ORIGINS. Both were hardcoded to allmaps.org and localhost. Unset, behaviour is unchanged.
  7. Regenerated worker types (c831f98). The committed worker-configuration.d.ts files predate the wrangler version in the lockfile, so every pnpm install or pnpm types rewrote them.
  8. The demo seed on PostgreSQL (21fe2d0). tools/seed only ran on PGlite. tools/seed/src/postgres.ts seeds a PostgreSQL database once; the fixture server's origins are now a parameter, and each demo organization gets an unmapped image so the Editor's homepage has something to show.
  9. allmaps.org's organizations move out of the migrations (914c49b). The two seed migrations now do nothing, so a new database starts without organizations. Their data is in packages/db/seed/allmaps-organizations.sql, which gives the same result as the original migrations, including the display_collections flags a later migration set, and is safe to run again. Databases that already ran the originals keep their hashes, which the history check accepts for these two migrations only. Drizzle picks pending migrations by name, so they don't run again.
  10. The default basemap is a setting (92bb01e). The Protomaps API key only works on allmaps.org, so elsewhere the Editor and Viewer had no basemap. PUBLIC_BASEMAP picks a preset (osm, the Esri ones, or protomaps, the default when unset). The presets move to @allmaps/basemap so both apps share them, and raster presets can set a maximum zoom level (19 for OpenStreetMap, whose tile server stops there).
  11. Only examples this instance has (a278578). The Viewer showed allmaps.org's examples, which 404 elsewhere; it now shows them only on allmaps.org and otherwise recently georeferenced maps from this instance. The Editor hides its collections section when there are no organizations, and no longer requests images for an empty organization list (REST answers 400).
  12. HTTP resources load when the Editor is served over HTTP (cf9a005). It refused every http:// URL, but browsers only block them on HTTPS pages. Local development and HTTP-only deployments need it.
  13. Docker images and a Compose deployment (1c2e8d8). One Dockerfile builds every service on ubuntu:24.04: Node for the APIs and apps, workerd for the Tile Server and Preview, and PostgreSQL with PostGIS. Caddy routes <service>.$DOMAIN and handles TLS four ways: off, Let's Encrypt, your own certificate, or behind a proxy that terminates it (which requires TRUSTED_PROXIES). The SvelteKit apps get their public URL as ORIGIN, since adapter-node otherwise assumes https and SvelteKit rejects form posts from any other origin.
  14. A Compose development mode (411263d). compose.override.yaml runs every service from the mounted source with hot reload. One install service installs and builds at startup and whenever pnpm-lock.yaml changes, and stops the other services while it does, so nothing runs while node_modules changes underneath it. A new database gets the demo seed, with the IIIF fixture server at one hostname per demo organization.
  15. A IIIF image server (6170cee). Cantaloupe, built from its official release, serves the TIFF, JPEG and PNG files in a volume, so an instance can host its own scans.
  16. Documentation (b3dcedd): docker/README.md covers both modes, every setting, TLS, GitHub login, images, organizations, backups and restores.

Needs a decision upstream

  • The rewritten seed migrations (914c49b) are the biggest ask. allmaps.org's database keeps its organizations and migrates as before, but a new allmaps.org environment would need packages/db/seed/allmaps-organizations.sql run by hand. One edge case: a database last migrated between the two seed migrations (26 March to 13 April 2026) would get the organizations but not their domains; running the file fills them in.
  • The Protomaps key. Adding *.allmaps.localhost to its allowed domains would give local development the allmaps.org basemap. Self-hosted instances still need PUBLIC_BASEMAP.

Tested

  • Development: a fresh start, edits reloading in an app, a package, an API and a worker, and a lockfile change triggering a clean reinstall.
  • Deployment: all services through Caddy, an empty database, and Cantaloupe serving JPEG and TIFF that the Tile Server warps.
  • Deployment, following docker/README.md from a fresh database: GitHub sign-in, the first administrator, two organizations created in the Console, scans uploaded with docker compose cp, and an 8400×7246 pyramidal TIFF georeferenced in the Editor with 22 control points. The map saves through Live, survives a reload, and shows in the Viewer with its organization, in the Tile Server and in Preview.
  • Upgrading: a database migrated with upstream's own code and migrations (10 migrations, 47 organizations, 49 domains) gets "No pending migrations" from this branch, with its organizations unchanged, including a history in the older format without migration names. Upstream's own checker rejects the rewritten files, which is what the hash exception is for. A wrong hash, on a seed migration or any other, is still rejected, and each original hash is only accepted for its own migration.
  • Restarts and backups: data and sessions survive docker compose -f compose.yaml down and up. A pg_dump restored into a new database volume (as the README describes) matches the original, and Live carries on from the restored edit history.
  • Other settings: a second deployment with DOMAIN=atlas.localhost and HTTP_PORT=8091, under its own project name, ran alongside the first without touching it. Every service, the sign-in redirect, the cookie domain and ORIGIN followed the settings, and georeferencing worked end to end. Nothing in the source or the built images hardcodes allmaps.localhost or 5490 outside the Compose defaults.
  • GitHub login through the Console, with the session shared across *.allmaps.localhost.
  • test/apis: 144 tests pass. node.test.ts fails inside the dev containers with or without these changes; I haven't looked into it yet.
  • Type checks: no new errors in the Editor, Viewer or the packages touched.

SelectTransformationWrapper copied the map's transformation into local
state once, then wrote that local value back whenever it differed from
the map. When someone else changed the transformation, the map no
longer matched the stale local copy, so the picker changed it straight
back. With two editors on the same map holding different
transformations, each kept undoing the other: in testing, two browser
windows wrote over 6,000 ShareDB ops in 20 seconds until one closed.

Bind the picker to the map itself with a getter and setter, and only
write when the user picks a different transformation.
The domain inputs used bind:value={editDomains[i]}, but editDomains is
a $derived array, which isn't deeply reactive. Typing changed the
array without notifying anything, so the hidden domains field kept its
original value and saving sent the old domains: new domains were never
saved, and edits to existing ones were lost.

Replace the array on input instead, and key the list by position so
editing a domain doesn't recreate its input on every keystroke.
turbo cached and restored dist/ but not tsconfig.tsbuildinfo, tsc's record of
what it compiled. After a restore (switching branches, for example) the record
could describe different source than dist/ held, and the next incremental build
skipped files it believed were current, so turbo cached stale output as
correct. With the record cached alongside dist/, the two can't drift apart.

Existing checkouts may already have such entries: remove .turbo/cache, the
packages' dist/ and tsconfig.tsbuildinfo once, then build.
These are separate pull requests against develop; this branch builds on
them.
Both APIs ran only as Cloudflare Workers. apis/rest/src/node.ts and
apis/annotations/src/node.ts serve the same apps on Node, with a plain
PostgreSQL connection, through serveNode() in @allmaps/api-shared/node.
Live already runs on Node.
They build with adapter-cloudflare. With ADAPTER=node they build with
adapter-node instead, so they run anywhere Node does. Their settings are
already read at runtime, so one build serves every environment.
packages/db/src/migrate.ts works like migrate-ci.ts, but sets up a new
database first (PostGIS and the migration history table), so a fresh install
migrates in one step. The package now publishes its migrations folder.
The Tile Server and Preview read from the cache only with USE_CACHE, but always
wrote to it. workerd without a configured cache rejects the write, so every
response failed.
It deploys them; it isn't needed to run them. As a dependency it put wrangler and workerd (about 200 MB) into every production install.
Two optional settings for running the APIs somewhere other than allmaps.org:

- BETTER_AUTH_COOKIE_DOMAIN shares the session cookie across subdomains
- BETTER_AUTH_TRUSTED_ORIGINS is a comma-separated list of trusted origins,
  wildcards allowed

Without them, behaviour is unchanged.
Regenerated with the wrangler version in pnpm-lock.yaml, so pnpm install and pnpm types no longer leave changes behind.
…zation

- tools/seed/src/postgres.ts seeds a PostgreSQL database once
- The fixture server origins are a parameter, so the demo can be served from
  hostnames other than localhost and 127.0.0.1
- Two more fixture images, one per demo organization, are left unmapped, so
  the Editor's homepage has examples for both
The two seed migrations now do nothing, so a new database starts without
organizations. Their data moves to packages/db/seed/allmaps-organizations.sql,
for allmaps.org to load where it needs them. It gives the same result as the
original migrations, including the display_collections flags that a later
migration set, and skips rows that already exist, so it is safe to run again.

Databases that already ran them keep the original hash in their history, so
the history check accepts it for these two migrations (rewrittenMigrations).
Drizzle selects pending migrations by name, so they don't run again.
PUBLIC_BASEMAP picks the default basemap preset for the Editor and Viewer:
protomaps (when unset), osm, esri-world-topo or esri-world-imagery. The
presets and the raster style move to @allmaps/basemap so both apps share them.
The Protomaps API key only works on allmaps.org.

Raster presets can set a maximum zoom level. OpenStreetMap's tile server
stops at 19; MapLibre scales its tiles up beyond that instead of requesting
tiles that don't exist.
- Viewer: the built-in examples only when they are on the configured
  Annotations API; otherwise recently georeferenced images from REST, and no
  section when there are none
- Editor: no collections section without organizations, and no request for
  organization images when there are none (the REST API rejects it)
Browsers only block HTTP resources on HTTPS pages.
`docker compose -f compose.yaml up --build` runs a self-hosted Allmaps:

- One Dockerfile builds every service from the monorepo, on ubuntu:24.04:
  the APIs and SvelteKit apps on Node, the Tile Server and Preview on workerd
  (docker/workerd-config.sh), and PostgreSQL with PostGIS
- Caddy routes <service>.$DOMAIN to each service, so the whole instance is one
  port, and server-side requests go through it too
- TLS (docker/caddy): off, Let's Encrypt, your own certificate, or a proxy in
  front that terminates it (TRUSTED_PROXIES required)
- Migrations run once at startup
- The Editor, Viewer and Console get their public URL as ORIGIN: adapter-node
  otherwise assumes https, and SvelteKit rejects form posts whose origin
  doesn't match
`docker compose up` applies compose.override.yaml: every service runs from
the mounted source and reloads on edit (Vite, node --watch, wrangler dev, and
`pnpm watch` rebuilding packages).

- docker/dev-install.sh installs and builds at startup and whenever
  pnpm-lock.yaml changes; docker/dev-run.sh stops each service during a
  reinstall and starts it again afterwards, so nothing runs while
  node_modules changes
- A new database gets the demo seed, with the IIIF fixture server at one
  hostname per demo organization (docker/caddy-dev)
- Development data has its own database volume
Cantaloupe, built from its official release on ubuntu:24.04, serves every
TIFF, JPEG and PNG file in the iiif-images volume at iiif.$DOMAIN, so an
instance can host its own scans.
docker/README.md: both modes, the demo data, every setting, TLS, GitHub login, adding images and organizations, backups and restores, and the files involved.
@nickmarden nickmarden changed the title WIP: One-command local development and self-hosting with Docker Compose One-command local development and self-hosting with Docker Compose Oct 23, 2026
@nickmarden
nickmarden marked this pull request as ready for review October 23, 2026 16:33
@nickmarden
nickmarden force-pushed the lineage/docker-compose branch from 5693608 to b3dcedd Compare October 23, 2026 16:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant