Repository navigation
One-command local development and self-hosting with Docker Compose - #1
Open
nickmarden wants to merge 20 commits into
Open
nickmarden wants to merge 20 commits into
nickmarden wants to merge 20 commits into
Conversation
10 tasks
SelectTransformationWrapper copied the map's transformation into local state once, then wrote that local value back whenever it differed from the map. When someone else changed the transformation, the map no longer matched the stale local copy, so the picker changed it straight back. With two editors on the same map holding different transformations, each kept undoing the other: in testing, two browser windows wrote over 6,000 ShareDB ops in 20 seconds until one closed. Bind the picker to the map itself with a getter and setter, and only write when the user picks a different transformation.
The domain inputs used bind:value={editDomains[i]}, but editDomains is
a $derived array, which isn't deeply reactive. Typing changed the
array without notifying anything, so the hidden domains field kept its
original value and saving sent the old domains: new domains were never
saved, and edits to existing ones were lost.
Replace the array on input instead, and key the list by position so
editing a domain doesn't recreate its input on every keystroke.
turbo cached and restored dist/ but not tsconfig.tsbuildinfo, tsc's record of what it compiled. After a restore (switching branches, for example) the record could describe different source than dist/ held, and the next incremental build skipped files it believed were current, so turbo cached stale output as correct. With the record cached alongside dist/, the two can't drift apart. Existing checkouts may already have such entries: remove .turbo/cache, the packages' dist/ and tsconfig.tsbuildinfo once, then build.
These are separate pull requests against develop; this branch builds on them.
Both APIs ran only as Cloudflare Workers. apis/rest/src/node.ts and apis/annotations/src/node.ts serve the same apps on Node, with a plain PostgreSQL connection, through serveNode() in @allmaps/api-shared/node. Live already runs on Node.
They build with adapter-cloudflare. With ADAPTER=node they build with adapter-node instead, so they run anywhere Node does. Their settings are already read at runtime, so one build serves every environment.
packages/db/src/migrate.ts works like migrate-ci.ts, but sets up a new database first (PostGIS and the migration history table), so a fresh install migrates in one step. The package now publishes its migrations folder.
The Tile Server and Preview read from the cache only with USE_CACHE, but always wrote to it. workerd without a configured cache rejects the write, so every response failed.
It deploys them; it isn't needed to run them. As a dependency it put wrangler and workerd (about 200 MB) into every production install.
Two optional settings for running the APIs somewhere other than allmaps.org: - BETTER_AUTH_COOKIE_DOMAIN shares the session cookie across subdomains - BETTER_AUTH_TRUSTED_ORIGINS is a comma-separated list of trusted origins, wildcards allowed Without them, behaviour is unchanged.
Regenerated with the wrangler version in pnpm-lock.yaml, so pnpm install and pnpm types no longer leave changes behind.
…zation - tools/seed/src/postgres.ts seeds a PostgreSQL database once - The fixture server origins are a parameter, so the demo can be served from hostnames other than localhost and 127.0.0.1 - Two more fixture images, one per demo organization, are left unmapped, so the Editor's homepage has examples for both
The two seed migrations now do nothing, so a new database starts without organizations. Their data moves to packages/db/seed/allmaps-organizations.sql, for allmaps.org to load where it needs them. It gives the same result as the original migrations, including the display_collections flags that a later migration set, and skips rows that already exist, so it is safe to run again. Databases that already ran them keep the original hash in their history, so the history check accepts it for these two migrations (rewrittenMigrations). Drizzle selects pending migrations by name, so they don't run again.
PUBLIC_BASEMAP picks the default basemap preset for the Editor and Viewer: protomaps (when unset), osm, esri-world-topo or esri-world-imagery. The presets and the raster style move to @allmaps/basemap so both apps share them. The Protomaps API key only works on allmaps.org. Raster presets can set a maximum zoom level. OpenStreetMap's tile server stops at 19; MapLibre scales its tiles up beyond that instead of requesting tiles that don't exist.
- Viewer: the built-in examples only when they are on the configured Annotations API; otherwise recently georeferenced images from REST, and no section when there are none - Editor: no collections section without organizations, and no request for organization images when there are none (the REST API rejects it)
Browsers only block HTTP resources on HTTPS pages.
`docker compose -f compose.yaml up --build` runs a self-hosted Allmaps: - One Dockerfile builds every service from the monorepo, on ubuntu:24.04: the APIs and SvelteKit apps on Node, the Tile Server and Preview on workerd (docker/workerd-config.sh), and PostgreSQL with PostGIS - Caddy routes <service>.$DOMAIN to each service, so the whole instance is one port, and server-side requests go through it too - TLS (docker/caddy): off, Let's Encrypt, your own certificate, or a proxy in front that terminates it (TRUSTED_PROXIES required) - Migrations run once at startup - The Editor, Viewer and Console get their public URL as ORIGIN: adapter-node otherwise assumes https, and SvelteKit rejects form posts whose origin doesn't match
`docker compose up` applies compose.override.yaml: every service runs from the mounted source and reloads on edit (Vite, node --watch, wrangler dev, and `pnpm watch` rebuilding packages). - docker/dev-install.sh installs and builds at startup and whenever pnpm-lock.yaml changes; docker/dev-run.sh stops each service during a reinstall and starts it again afterwards, so nothing runs while node_modules changes - A new database gets the demo seed, with the IIIF fixture server at one hostname per demo organization (docker/caddy-dev) - Development data has its own database volume
Cantaloupe, built from its official release on ubuntu:24.04, serves every TIFF, JPEG and PNG file in the iiif-images volume at iiif.$DOMAIN, so an instance can host its own scans.
docker/README.md: both modes, the demo data, every setting, TLS, GitHub login, adding images and organizations, backups and restores, and the files involved.
nickmarden
marked this pull request as ready for review
October 23, 2026 16:33
nickmarden
force-pushed
the
lineage/docker-compose
branch
from
October 23, 2026 16:46
5693608 to
b3dcedd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Work in progress. This is the self-hosting work from allmaps#672, against upstream
developatf3633bf.It builds on three small fixes for bugs found while testing it. Each is its own pull request against
develop, and this branch merges them first (ae3411f):$derivedarray in place, so the form always sent the old domains.tsc's build info with each package's build output: Cache tsc's build info with each package's build output allmaps/allmaps#684 (bf25840). turbo restoreddist/withouttsconfig.tsbuildinfo, so after switching branches an incremental build could skip changed files and turbo cached the stale output.What and why
Two things that are hard today:
wasm-packon your machine, and a local API runner that only talks tolocalhost.allmaps.org, and a new database comes with allmaps.org's partner organizations in it.This PR makes both one command:
Every service gets a hostname under one domain behind a Caddy proxy (
http://editor.allmaps.localhost:5490and so on locally), so there is one port to remember, and the same layout works on a server with TLS.docker/README.mdcovers both modes and every setting.Those local addresses are the same for everyone, and that's on purpose. On your own machine, every developer gets these:
https://editor.$DOMAINhttps://viewer.$DOMAINhttps://console.$DOMAINhttps://api.$DOMAINhttps://annotations.$DOMAINhttps://live.$DOMAINhttps://tiles.$DOMAINhttps://preview.$DOMAINhttps://iiif.$DOMAINThe fixture hostnames only exist in development mode. Both are the same fixture server; the demo organizations are told apart by domain, so each gets its own. The server column assumes TLS; with
TLS=offit'shttp://plus:$HTTP_PORT. So, locally:http://api.allmaps.localhost:5490/auth/callback/github) instead of "whatever your ports are".*.allmaps.localhostto the Protomaps key would cover every local install.*.allmaps.localhost, so logging in once works for the Console, the Editor and the APIs.*.localhostto your own machine without touching/etc/hosts(as do macOS and mainstream Linux for other tools).If port 5490 is taken, or you'd like different hostnames, set
HTTP_PORTorDOMAIN. Most people won't need to.Nothing changes for allmaps.org unless it opts in: every new setting defaults to today's behaviour, and the Cloudflare builds and deploy scripts are untouched.
Commits
serveNode()in@allmaps/api-shared/nodeserves the same apps on Node with plain PostgreSQL. Live already ran on Node.ADAPTER=nodebuilds the Editor, Viewer and Console withadapter-node(8a18091). They already read their settings at runtime, so one image serves every environment.migrate.tsworks likemigrate-ci.ts, but creates PostGIS and the migration history table first, so a fresh install migrates in one step.USE_CACHEis on (3d8c1ff). They already only read with it on.workerdwithout a cache rejects the write, so every response failed.wrangleris a devDependency of the REST and Annotations APIs (248adcc). It deploys them; as a dependency it added about 200 MB (wrangler and workerd) to every production install.BETTER_AUTH_COOKIE_DOMAINandBETTER_AUTH_TRUSTED_ORIGINS. Both were hardcoded toallmaps.organdlocalhost. Unset, behaviour is unchanged.worker-configuration.d.tsfiles predate the wrangler version in the lockfile, so everypnpm installorpnpm typesrewrote them.tools/seedonly ran on PGlite.tools/seed/src/postgres.tsseeds a PostgreSQL database once; the fixture server's origins are now a parameter, and each demo organization gets an unmapped image so the Editor's homepage has something to show.packages/db/seed/allmaps-organizations.sql, which gives the same result as the original migrations, including thedisplay_collectionsflags a later migration set, and is safe to run again. Databases that already ran the originals keep their hashes, which the history check accepts for these two migrations only. Drizzle picks pending migrations by name, so they don't run again.PUBLIC_BASEMAPpicks a preset (osm, the Esri ones, orprotomaps, the default when unset). The presets move to@allmaps/basemapso both apps share them, and raster presets can set a maximum zoom level (19 for OpenStreetMap, whose tile server stops there).http://URL, but browsers only block them on HTTPS pages. Local development and HTTP-only deployments need it.ubuntu:24.04: Node for the APIs and apps,workerdfor the Tile Server and Preview, and PostgreSQL with PostGIS. Caddy routes<service>.$DOMAINand handles TLS four ways: off, Let's Encrypt, your own certificate, or behind a proxy that terminates it (which requiresTRUSTED_PROXIES). The SvelteKit apps get their public URL asORIGIN, since adapter-node otherwise assumes https and SvelteKit rejects form posts from any other origin.compose.override.yamlruns every service from the mounted source with hot reload. Oneinstallservice installs and builds at startup and wheneverpnpm-lock.yamlchanges, and stops the other services while it does, so nothing runs whilenode_moduleschanges underneath it. A new database gets the demo seed, with the IIIF fixture server at one hostname per demo organization.docker/README.mdcovers both modes, every setting, TLS, GitHub login, images, organizations, backups and restores.Needs a decision upstream
packages/db/seed/allmaps-organizations.sqlrun by hand. One edge case: a database last migrated between the two seed migrations (26 March to 13 April 2026) would get the organizations but not their domains; running the file fills them in.*.allmaps.localhostto its allowed domains would give local development the allmaps.org basemap. Self-hosted instances still needPUBLIC_BASEMAP.Tested
docker/README.mdfrom a fresh database: GitHub sign-in, the first administrator, two organizations created in the Console, scans uploaded withdocker compose cp, and an 8400×7246 pyramidal TIFF georeferenced in the Editor with 22 control points. The map saves through Live, survives a reload, and shows in the Viewer with its organization, in the Tile Server and in Preview.docker compose -f compose.yaml downandup. Apg_dumprestored into a new database volume (as the README describes) matches the original, and Live carries on from the restored edit history.DOMAIN=atlas.localhostandHTTP_PORT=8091, under its own project name, ran alongside the first without touching it. Every service, the sign-in redirect, the cookie domain andORIGINfollowed the settings, and georeferencing worked end to end. Nothing in the source or the built images hardcodesallmaps.localhostor 5490 outside the Compose defaults.*.allmaps.localhost.test/apis: 144 tests pass.node.test.tsfails inside the dev containers with or without these changes; I haven't looked into it yet.