Arch-Orchestrator

A production-grade, containerized deployment template for high-concurrency multiplayer virtual environments.

Zero-downtime · Resource-bounded · Self-healing · Infrastructure-as-Code

1. Executive Summary

Arch-Orchestrator is an opinionated reference deployment for the back-end of a real-time multiplayer service. It packages a 3-tier stack — persistence, cache/messaging, and application core — into an immutable, declarative artefact suitable for staging, production, or air-gapped environments.

The repository is intentionally narrow in scope: it does not ship game logic. It ships the plumbing — the orchestration topology, the hardened service configuration, the operational scripts, and the JVM scaffolding — that a Systems Architect or Platform Engineer needs to land a multiplayer runtime on Linux hosts with the rest of the team focused on gameplay code.

Design pillars

Pillar	How it's enforced
Separation of Concerns	One directory per responsibility (`/deploy`, `/configs`, `/core`, `/scripts`).
Infrastructure as Code	Every host-affecting parameter lives in `docker-compose.yml`, `*.cnf`, or `.env`. No clickops.
Network isolation	Two bridge networks; the data plane is `internal: true` and unreachable from outside the host.
Resource ceilings	Every container has explicit CPU/RAM `limits` and `reservations`.
Observable startup	Health checks gate `depends_on`, so the core only boots once DB+cache report healthy.
Twelve-factor config	All secrets and tunables are environment variables sourced from `deploy/.env`.
Defensive scripting	Bash entry points use `set -Eeuo pipefail`, `trap ERR`, structured logging, retention policies.

2. System Architecture

2.1 High-level topology

flowchart LR
    subgraph host["Linux Host"]
        direction LR
        subgraph edge["archo_edge (bridge, public)"]
            CORE["core<br/>JVM 21 + virtual threads<br/>:8080 (HTTP) :7777 (TCP/UDP)"]
        end
        subgraph data["archo_dataplane (bridge, internal: true)"]
            DB[("MariaDB 11.4<br/>InnoDB · pool-of-threads")]
            CACHE[("Redis 7.4<br/>AOF + RDB · Pub/Sub")]
        end
        CORE -- "JDBC<br/>HikariCP pool" --> DB
        CORE -- "RESP3<br/>Lettuce client" --> CACHE
    end

    Clients[/"Game clients<br/>web / mobile / native"/] -- "TCP/UDP/HTTPS" --> CORE
    Ops[/"Operator<br/>scripts/*.sh"/] -. "docker exec / stats" .-> host

    classDef storage fill:#0b3d2e,stroke:#0f6,color:#fff;
    classDef app     fill:#103a5f,stroke:#5af,color:#fff;
    class DB,CACHE storage;
    class CORE app;

Why two networks? The application core is the only component on the public-facing bridge. MariaDB and Redis sit on a network with internal: true — Docker will refuse to give it a default gateway, so even a compromised host-side process can't dial the DB without going through the orchestrator.

2.2 Boot sequence (gated by health checks)

sequenceDiagram
    autonumber
    participant Op as Operator
    participant DC as Docker Compose
    participant DB as MariaDB
    participant RC as Redis
    participant CR as Core (JVM)

    Op->>DC: docker compose up -d
    DC->>DB: create container, mount /configs/mariadb/server.cnf
    DC->>RC: create container, mount /configs/redis/redis.conf
    par
        DB-->>DC: healthcheck (mariadbupgrade & innodb_initialized)
    and
        RC-->>DC: healthcheck (redis-cli ping → PONG)
    end
    DC->>CR: start once DB.healthy && Redis.healthy
    CR->>DB: HikariCP warm-up (min idle = DB_POOL_MIN)
    CR->>RC: Lettuce connect
    CR-->>DC: HTTP /health → 200 OK
    DC-->>Op: stack READY

2.3 Repository layout

arch-orchestrator/
├── deploy/                 # IaC entry point
│   ├── docker-compose.yml  # Compose Spec — services, networks, limits, healthchecks
│   └── .env.example        # Single source of truth for secrets & tunables
├── configs/                # Hardened service configuration (mounted read-only)
│   ├── mariadb/
│   │   └── server.cnf      # InnoDB tuning, thread-pool, slow-query log, binlog
│   └── redis/
│       └── redis.conf      # AOF+RDB, lazyfree, I/O threads, dangerous cmds disabled
├── core/                   # JVM application scaffolding
│   ├── Dockerfile          # Multi-stage: gradle:8 → temurin JRE (non-root, tini)
│   ├── build.gradle.kts    # Kotlin DSL · Shadow plugin · JDK 21 toolchain
│   ├── settings.gradle.kts
│   └── src/main/java/com/archorchestrator/core/
│       ├── Application.java        # Boot orchestration + graceful shutdown
│       ├── AppConfig.java          # 12-factor env loader
│       ├── DataSourceFactory.java  # HikariCP + MariaDB JDBC
│       ├── RedisClientFactory.java # Lettuce non-blocking client
│       └── HealthServer.java       # /health /readyz /metrics on virtual threads
├── scripts/                # Operational automation
│   ├── _lib.sh             # Shared logger / strict-mode / compose detection
│   ├── setup.sh            # Bootstrap, secret generation, build/pull
│   ├── backup.sh           # Hot-backup (mariadb-dump --single-transaction)
│   └── monitor.sh          # Health, stats, readiness, logs (one-shot or watch)
├── Makefile                # Convenience targets that wrap the scripts
└── README.md

3. Feature Matrix

3.1 Persistence — MariaDB 11.4 LTS

InnoDB tuned for OLTP: buffer pool sized to ~70 % of container RAM, four buffer-pool instances, innodb_io_capacity=2000, O_DIRECT flush method.
Thread-pool of threads instead of one-thread-per-connection — graceful behaviour at 5 000+ concurrent sessions.
Row-based binary logging with minimal images, ready for replicas or PITR.
Slow query log enabled at the 1 s threshold for production diagnostics.
Strict SQL mode, local_infile=OFF, secure_file_priv quarantined.
Healthcheck uses upstream healthcheck.sh from the official image.

3.2 Cache / Messaging — Redis 7.4

Hybrid persistence: AOF (everysec) + periodic RDB snapshots — durable enough for sessions, fast enough for cold restarts.
I/O threads enabled (io-threads 4, io-threads-do-reads yes) for NIC- bound workloads.
Lazy-free everywhere — eviction, expire, server-del, user-del, user-flush.
Hardening: FLUSHALL, DEBUG, FLUSHDB renamed to ""; CONFIG and SHUTDOWN namespaced behind ARCHO_* prefixes.
notify-keyspace-events "KEAxe" — out-of-the-box hooks for session expiry.
Eviction policy and memory ceiling injected via .env, not the conf file.

3.3 Application — JVM 21 core

Java 21 with virtual threads powering the management HTTP server.
HikariCP sized via env (DB_POOL_MIN, DB_POOL_MAX) with leak detection.
Lettuce as the Redis client — non-blocking, virtual-thread friendly.
Built-in /health, /readyz, /metrics endpoints (Prometheus exposition).
JVM flags: +UseZGC +ZGenerational +UseContainerSupport MaxRAMPercentage=75 +ExitOnOutOfMemoryError.
Multi-stage image under 250 MB: Gradle builder → Temurin JRE, non-root UID 10001, tini as PID 1, HEALTHCHECK baked into the image.

3.4 Orchestration

Compose Spec v3.9 — works with Compose V2 and Swarm without modification.
Per-service deploy.resources.limits + reservations (CPU and memory).
depends_on with condition: service_healthy for both data services.
restart: unless-stopped everywhere, stop_grace_period: 30s on the core.
JSON-file logging driver with rotation (max-size: 10m, max-file: 5, compressed) — bounded disk usage even with chatty applications.
security_opt: no-new-privileges on every container.
ulimits.nofile = 65536 — no surprise EMFILE under heavy fan-out.

3.5 Operations

One-shot setup.sh generates secrets, materialises .env, pulls/builds.
Cron-friendly backup.sh with retention rotation and partial-file safety.
Live monitor.sh dashboard (containers / health / stats / logs / probes).
Every script: strict mode, trap ERR, RFC 3339 timestamps, ANSI-aware logger.

4. Prerequisites

Component	Minimum	Notes
Linux kernel	5.10+	cgroups v2 strongly recommended
Docker Engine	24.0+	needed for Compose Spec v3.9 + BuildKit cache mounts
Docker Compose	v2.20+	Compose V2 plugin; `docker-compose` legacy works too
RAM	8 GiB	default ceilings = 4 GiB (core) + 2 GiB (db) + 1 GiB (cache)
CPU	4 vCPU	default ceilings sum to 7 vCPU; over-subscription is intentional
Disk	20 GiB	InnoDB + binlogs + AOF + backups
Bash	4.4+	scripts use `set -Eeuo pipefail` and arrays

5. Quick Start

git clone https://github.com/your-org/arch-orchestrator.git
cd arch-orchestrator

# 1. Bootstrap: verifies prerequisites, generates secrets, pulls + builds.
./scripts/setup.sh

# 2. Inspect deploy/.env and rotate any value you don't trust.
$EDITOR deploy/.env

# 3. Bring the stack up (detached).
./scripts/setup.sh --start
# or, equivalently:
make up

# 4. Watch the dashboard.
./scripts/monitor.sh --watch 5

# 5. Take a hot backup.
./scripts/backup.sh

# 6. Tear down (preserves volumes).
make down

The application core publishes:

Port	Protocol	Purpose
`8080`	HTTP	management surface (`/health`, `/readyz`, `/metrics`)
`7777`	TCP	game server primary listener
`7777`	UDP	low-latency real-time channel (movement, voice, telemetry)

6. Configuration Reference

All values are defined in deploy/.env.example and overridden in deploy/.env. The compose file fails fast (? operator) when a required secret is missing, so unset variables are surfaced immediately instead of silently defaulting to "".

6.1 Core / JVM

Variable	Default	Description
`APP_ENV`	`production`	Logical environment label.
`APP_HTTP_PORT`	`8080`	Management surface port (host-side).
`APP_GAME_PORT`	`7777`	Game listener port (TCP+UDP).
`JVM_MAX_RAM_PCT`	`75.0`	Heap ceiling as % of container memory.
`JVM_INIT_RAM_PCT`	`50.0`	Initial heap as % of container memory.
`CORE_CPU_LIMIT`	`4.0`	Hard CPU ceiling (cgroup quota).
`CORE_MEM_LIMIT`	`4g`	Hard RAM ceiling.

6.2 MariaDB

Variable	Default	Description
`MARIADB_ROOT_PASSWORD`	(required)	Root password — never bake into images.
`MARIADB_DATABASE`	`archo`	Database created on first boot.
`MARIADB_USER`	`archo`	Application user (granted on `MARIADB_DATABASE`).
`MARIADB_PASSWORD`	(required)	Application user password.
`MARIADB_CPU_LIMIT` / `_MEM_LIMIT`	`2.0` / `2g`	cgroup ceilings.
`DB_POOL_MIN` / `DB_POOL_MAX`	`10` / `50`	HikariCP sizing on the core.

6.3 Redis

Variable	Default	Description
`REDIS_PASSWORD`	(required)	Used both server-side (`--requirepass`) and by clients.
`REDIS_MAXMEMORY`	`768mb`	Hard ceiling enforced inside Redis (separate from cgroup limit).
`REDIS_EVICTION_POLICY`	`allkeys-lru`	Use `volatile-lru` if you want only TTL-bearing keys evicted.

6.4 Backups

Variable	Default	Description
`BACKUP_DIRECTORY`	`./backups`	Host-side dump destination.
`BACKUP_RETENTION_DAYS`	`14`	Older dumps are pruned by `backup.sh`.

7. Operations Runbook

7.1 Lifecycle

make up         # docker compose up -d
make down       # docker compose down (volumes preserved)
make restart    # rolling restart of all services
make logs       # tail -f for every container
make ps         # docker compose ps
make build      # rebuild local images, pulling base layers

7.2 Backup & restore

Backups are written to ./backups/archo-<db>-<UTC-timestamp>.sql.gz.

# Hot backup (no write lock thanks to --single-transaction)
./scripts/backup.sh

# Restore (drop+recreate the schema first if needed)
gunzip -c backups/archo-archo-20260101T020000Z.sql.gz \
  | docker exec -i archo-mariadb mariadb -uroot -p"$MARIADB_ROOT_PASSWORD"

Schedule via cron (UTC, daily 02:30):

30 2 * * *  cd /opt/arch-orchestrator && ./scripts/backup.sh >> /var/log/archo-backup.log 2>&1

7.3 Monitoring

./scripts/monitor.sh              # one-shot snapshot
./scripts/monitor.sh --watch 5    # refreshing dashboard, 5 s interval
./scripts/monitor.sh --service core --tail 50
curl -s http://127.0.0.1:8080/metrics

7.4 Zero-downtime image updates

The compose topology is intentionally simple — each container is replaceable. For a true rolling update of the core service:

# Build the new image.
make build

# Replace the container with health-gated rollout.
docker compose --env-file deploy/.env -f deploy/docker-compose.yml \
    up -d --no-deps --build --force-recreate core

When you outgrow a single host, the same compose file is deploy-compatible with Docker Swarm (docker stack deploy -c docker-compose.yml archo) because every limit lives under deploy.resources.

8. Performance Tuning Cookbook

8.1 MariaDB

Knob	Heuristic	Where
`innodb_buffer_pool_size`	~70 % of `MARIADB_MEM_LIMIT`	`configs/mariadb/server.cnf`
`innodb_buffer_pool_instances`	1 per ~1 GiB of buffer pool	same
`innodb_io_capacity`	2 000 (SSD) — 20 000 (NVMe)	same
`thread_pool_size`	= number of vCPU	same
`max_connections`	1.5× `DB_POOL_MAX` × number of cores	same
`wait_timeout`	600 s for sticky game sessions	same

8.2 Redis

Run redis-cli --latency-history -h 127.0.0.1 -p 6379 against a stable baseline; aim for p99 < 1 ms inside the container.
If your hot keyspace fits in RAM, set REDIS_EVICTION_POLICY=noeviction and size REDIS_MAXMEMORY to ~80 % of the cgroup limit — surprises become OOMs instead of silent evictions.
For Pub/Sub fan-out > 10 K subscribers, raise client-output-buffer-limit pubsub and watch INFO clients > total_blocking_clients.

8.3 JVM

The compose file already enables ZGC + generational — best-in-class for multiplayer back-ends with low latency requirements (sub-ms pause goal).
Profile with -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinal once, capture the result into your runbook.
Virtual threads make the correct concurrency story per-request; avoid pinning by never holding monitor locks across blocking I/O. Use ReentrantLock if you need critical sections that span DB calls.
If you observe heap saturation, tune JVM_MAX_RAM_PCT down and let the container OOM rather than thrashing — +ExitOnOutOfMemoryError is set, so the orchestrator restart policy will recover within seconds.

8.4 Host kernel

For load tests on Linux hosts (do not apply blindly to small VPS instances):

# Increase ephemeral port range and TIME_WAIT reuse
sysctl -w net.ipv4.ip_local_port_range="10240 65535"
sysctl -w net.ipv4.tcp_tw_reuse=1

# Larger conntrack table (game servers chew through entries)
sysctl -w net.netfilter.nf_conntrack_max=524288

# File descriptors
ulimit -n 1048576

9. Security Posture

No public DB. MariaDB and Redis are bound to an internal: true bridge — they have no route to the internet.
Secrets in .env only. setup.sh generates them with openssl rand, writes the file chmod 600, and the compose file errors out if they're missing. .env itself is .gitignored.
Non-root containers. The core image runs as UID 10001; MariaDB and Redis drop to their image-default unprivileged users.
security_opt: no-new-privileges on every container — prevents setuid/setgid escalation paths inside the namespace.
Dangerous Redis commands renamed. FLUSHALL, FLUSHDB, DEBUG are effectively disabled; CONFIG and SHUTDOWN are namespaced.
MariaDB hardening. local_infile=OFF, secure_file_priv quarantined, strict SQL mode.
TLS termination is intentionally out of scope — terminate at your edge (Caddy / Nginx / Traefik / Cloud LB) and forward to 127.0.0.1:8080.

10. Troubleshooting

Symptom	Likely cause	Fix
`core` is `unhealthy`, others healthy	DB or Redis password mismatch	Re-run `./scripts/setup.sh --force` to regenerate `.env`.
`mariadb` flaps with `InnoDB: Cannot allocate memory`	`MARIADB_MEM_LIMIT` < `innodb_buffer_pool_size`	Raise the cgroup limit or lower `innodb_buffer_pool_size` in `configs/mariadb/server.cnf`.
Compose: "network has active endpoints" on `down`	Containers from a previous run still attached	`docker network rm archo_dataplane archo_edge` after stopping.
Healthcheck reports `redis-cli: NOAUTH`	Stale `REDIS_PASSWORD` env in core but Redis is using a new value	`make restart` so both pick up the same `.env`.
Backups grow without bound	`BACKUP_RETENTION_DAYS` set, but cron job not running	Verify cron, or call `backup.sh` with `--retention 7`.
`core` startup blocked > 60 s	DB still applying `mariadb-upgrade` after a major version bump	Patience — the start-period is 60 s; the orchestrator will eventually mark it healthy.

11. Roadmap

Optional docker-compose.observability.yml overlay (Prometheus + Grafana + Loki).
First-class MariaDB Galera profile for multi-writer HA.
Redis Sentinel profile for cache failover.
Helm chart equivalent for Kubernetes targets.
mariabackup (physical, point-in-time) profile alongside the existing logical dump.

12. Project Conventions

Bash: every script begins with set -Eeuo pipefail, sources _lib.sh, uses log / log_warn / log_error, and exits with documented codes (64=usage, 66=missing input, 69=service unavailable, 70=software, 74=I/O, 75=temp-fail).
YAML: 2-space indent, anchors for cross-cutting concerns (x-logging, x-restart).
Java: JDK 21, virtual threads for I/O, record for immutable config, no Lombok, no reflective magic — keep the audit trail short.
Compose: every service has restart, logging, healthcheck, deploy.resources.limits, deploy.resources.reservations.

13. License

Released under the MIT License. Use it, fork it, ship it.

Navigation Menu

Repository files navigation

Arch-Orchestrator

1. Executive Summary

Design pillars

2. System Architecture

2.1 High-level topology

2.2 Boot sequence (gated by health checks)

2.3 Repository layout

3. Feature Matrix

3.1 Persistence — MariaDB 11.4 LTS

3.2 Cache / Messaging — Redis 7.4

3.3 Application — JVM 21 core

3.4 Orchestration

3.5 Operations

4. Prerequisites

5. Quick Start

6. Configuration Reference

6.1 Core / JVM

6.2 MariaDB

6.3 Redis

6.4 Backups

7. Operations Runbook

7.1 Lifecycle

7.2 Backup & restore

7.3 Monitoring

7.4 Zero-downtime image updates

8. Performance Tuning Cookbook

8.1 MariaDB

8.2 Redis

8.3 JVM

8.4 Host kernel

9. Security Posture

10. Troubleshooting

11. Roadmap

12. Project Conventions

13. License

About

Contributors

Languages

Name		Name	Last commit message	Last commit date
configs		configs
core		core
deploy		deploy
scripts		scripts
.dockerignore		.dockerignore
.gitignore		.gitignore
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md

Latest commit

History

Folders and files

Repository files navigation

Arch-Orchestrator

1. Executive Summary

Design pillars

2. System Architecture

2.1 High-level topology

2.2 Boot sequence (gated by health checks)

2.3 Repository layout

3. Feature Matrix

3.1 Persistence — MariaDB 11.4 LTS

3.2 Cache / Messaging — Redis 7.4

3.3 Application — JVM 21 core

3.4 Orchestration

3.5 Operations

4. Prerequisites

5. Quick Start

6. Configuration Reference

6.1 Core / JVM

6.2 MariaDB

6.3 Redis

6.4 Backups

7. Operations Runbook

7.1 Lifecycle

7.2 Backup & restore

7.3 Monitoring

7.4 Zero-downtime image updates

8. Performance Tuning Cookbook

8.1 MariaDB

8.2 Redis

8.3 JVM

8.4 Host kernel

9. Security Posture

10. Troubleshooting

11. Roadmap

12. Project Conventions

13. License

About

Topics

Resources

Stars

Watchers

Forks

Contributors

Languages