From 5dfeb48b3466da9a19a4c238b0606e645c5bdd69 Mon Sep 17 00:00:00 2001 From: MasterPtato Date: Thu, 25 Jun 2026 16:04:12 -0700 Subject: [PATCH] [slopfix] docs(self-hosting): promote postgres from experimental to recommended OSS multi-node backend --- .../src/content/cookbook/vpc-air-gapped.mdx | 4 +- .../docs/self-hosting/configuration.mdx | 4 +- .../docs/self-hosting/foundationdb.mdx | 2 +- .../content/docs/self-hosting/postgres.mdx | 57 +++++++++++++++++++ .../self-hosting/production-checklist.mdx | 12 ++-- 5 files changed, 70 insertions(+), 9 deletions(-) diff --git a/website/src/content/cookbook/vpc-air-gapped.mdx b/website/src/content/cookbook/vpc-air-gapped.mdx index 843aba522d..db26b1e0ed 100644 --- a/website/src/content/cookbook/vpc-air-gapped.mdx +++ b/website/src/content/cookbook/vpc-air-gapped.mdx @@ -107,10 +107,10 @@ If you ship software that runs inside your customers' VPCs, the same setup turns | Backend | Use when | Status | | --- | --- | --- | | [File System](/docs/self-hosting/filesystem) (RocksDB-based) | Single-node deployments, including air-gapped installs | Production-ready, single node only | -| [PostgreSQL](/docs/self-hosting/postgres) | Multi-node deployments | Recommended for multi-node today, but experimental | +| [PostgreSQL](/docs/self-hosting/postgres) | Multi-node and multi-region deployments | Production-ready for multi-node | | FoundationDB | Largest production deployments | [Enterprise](/sales) | -For multi-node deployments, run two or more engine nodes behind a load balancer and add NATS for pub/sub, which replaces the default PostgreSQL `LISTEN`/`NOTIFY` path at high throughput. Neither is needed for a single-node file system install. See the [Production Checklist](/docs/self-hosting/production-checklist). +For multi-node deployments, run two or more engine nodes behind a load balancer, all sharing one PostgreSQL instance. The built-in PostgreSQL pub/sub is sufficient for most deployments; very high-throughput deployments can add NATS as a dedicated pub/sub layer. Neither is needed for a single-node file system install. See the [Production Checklist](/docs/self-hosting/production-checklist). ## Perimeter Checklist diff --git a/website/src/content/docs/self-hosting/configuration.mdx b/website/src/content/docs/self-hosting/configuration.mdx index 781f8bba6a..8fb19f129c 100644 --- a/website/src/content/docs/self-hosting/configuration.mdx +++ b/website/src/content/docs/self-hosting/configuration.mdx @@ -76,5 +76,5 @@ Use `samples: 1` for a uniform random pick that skips slot reads. Use `samples > ## Related - RivetKit actor runtime persistence lives in SQLite. Existing actor KV data is imported into SQLite the first time an actor wakes on the migrated runtime, then the original KV data is left frozen for downgrade safety. -- [PostgreSQL](/docs/self-hosting/postgres): Configure the PostgreSQL backend for multi-node deployments -- [File System](/docs/self-hosting/filesystem): Configure file system storage for development +- [PostgreSQL](/docs/self-hosting/postgres): Configure the PostgreSQL backend for multi-node and multi-region deployments +- [File System](/docs/self-hosting/filesystem): Configure file system storage for single-node deployments diff --git a/website/src/content/docs/self-hosting/foundationdb.mdx b/website/src/content/docs/self-hosting/foundationdb.mdx index bce99b66dd..5de8408f51 100644 --- a/website/src/content/docs/self-hosting/foundationdb.mdx +++ b/website/src/content/docs/self-hosting/foundationdb.mdx @@ -25,7 +25,7 @@ Its strict serializability guarantees, fault tolerance, and ability to scale lin | | RocksDB (File System) | PostgreSQL | FoundationDB | |---|---|---|---| -| **Scalability** | Single node | Primary/replica failover | Linear horizontal scaling | +| **Scalability** | Single node | Multi-node and multi-region | Linear horizontal scaling | | **Fault tolerance** | None | Primary/replica failover | Automatic recovery with no data loss | | **Production readiness** | Development and small deployments | Production-ready for light-to-moderate multi-node workloads | Battle-tested at global scale | diff --git a/website/src/content/docs/self-hosting/postgres.mdx b/website/src/content/docs/self-hosting/postgres.mdx index 34b2fc5809..ef737738fc 100644 --- a/website/src/content/docs/self-hosting/postgres.mdx +++ b/website/src/content/docs/self-hosting/postgres.mdx @@ -8,6 +8,16 @@ skill: true PostgreSQL is the recommended backend for multi-node self-hosted deployments. It is production-ready for light-to-moderate workloads, up to roughly 1,000 concurrent actors, but is not built for enterprise scale beyond that. For a single-node deployment, use the file system backend (RocksDB-based). Teams running larger or high-throughput realtime workloads should contact [enterprise support](https://rivet.dev/sales) about FoundationDB. +## Overview + +PostgreSQL is the storage and coordination backend for self-hosted Rivet deployments that run more than one engine node. Multiple engine nodes can share a single PostgreSQL instance with no extra coordination service to deploy. Rivet handles leader election, failover, and version sequencing internally. + +Use PostgreSQL when you need: + +- **Multiple engine nodes** behind a load balancer for redundancy and horizontal scaling. +- **Multi-region deployments** (deploy one PostgreSQL instance per region, see [Multi-Region](/docs/self-hosting/multi-region)). +- **High availability** with a managed or self-managed primary/replica failover setup. + ## Choosing a Backend Pick your database backend based on how many engine nodes you run: @@ -59,6 +69,41 @@ Multi-node PostgreSQL deployments require NATS as the pub/sub backend so engine See the [production checklist](/docs/self-hosting/production-checklist#nats) and [Configuration](/docs/self-hosting/configuration) for details. +## Requirements and Recommendations + +### Version + +Use PostgreSQL 14 or newer. Rivet is tested against PostgreSQL 18, which is recommended for new deployments. + +### Connection Limits + +Each Rivet engine node opens a pool of direct connections to PostgreSQL and can use well over a hundred connections per node under load. PostgreSQL's default `max_connections` of `100` is too low for even a single busy engine node. + +- Set PostgreSQL `max_connections` to comfortably exceed `(number of engine nodes × 150)` plus headroom for backups, monitoring, and your own queries. +- If you use a managed PostgreSQL service, confirm its connection limit is high enough or pick a tier that allows raising it. Connection exhaustion shows up as engine startup failures or stalled requests under load. + + +Do not work around the connection limit with a connection pooler. See [Do Not Use Connection Poolers](#do-not-use-connection-poolers) below. + + +### Resources + +PostgreSQL is the system of record for the entire deployment, so size it accordingly: + +- Give PostgreSQL dedicated CPU, memory, and fast disk (SSD/NVMe with high IOPS). Avoid co-locating it with other heavy workloads. +- Rivet generates steady write and row-turnover on its internal tables. Keep autovacuum enabled and healthy so dead tuples do not accumulate. + +### High Availability and Backups + +A single PostgreSQL instance is a single point of failure for your whole deployment. + +- Configure a standby replica with automatic failover (managed services such as Amazon RDS, Cloud SQL, and Azure Database provide this). +- Enable automated backups and point-in-time recovery, and periodically test restoring from them. + +### Multi-Region + +Deploy one PostgreSQL instance per region or datacenter. Engine nodes connect to the PostgreSQL instance in their own region. See [Multi-Region](/docs/self-hosting/multi-region) for the full topology. + ## Managed Postgres Compatibility Some hosted PostgreSQL platforms require additional configuration due to platform-specific restrictions. @@ -209,3 +254,15 @@ Do not use: - PgBouncer - Supavisor - AWS RDS Proxy + + +## Troubleshooting + +### Too Many Connections + +Errors like `FATAL: sorry, too many clients already` or engine nodes failing to start under load mean PostgreSQL's `max_connections` is too low. Raise it to account for every engine node (see [Connection Limits](#connection-limits)). Do not add a connection pooler to work around this. + +### Connection Refused or TLS Errors + +- Confirm the engine connects directly to PostgreSQL and not through a pooler (PgBouncer, Supavisor, RDS Proxy). Rivet requires direct connections. +- For TLS errors, verify `sslmode` matches your server and, for custom certificate authorities, that `ssl.root_cert_path` points to the correct CA certificate. See [SSL/TLS Support](#ssltls-support). diff --git a/website/src/content/docs/self-hosting/production-checklist.mdx b/website/src/content/docs/self-hosting/production-checklist.mdx index c554dc2c3c..ad9c653e0d 100644 --- a/website/src/content/docs/self-hosting/production-checklist.mdx +++ b/website/src/content/docs/self-hosting/production-checklist.mdx @@ -34,10 +34,14 @@ Also review the [general production checklist](/docs/general/production-checklis ## PostgreSQL -- **PostgreSQL is recommended for multi-node deployments.** It is production-ready for light-to-moderate workloads (up to roughly 1,000 concurrent actors) but is not built for enterprise scale. Validate the deployment carefully before rollout. -- **Configure automated backups.** Set up regular backups for your PostgreSQL database to prevent data loss. -- **Configure failover.** Set up a standby replica with automatic failover to ensure high availability. -- **Use FoundationDB for the most scalable production-ready deployments.** FoundationDB provides the best performance, scalability, and uptime for Rivet. Contact [enterprise support](https://rivet.dev/sales) for FoundationDB guidance. +- **Use PostgreSQL for multi-node and multi-region deployments.** Multiple engine nodes can share one PostgreSQL instance; no extra coordination service is required. PostgreSQL is production-ready for light-to-moderate workloads (up to roughly 1,000 concurrent actors) but is not built for enterprise scale. See [PostgreSQL](/docs/self-hosting/postgres). +- **Raise `max_connections`.** Each engine node opens well over a hundred connections under load. Size `max_connections` to at least `(number of engine nodes × 150)` plus headroom. PostgreSQL's default of `100` is too low. See [Connection Limits](/docs/self-hosting/postgres#connection-limits). +- **Do not use a connection pooler.** Rivet requires direct connections. Do not put PgBouncer, Supavisor, or RDS Proxy in front of PostgreSQL. +- **Give PostgreSQL dedicated resources.** Provision dedicated CPU, memory, and fast disk, and keep autovacuum healthy. PostgreSQL is the system of record for the whole deployment. +- **Configure automated backups.** Set up regular backups and point-in-time recovery, and test restoring from them. +- **Configure failover.** Set up a standby replica with automatic failover to ensure high availability. A single instance is a single point of failure. +- **Use one PostgreSQL instance per region.** For multi-region deployments, deploy a separate PostgreSQL instance in each region. +- **Use FoundationDB for the largest deployments.** Enterprise teams running at very large scale can contact [enterprise support](https://rivet.dev/sales) for FoundationDB guidance. ## NATS