Add GOMEMLIMIT and container memory limit guidance to the Generator and Host setup docs.

[johnnymatthews]

From a Discord thread (Menna, Source):

can we add GOMEMLIMIT to the generator and host setup docs? not sure if it's already there. the Go runtime can't see a container's memory limit, so without it the gc has no ceiling to work against and the heap grows until the kernel kills the container. defradb also falls back to capping its inbound p2p queue at a flat 1 GiB, so a 16GiB node drops messages it had the memory for. Just a one line like: Set GOMEMLIMIT below your container's memory limit. We recommend 14GiB against 16GiB.

GOMEMLIMIT already shows up in a lot of places, so this is mostly filling gaps and making the wording consistent, not writing it from scratch. Both config-reference pages have a table row for it, and around six deployment-example pages explain it (managed-gcp-node, nginx-with-snapshots, high-throughput-tuning, quicknode-setup, prod-vm-nginx-tls, archival-vs-pruned). The infra scripts in both client repos already set mem_limit: 16g with GOMEMLIMIT=14GiB.

What's missing:

• The Host quickstart ships GOMEMLIMIT=14GiB and mem_limit: 16g in its compose snippet but never explains them. Someone following the guide copies the file without learning what the values do or that they should rescale them for a different VM.
• The Generator install page has no memory limits at all. The first-run docker run example and the source-build .env both omit GOMEMLIMIT and any container limit.
• The DefraDB inbound p2p queue fallback isn't documented anywhere. This is the strongest reason to set GOMEMLIMIT beyond avoiding OOM kills.
• The keep-both-limits point isn't written down. The closest is high-throughput-tuning mentioning headroom for non-Go memory, but nothing states that the container mem_limit and GOMEMLIMIT bound different things and both should stay.

The same thread asked whether the container mem_limit becomes redundant once GOMEMLIMIT is set. It doesn't, and the section below explains why.

Why set GOMEMLIMIT

The Go runtime can't see a container's memory limit on its own. Without GOMEMLIMIT, the garbage collector has no ceiling to work against, so the heap grows until the kernel OOM-kills the container.

Setting GOMEMLIMIT also fixes a second problem. DefraDB sizes its inbound p2p queue off the runtime's memory limit. When GOMEMLIMIT is unset, DefraDB falls back to capping that queue at a flat 1 GiB, so a 16 GiB node drops p2p messages it had the memory to hold.

Set GOMEMLIMIT below the container's memory limit. We use 14GiB against a 16g container limit and scale both together on larger or smaller hosts.

Keep both limits

The container mem_limit and GOMEMLIMIT are not redundant, so keep both. The container limit is a hard ceiling the kernel enforces. GOMEMLIMIT is a soft advisory the Go garbage collector uses to decide when to run. They bound different memory, and you want both in place.

Proposed changes

• Enrich the GOMEMLIMIT row on both config-reference pages as the one place that fully explains it: add the OOM-kill risk, the rule to set it below the container limit, the 14GiB-against-16g recommendation, the DefraDB p2p queue fallback, and the keep-both-limits note. Pages: content/run/run-a-host/config-reference/index.md and content/run/run-a-generator/config-reference/index.md.
• Add a short one-line callout next to the compose snippet in the Host quickstart that links to the config-reference explanation. The snippet already has the values. Page: content/run/run-a-host/quickstart/index.md.
• Add a short memory note to the Generator install page, keeping the first-run docker run example as-is and linking to the config-reference explanation instead of adding flags. Page: content/run/run-a-generator/install/index.md.
• Tie the 16 GB RAM recommendation to the limits on both hardware-requirements pages: the 16 GB figure assumes a 16g container limit with GOMEMLIMIT at 14GiB, and both scale together. Pages: content/run/run-a-generator/hardware-requirements/index.md and content/run/run-a-host/hardware-requirements/index.md.
• Standardize the GOMEMLIMIT wording across the existing deployment examples so they agree and link to the canonical config-reference row instead of restating it differently each time. Pages: prod-vm-nginx-tls, quicknode-setup, managed-gcp-node, nginx-with-snapshots, high-throughput-tuning, archival-vs-pruned.
• Verify the DefraDB inbound p2p queue fallback against the pinned DefraDB version (v1.0.1-0.20260724174804-b05811e709e1) before publishing that claim. It's DefraDB-internal, not in either client's source, so confirm the exact behavior and wording first.

Notes

The infra already does the right thing. host-client/host-prod-setup.sh and the generator-client compose files all set mem_limit: 16g with GOMEMLIMIT=14GiB. This issue is about making the docs match and explain it.