Skip to content

feat(limits): raise PHP memory_limit automatically when a site runs out - #208

Merged
nechodom merged 1 commit into
mainfrom
claude/elementor-memory-exhaustion-3c5949
Oct 5, 2026
Merged

nechodom merged 1 commit into
mainfrom
claude/elementor-memory-exhaustion-3c5949

Conversation

@nechodom

@nechodom nechodom commented Oct 5, 2026

Copy link
Copy Markdown
Owner

What

Opt-in automatic PHP memory_limit per hosting. The owning node watches the site's logs/error.log for Allowed memory size … exhausted. On an out-of-memory error at the limit in force, it raises the pool's memory_limit one 128 MB step, but never above a user-set ceiling or what the node's RAM can hold. After two quiet weeks it steps back down, never below the operator's own value.

Why

dluhopisar.cz (Elementor) keeps dying with Allowed memory size of 268435456 bytes exhausted. The pool sets memory_limit as php_admin_value, so WP_MEMORY_LIMIT / ini_set cannot raise it. Until now the only fix was a manual edit of the limits card after someone noticed.

Operator-facing change?

  • No operator-facing change.

  • Operator-facing change — described below:

  • Hosting → PHP-FPM & DB limits gains two controls. The first is the checkbox "Raise PHP memory automatically when the site runs out", which is off by default. The second is "Automatic ceiling" (MB, default 512, valid range 16–8192). A status line shows whether the limit has been raised from your value and when the site last ran out of memory.

  • How it behaves:

    • The check runs every 2 minutes.
    • Each raise is +128 MB, with at least 10 minutes between raises.
    • The limit never exceeds the ceiling, and limit × max_children stays under ¾ of the node's RAM.
    • After 14 days without an out-of-memory error, the limit drops back one step.
    • Unticking the checkbox restores the operator's value.
  • New notifications (operator language, en + cs):

    • ops.php_mem_raised (info) on each raise.
    • ops.php_mem_capped (warn, at most once a day) when the site still runs out at its ceiling.
    • ops.php_mem_capped_ram (warn) when the node's RAM, not the ceiling, blocks a further raise.
  • Audit entries: php.mem_auto.raise, php.mem_auto.lower and php.mem_auto.restore.

Notes for review

  • Floor rule: set_limits resets the floor (base_mb) only when the memory value actually changes. The card shows the raised value, so saving it unchanged must not make that raised value the floor, or the limit would never come back down. The tick goes through the new store_and_apply_limits, which never resets the floor.
  • Reading tenant logs as root: the log is read with fs::read_appended, which opens it via open_no_follow. That function now also passes O_NONBLOCK, so a FIFO swapped in for the log is refused instead of hanging the open; this also applies to the existing log viewer. The reader tracks inode and offset, so rotation and truncation are handled. It reads at most 8 MiB per tick, holds back a half-written last line, and skips the existing history the first time it sees a log.
  • Stale errors: out-of-memory errors at a limit lower than the current one are ignored. These are lines written before the last raise.
  • Where state lives: in the owning node's hosting_kv (php.mem_auto, php.mem_auto_max_mb, php.mem_auto_state). No migration and no RPC change. An older node accepts the settings but does nothing with them.
  • Scheduling: the tick runs on its own agent task, so slow log reads never hold up fail2ban or quota enforcement.
  • Forged errors: a tenant who writes fake out-of-memory lines into their own log can only push the limit up to the operator's ceiling.

Test plan

  • cargo test --workspace passes. Not run in full; the touched crates (types, adapters, core, web) all pass.
  • cargo clippy --workspace --all-targets passes (no new warnings).
  • cargo fmt --all -- --check passes.
  • Manually tested on a real Debian node (when applicable). Not yet; needs a real Elementor out-of-memory error to confirm the nginx line format end to end.
  • Added / updated tests covering the change:
    • phpmem unit tests: parser, stale-error filter, raise, cooldown, ceiling, RAM cap, alert throttle, lowering, ceiling lowered below the current limit.
    • read_appended tests: growth, partial line, truncation, burst larger than the read limit, symlink refused.
    • Service test php_mem_auto_raises_rebases_and_restores.
  • Dev panel: the settings save and show again after a reload; an invalid ceiling (99999) is refused.

Anti-scope

  • No per-request split (a higher limit only for /wp-admin or the Elementor editor).
  • No profile-level default for the automation or its ceiling; it is per hosting only.
  • Not on by default, so sites with a deliberately low limit keep it.

🤖 Generated with Claude Code

The pool sets memory_limit as php_admin_value, so a site that genuinely
needs more (Elementor editor, WooCommerce imports) cannot raise it itself
and just dies with "Allowed memory size ... exhausted" until someone edits
the limits card.

Opt-in per hosting on the PHP-FPM & DB limits card, with a user-set
ceiling (default 512 MB). The owning node reads what nginx appended to the
site's error.log every two minutes (O_NOFOLLOW, inode/offset tracked,
history skipped on first sight) and, on an OOM at the limit in force,
raises the pool 128 MB, bounded by the ceiling and by node RAM
(limit x max_children <= 3/4 MemTotal), with a 10-minute cooldown. Two
quiet weeks step it back down, never below the operator's own value;
turning it off restores that value. Raises notify (info); a site still
OOMing at its ceiling warns once a day.

A hand-typed memory value becomes the new floor; saving the card
unchanged does not adopt the raised value as the floor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nechodom
nechodom merged commit 6b47554 into main Oct 5, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant