Repository navigation
PHP memory exhaustion when reading large log files via LogFileModelTraits #806
Description
Bug: PHP memory exhaustion when reading large log files via LogFileModelTraits
Description
The REST API crashes with a PHP fatal error when attempting to read large log files. The read_uncompressed_log() method in LogFileModelTraits.inc uses PHP's file() function which loads the entire file into memory, causing memory exhaustion on large rotated log files.
Environment
- pfSense Version: 2.8.1-RELEASE (amd64)
- REST API Version: pfSense-pkg-RESTAPI-2.6_8
- PHP Memory Limit: 512 MB
Error
PHP Fatal error: Allowed memory size of 536870912 bytes exhausted
(tried to allocate 541402264 bytes) in
/usr/local/pkg/RESTAPI/ModelTraits/LogFileModelTraits.inc on line 38
Root Cause
In LogFileModelTraits.inc, the read_uncompressed_log() method:
private function read_uncompressed_log(string $filepath): array {
$this->check_file_exists($filepath);
return file($filepath, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES); // Line 38
}This loads the entire file into memory as an array. When log rotation produces large files (500MB+ in my case due to misconfigured log limits), API calls that query logs will crash.
The same issue exists in read_bzip2_log() and read_gzip_log() which decompress entire files into memory.
Steps to Reproduce
- Have a large rotated log file (e.g.,
/var/log/system.log.X> 500MB) - Make an API call that reads system logs
- PHP crashes with memory exhaustion
Suggested Fix
Consider one of:
- Add file size check - Reject files over a threshold with a clear error
- Stream reading - Use
fgets()in a loop with line limit - Generator pattern - Yield lines instead of returning full array
- Pagination - Only read requested line ranges from file
Example size-limited approach:
private function read_uncompressed_log(string $filepath, int $max_bytes = 50000000): array {
$this->check_file_exists($filepath);
if (filesize($filepath) > $max_bytes) {
throw new NotAcceptableError(
message: "Log file too large to read. Size: " . filesize($filepath) . " bytes.",
response_id: 'LOG_FILE_TRAITS_FILE_TOO_LARGE',
);
}
return file($filepath, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
}Workaround
Reduce log file size limits in pfSense:
- Status → System Logs → Settings → Set log file size to 500KB or less
- Or add to config.xml:
<logfilesize>512000</logfilesize>
Activity
jaredhendrickson13 commentedon Jan 2, 2026
This should be relatively simple to address. Currently, Models that are not based on config values are not paginated until all objects have been loaded into memory. This was intentional as most of these models simply call internal pfSense callables that always return the full set of data, and cannot be paginated beforehand anyway. For logs however, this does demonstrate a legitimate use case for passing pagination parameters all the way down to these callables. Should be able to have that included in v2.7.0.
In the meantime, my suggestions would be:
fix: pass pagination limit down to log-reading callables to prevent O…
JeremiahChurch commentedon Mar 21, 2026
I've been hitting this OOM consistently on two of my firewalls (~30MB filter.log + 30 rotated bz2 files) via an MCP integration that calls ?limit=<whatever IE 50>. I put together a fix that passes the pagination limit down to the log-reading callables so they can stop early, as suggested by the OP
What it does:
Tested on a live pfSense CE 2.7.0 box (FreeBSD 15.0, REST API v2.5):
?limit=50memory?limit=10response timeI'm absolutely BEYOND rusty in PHP and I have zero experience with the API code (beyond using the snot out of it) so I have no idea if this is a good pattern or hallucinated trash. It generally seems reasonably limited in scope, and most importantly, it worked on real hardware for me.
Take a glance at https://github.com/JeremiahChurch/pfSense-pkg-RESTAPI/compare/fix/log-pagination-memory-exhaustion
If it looks like a workable path I'm happy to open a PR against this repo. If it's hot garbage, lemme know and I'll toss it in the trash.
jaredhendrickson13 commentedon Mar 22, 2026
@JeremiahChurch Seems like those changes should handle this okay. Definitely open to a PR into the
next_minorbranch so it can be tested more thoroughly.Thanks!
fix: pass pagination limit to log-reading callables to prevent OOM
fix: pass pagination limit to log-reading callables to prevent OOM
fix: pass pagination limit to log-reading callables to prevent OOM (#860