503 “No Available Server” Errors in Production #1893

Hello @ralf-berger ,
since merging the latest dev changes into main and deploying them to production, I have noticed intermittent server availability problems. Sometimes the data loads correctly in the UI and the system works fine, while at other times requests fail with '503 Service Unavailable' and the response 'no available server'. There are also repeated failed socket.io requests in the browser.

I then checked the production application in Argo CD. The webserver deployment currently shows one healthy pod, but the pod events contain a very large number of restart/backoff events. The pod summary also shows that the container had recently terminated with exit code 134.

Whereas in br-dev, I observed no such events:

In the webserver logs, I found the following error shortly before the container restarted:
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
The logs show the JavaScript heap reaching roughly 2.2 GB before the process aborts.

My assumption is therefore that the webserver runs out of memory, restarts, and during that period there is temporarily no available backend server, which would explain the intermittent 503 responses and missing data in the UI.
Based on this, it looks like the webserver is reaching its available memory/heap limit and restarting, which would explain the temporary 503 responses and missing data in the UI.
Could increasing the available memory for the webserver be an appropriate solution here? Could you please have a look and advise how we should proceed?

I also noticed that the webserver pod does not remain consistently healthy in Argo CD. At times it is shown as Degraded, which seems to coincide with the container restarts and the periods when the frontend receives 503 / no available server responses.

Since there is currently only one active webserver pod, would increasing the number of replicas help avoid downtime when one pod restarts, or is a single replica required by the current setup? I would really appreciate your help in this matter.