Repository navigation
Script crash when not enough RAM is available #21
Description
Activity
Is there any update on this? You can perhaps implement a very simple traffic light system where a variable is set to R (default) and G (if the prev command is successful).
This prevents for a long list of if/elif you can set the variable to R when one step begins and sets to G only if the step exit code is "0" so positive.
Else it stops.
BTW this traffic light will reduce ram demand as any parallel execution (parallel also in terms of RAM demand) is to be prevented.
Apart from the "unzip on the fly" and upgrade subshell, there's nothing really that runs in parallel and even there I use the wait command or loops that track the PID so the script pause and wait for the backgrounds tasks to complete before doing anything else. The script is already procedural and won't go to an other task before completing the preceding one.
I think the solution would be to stop the script if the free memory fall under a certain threshold and maybe even add a way to free some memory. The script already handle its own working directory and it use the upgrade service that stop and close a whole bunch of services.
You can see in my output above that it started to fail when mtd-write started. The script use 2 loops to track the mtd-write process, one that look for when the process start and the other one for when it is actually flashing. It failed at the writing stage after only 1 second. Then I don't know exactly what happened, but mtd-write did flash correctly but the script skipped out of the loop and went on with the other commands. You can see that the script identified the situation as a fail and went into the fail handling part of the code and this is where things got all messed up. First the step counter regression is impossible. There's nothing other than corrupted memory that can explain why it regressed, it should have counted up or if it missed the addition it would have stayed the same. There's nothing in the code that subtract the step counter. After this, you can see that other than the print of echo command, all other commands failed. Every segmentation fault you see is a failed command. At this point it doesn't matter anymore what is in the script if the shell itself as forgotten how to shell 😆.
Why this happened is still a mystery. I suppose the memory was critically low and some stuff got killed which broke the shell and script along it. Since the flash was ok, I suppose mtd-write have some mechanism to protect itself and make sure it write the firmware correctly, so maybe it is mtd-write that killed stuff????
I tried to kill the loop to check thing manually but I got kicked out of the SSH session and any subsequent SSH session would not connect. The GUI was still up but acting weird. I used the reboot "button" from the GUI which behaved strangely --> no waiting for reboot page with the countdown and continue button. My web browser just timed out, but eventually the router rebooted and it was correctly flashed and everything was working correctly.
I did many other flash after this and it went fine every time. I suppose our best guess is to closely monitor RAM usage and stop things and tell the user to reboot if it is under a certain threshold.
An other idea: my script is in USB storage and maybe, maybe since the system was low on memory, maybe it didn't keep the whole script in memory and when it tried to fetch the "missing" part it failed since USB storage get unmounted by the upgrade service. I remember having the discussion with @xixix about how the shell cache shell script.... It is somewhere on the forum...
Little update: I redesigned the upgrade function to make it more resilient. I didn't commit the code yet, soon.
I still need to find a way to handle RAM and put prevention validation in place. And I need to finish the new fname stuff. After all this, I suppose it will be time to start thinking about a GUI integration.
@wellloaded you didn't answer my question on the forum, can JavaScript really interact in real time with the script execution to make it interactive?
@wellloaded you didn't answer my question on the forum, can JavaScript really interact in real time with the script execution to make it interactive?
Yes of course it's possible, but why?
I would invite not to use the GUI as a way to react to the script output. I used Zyxel devices, and they have a pretty clean interface which I particularly like. When an update is found (like we have today) there's a simple button: "Upgrade". The rest is a progress bar and an automated reboot. I think this is a good way to implement the solution. If it fails we can report back the error, sure; just make sure the script spits out the exit codes and it should be relatively straight-forward to implement.Yes of course it's possible, but why? I would invite not to use the GUI as a way to react to the script output.
Because you asked me to...
Maybe I misunderstood you, but you expressly asked for the opposite.
But no interaction please me, way easier IMO. The script is already error codes rich. I need to do a cleanup to reduce the size, but right now there's pretty much a unique error codes for every scenario. Maybe I can regroup some of them an drop some others to shave weight, this is not urgent though. Right now my priority is to make sure the router doesn't run out of memory.
This is AI suggestion:
# --- START MEMORY PREP --- sync # Flush any pending writes echo 3 > /proc/sys/vm/drop_caches # Clear the cache echo 1000 > /proc/sys/vm/vfs_cache_pressure # Keep filesystem cache tiny echo 8192 > /proc/sys/vm/min_free_kbytes # Force a higher free-RAM floor # --- END MEMORY PREP --- # Run your flash here... mtd-write ...I tried
echo 3 > /proc/sys/vm/drop_cachesand this works. No idea if the other stuff the AI is saying make sense? I don't have enough knowledge in the matter.I might have an idea for low memory MIPS router... Instead of downloading and unzipping at the same time "unzip on the fly", the AI suggested flashing on the fly.
I was asking giving my RT-N12 as an example, here the AI reply:
Scenario B: You "Stream" the flash (Piping)
If your script does something like wget -O - http://server/fw.trx | mtd-write -i -, the firmware is never fully stored in RAM.
Threshold: ~6MB absolute free RAM.
This is the only way to reliably flash a 32MB device without a segfault.AI suggest to keep firmware size + 3MB free for low memory devices and 15% of the total RAM for "high end" devices. Any opinions on this?
# Get total RAM in KB TOTAL_RAM=$(awk '/MemTotal/ {print $2}' /proc/meminfo) # Define the "Safe Zone" based on hardware class if [ "$TOTAL_RAM" -lt 65536 ]; then # LOW RAM (e.g., RT-N12 / 32MB-64MB) # Strategy: Strict absolute minimum. # Must have at least FW_SIZE + 3MB free. MODEL_CLASS="LOW" REQUIRED_OVERHEAD=3072 else # HIGH RAM (e.g., RT-AC68U / 256MB+) # Strategy: Percentage based + Buffer. # Must have at least 15% free to prevent shell fragmentation. MODEL_CLASS="HIGH" REQUIRED_OVERHEAD=$((TOTAL_RAM * 15 / 100)) fiI'm no expert here. No idea if those values make sense. A bit of help would be greatly appreciated 🙂
I'm trying to remember what I meant by that but I suppose it was what I now call a: a new version is available XXXXX do you want to upgrade.
The way this works in my mind is:
- try the automated approach e.g. click upgrade
- If that fails bring the error to the GUI for user attention.
In this sense, the -a and -y together?
P.S. I have never had any success controlling the Linux kernel memory usage. You can find some info searching the adblock-V2 thread on the forum where the topic was discussed extensively. This said it was way before AI, so perhaps the AI approach can be tested?
flashing on the fly.
But this would brick the device if any packet loss is experienced right? e.g. over 4G.I'm trying to remember what I meant by that but I suppose it was what I now call a: a new version is available XXXXX do you want to upgrade.
The way this works in my mind is:
* try the automated approach e.g. click upgrade * If that fails bring the error to the GUI for user attention.In this sense, the -a and -y together?
Ok yeah. Yup I planned to make -y implied when using -a
P.S. I have never had any success controlling the Linux kernel memory usage. You can find some info searching the adblock-V2 thread on the forum where the topic was discussed extensively. This said it was way before AI, so perhaps the AI approach can be tested?
flashing on the fly.But this would brick the device if any packet loss is experienced right? e.g. over 4G.Hmmm good point. Ok this is a bad idea.
This commit should help prevent this issue.
I did not have time to test all possibilities, especially for MIPS device since I have none in use right now.I'm testing the latest beta, it did upgrade, but I had to restart manually, also please notice the i/o errors.
I'm admittedly running from USB this shouldn't happen if the script is found in the squashfs I suppose?Starting FreshTomato Upgrade Script v0.52-test-008 (2026-02-21) [1/7] Download firmware... Download mode: FAST -- direct download Connecting to 10.10.10.50 (10.10.10.50:80) saving to '/tmp/ftupwd/freshtomato-RT-AC1900P-K26ARM-2026.1-AIO-64K.trx' freshtomato-RT-AC190 100% |*********************************************************************************************************| 30.2M 0:00:00 ETA '/tmp/ftupwd/freshtomato-RT-AC1900P-K26ARM-2026.1-AIO-64K.trx' saved Downloaded in 00:00:57 Saved: /tmp/ftupwd/freshtomato-RT-AC1900P-K26ARM-2026.1-AIO-64K.trx [2/7] Prepare files... Done [3/7] Identifying device... Done [4/7] Verifying files... Done ------------------ POINT OF NO RETURN ------------------ WARNING: Make sure you are using the right firmware before proceeding! Current: Router : Asus RT-AC1900P NVRAM : 64K Version : 2026.1 K26ARM USB AIO-64K Filename: freshtomato-RT-AC1900P-K26ARM-2026.1-AIO-64K.trx Upgrade with: Filename: freshtomato-RT-AC1900P-K26ARM-2026.1-AIO-64K.trx WARNING: Do not interrupt the router! Proceed dirty upgrade? To cancel press CTRL+C Starting dirty upgrade with firmware 'freshtomato-RT-AC1900P-K26ARM-2026.1-AIO-64K.trx' [5/7] Stop services... Done [6/7] Writing firmware... /mnt/USB/ftup: line 666: pidof: Input/output error Done /mnt/USB/ftup: line 676: cat: Input/output error mtd-write: /mnt/USB/ftup: line 117: basename: Input/output error logger: line 1: Device: Input/output error Successfully flashed freshtomato-RT-AC1900P-K26ARM-2026.1-AIO-64K.trx [7/7] Rebooting now... /mnt/USB/ftup: line 690: date: Input/output error /mnt/USB/ftup: line 690: uptime: Input/output error /mnt/USB/ftup: line 690: sed: Input/output error Rebooting @ , uptime = /mnt/USB/ftup: line 691: sync: Input/output errorOh wow..... 🤔
I'll need to investigate this one... First time I see this. Input/output error..... 🤔
I made some mistake in my latest commit, but it shouldn't be catastrophic like this.... Is this the version with the new "drop_caches" command? I'm wondering if clearing the cache is the cause here. With USB unmounted the script must be kept in cache I suppose...??? I also run everything from USB storage and I didn't encounter this one yet...
Personally I would keep it simple, fiddling with those RAM related commands you are always going to hit the wall in a process that it's instead meant to be the most stable as possible. I think a failure due to lack of RAM is acceptable. Not nice, but better than being force to power-cycle a remote device to unblock the situation.
According to AI (I haven't' tested this yet) we can hard reboot like this:
echo 1 > /proc/sys/kernel/sysrq
echo b > /proc/sysrq-triggerWhich triggers an immediate hard reboot, skipping the shutdown procedure.
This is meant to be the equivalent of power cycling which is ok on a router with squashfs and assuming the USB is unmounted. I discovered this while looking in way to shorten the reboot time. What it stands out here is that, apart from the echo, there are no commands involved. Worth a check.
kill -6 1
Seemed to work like you describe, no proper shutdown, direct reboot.
Look here:
FreshTomato-Project/freshtomato-mips#32 (comment)Reacted by rs232@wellloaded I did a commit that remove the dump_cache command. Can you please test and tell me if it fix the I/O errors and if the reboot still hang, thanks :-)
Also I removed all the dark blue fonts ;-)
I'm testing the latest beta, it did upgrade, but I had to restart manually
I'm gonna ask for a lot, but I'm very curious. I have this watchdog idea... Can you please run this command before upgrading:
( ( sleep 600 && kill -6 1 ) & ) &Then make it so the reboot will hang and do the upgrade. I hope this works, if it does it should reboot the router after 10 minutes. You can adjust the time if it is too long. I want to see if it could be a solution to hanging upgrades... If it works, maybe it could be added directly in rc/init.c or anywhere relevant.
I put some barrier in place. I hope they work ok and also not too aggressive.
I encountered a situation where the router only had just enough RAM for the firmware, but not enough to flash it. This will need to be investigated or at least find a way to free memory or just stop the script if RAM is too low.
See this post:
https://www.linksysinfo.org/index.php?threads/freshtomato-upgrade-script-v0-51.78326/post-365849
The router was already acting weird and the script already had difficulties downloading the firmware.
I ran the
freecommand before retrying the upgrade:root@AC68U-Relais:/tmp/home/root# free total used free shared buff/cache available Mem: 255572 29688 37724 0 188160 0 -/+ buffers/cache: 29688 225884 Swap: 0 0 0After the script stopped, SSH connection was NOT possible anymore. My Wireless Ethernet Bridge also died, but the second band AP was still up and httpd was still running to, so I could reboot from the GUI. But the GUI was broken too. The reboot button acted weird. The GUI ports where broken...
Miraculously the upgrade went fine and after rebooting the router was OK and running the right firmware.