I have a NC 24.0.4 installation on an Odroid HC2, inside Docker, using the Apache image. What I noticed is that opening certain pages in a single browser makes my whole box to kneel and reboot within seconds.
What I could determine was that those pages initiate a ton of requests simultaneously (separate requests for all the displayed elements at once), which drives RAM usage of the NC container to skyrocket, exhaust any physical memory quickly, and then the server kernel panicks.
The pages I observed this behavior are the Trash bin view, which, if I scroll down fast, keeps loading new pages of deleted files, and initites requests for each displayed file’s thumbnails without any throttling; and also the Maps app, when showing photos is enabled, and there’s a lot of photos to be put on the map.
First of all, I wonder if this is normal behaviour, to let a single page initiate requests by the dozens or hundreds, simultaneously? This seems like a good recipe for a quick DOS. Shouldn’t pages like these apply some throttling on the browser side?
Also, what would be the best practice to fine tune both my NC container and the MySQL DB? Maybe I should use a different image? Should I set different limits for my containers? Perhaps tune thee DB in some way? Is there maybe any kind of server-side queuing/throttling solution that could prevent such overloads?
I can’t give you a solution off the bat, but it makes sense that it is about thumbnail generation happening. And map tiles generation is probably similar. Is it only these two it happens on?
I would guess that thumbnail generation is done using e.g. ImageMagick, and it can take up quite some resources to process images, of course also depending on size and type of images. You can check using process monitoring tools in your OS what processes are working and eating up memory when you request the offending pages.
Thumbnail generation can generally speaking be done in some background job. I have an (old) NC where it’s all about showing images, and there I recall that thumbnails are generated using a background job. So maybe that is something to look into.
But at the end of the day, it’s also a matter of how much resources you actually have on the system. What are they?
On a related note, I’m not saying this is the case here, but theoretically your system could be rebooting due to processing being so intense that it overheats - this is not uncommon. Of course whether this is relevant or not depends on how well your CPU is cooled in that device.
Generally speaking, you need to start isolating the matter to know what the culpruit is, and then act on that information.
I think there is an improvement in thumbnail generation in Nc26. You could try to update to that and check if it improves the situation. Please create a backup first!
This happens on v25 too, and I’m about to upgrade to v26 and see.
I did try to investigate the issue. Now that I started using the Bookmarks app too recently, it also causes the system to crash when I perform a search, and all those results try to display the thumbnails all at once.
It is not the generation of the thumbnails that causes the crash either. First of all, I have a cron job for generating all the thumbnails, and files in the trash can are expected to already have cached thumbnails when this happens. Also, if I have all the thumbnails created for my images by panning around on the map, whenever I open a wide enough view with lots of images to show, it crashes. The bookmarks should generate a thumbnail once they are saved and be done; and yet anytime I do a search, the displaying of the results takes my server down.
Also, as far as I could tell, it’s a lot of MySQL processes which seem to saturate the server. It has 2GB of RAM (it is a NAS appliance). One of the issues is that for some reason, I tend to see nothing relevant in the system logs after such a crash. Which might have to do with Armbian’s log2ram system, which is supposed to lift the burden of storing logfiles on the system SD card. But I don’t know enough about it to figure out how to make sure to persist all system logs in the event of a kernel panic.
Upgraded to v26, same symptom. Tried to monitor the syslog while trying to search among bookmarks, but I saw nothing, just suddenly the server rebooted.