Preview generation taking all the RAM

dlalonde · May 12, 2023, 11:39am

Nextcloud version (eg, 20.0.5): 26.0.1
Operating system and version (eg, Ubuntu 20.04): Ubuntu 22.04
Apache or nginx version (eg, Apache 2.4.25): 2.4.52
PHP version (eg, 7.4): 8.1

The issue you are facing:

I’ve installed NextCloud several times using a step by step document I wrote and everything always worked perfectly. But this time I’m facing an issue I’ve never had before. The first time I try to navigate through my files or even when I log into my Nextcloud and see the “Suggested files” widget, the loading image shows and the page never actually loads. When I check my system, I have nearly all my RAM taken by a process in the /tmp folder called appimage_[random numbers and letters]. After trying and reinstalling several times, I realized that the problem was the preview generation. If I don’t explicitely set enable_previews to false, the problem occurs. That’s on a fresh install with no extensions and the default files. If I set enable_previews to false, all is good. But if I don’t set it at all or leave it set to true then this happens.

I then tried installing the Preview Generator app and used the command to generate all the previews “occ preview:generate-all” but that never ends either. The command starts but never ends. But there are no PHP errors and litterally nothing in my nextcloud.log file. So it just sort of loads, takes up all the memory which blocks the app and that all.

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:

Install Nextcloud

The output of your Nextcloud log in Admin > Logging:

Nothing

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => ** REMOVED **,
  'passwordsalt' => ** REMOVED **,
  'secret' => ** REMOVED **,
  'trusted_domains' =>
  array (
    0 => ** REMOVED **,
  ),
  'datadirectory' => ** REMOVED **,
  'dbtype' => 'mysql',
  'version' => '26.0.1.1',
  'overwrite.cli.url' => ** REMOVED **,
  'dbname' => ** REMOVED **,
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => ** REMOVED **,
  'mysql.utf8mb4' => true,
  'dbuser' => ** REMOVED **,
  'dbpassword' => ** REMOVED **,
  'installed' => true,
  'default_phone_region' => 'CA',
  'filelocking.enabled' => true,
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
        'host' => '127.0.0.1',
        'port' => 6379,
  ),
  'allow_local_remote_servers' => true,
  'enable_previews' => true,
  'ffmpeg' => '/usr/bin/ffmpeg',
  'preview_libreoffice_path' => '/usr/bin/libreoffice',
  'enabledPreviewProviders' => array (
    'OC\Preview\PNG',
    'OC\Preview\HEIC',
    'OC\Preview\GIF',
    'OC\Preview\TIFF',
    'OC\Preview\BMP',
    'OC\Preview\XBitmap',
    'OC\Preview\MP3',
    'OC\Preview\TXT',
    'OC\Preview\MarkDown',
    'OC\Preview\Krita',
    'OC\Preview\MSOffice2003',
    'OC\Preview\MSOffice2007',
    'OC\Preview\MSOfficeDoc',
    'OC\Preview\PDF',
    'OC\Preview\SVG',
    'OC\Preview\Movie',
    'OC\Preview\Font',
    'OC\Preview\MKV',
    'OC\Preview\MP4',
    'OC\Preview\AVI',
    'OC\Preview\Photoshop',
    'OC\Preview\CSV',
    'OC\Preview\OpenDocument',
    'OC\Preview\Image',
  ),
  'htaccess.RewriteBase' => '/',
);

Can anyone help me or does anyone have the same issue? I could just disable the previews I know but I want them enabled and NC has been built for it to work so there must be a way.

Thank you!

jtr · May 12, 2023, 1:23pm

You appear to have enabled several non-default preview provider, several of which are disabled for performance reasons typically.

https://docs.nextcloud.com/server/stable/admin_manual/configuration_server/config_sample_php_parameters.html?highlight=enabledPreviewProviders#previews

You may also want to play with reducing your concurrency settings (also at the above link).

dlalonde · May 12, 2023, 2:16pm

Yeah I thought of that so I tried removing everything that had to do with previews in the config.php. So instead I had

'redis' =>
  array (
        'host' => '127.0.0.1',
        'port' => 6379,
  ),
'htaccess.RewriteBase' => '/',

But that didn’t help. I had to add

'enable_previews' => false,

And that solved it.

jtr · May 12, 2023, 2:40pm

I assume your config.php still had a bit more than that. But I get your point.

Config changes may take a few moments (60 seconds generally) depending on how you’re running PHP and opcache settings. Any chance it was that?

The /tmp folder being filled with appimage_* sounds a lot like easy-install integrated Nextcloud Office/CODE - that’s where it ends up. On my nearest test NC instance it consumes about 800MB.

It sounds like you’re using a tmpfs for /tmp so it’ll consume RAM rather than disk space.

You could be having a bit of an interaction between these otherwise unconnected pieces (previews, office, etc.) simply because they’re competing for resources (RAM namely in your case from the sounds of it).

Not sure your RAM capacity, but things may require some adjustment if you’re on a lower RAM system and want to run some of the beefier elements of NC. You may need to not use tmpfs or at least tune it to be much smaller (though that may have other implications) so moving to a disk rather than RAM might still be better.

dlalonde · May 13, 2023, 12:20pm

Yeah I was just showing the part I removed because the rest was the same

The appimage_* was from another test I did after installing Nextcloud Office.

I’ll check for the tmpfs but what bothers me the most is that of all the installations I’ve done with the exact same method, it’s the first time this has happened which makes me think it might have something to do with a change in NC26 or a package on the system that was updated and acts differently in PHP or something else. I’m hoping to find what that is to adjust my installation method. I’m going to test with an earlier version of NC on the same machine to see if it does the same thing.

dlalonde · May 13, 2023, 12:50pm

I can confirm this comes from a change in NC26. I just tried with NC25.0.5, which is the last version before 26 came out and it works perfectly and quickly. I then installed NC26.0 and even without adding the list of preview providers, it’s unable to create the previews in the “Suggested Files” widget after the “welcome” screen, so that’s directly out of the box after installation.

Now I need to figure out why.

I tried playing with preview concurrency but whether I set it lower or higher, it doesn’t work.

jtr · May 13, 2023, 7:07pm

The on-demand preview concurrency stuff is new in the most recent v26. It’s not v25.

In case you’re curious here’s one of the more informative comments from the PR while it was being evaluated:

github.com/nextcloud/server

Comment by szaimen to Limit number of concurrent preview generations

nextcloud:master ← dbw9580:master

I've tested this PR now and it looks like this solves the CPU meltdown problem w…hen loading a folder with a lot of pictures for which no previews already exists. So I am all in for merging this! 👍 Note that I've tested this on a bare-metal NC installation intel core i5-8259U with 4-cores and 8-threads, the app-data directory on a Nvme SSD and while using local external storage. (the testfolder with 258 pictures is on a USB-HDD) (I use nachoparker's recommended settings while testing https://ownyourbits.com/2019/06/29/understanding-and-improving-nextcloud-previews/) --- ### Without this fix: when I open the testfolder and scroll down, it tries to generate all previews at once, which results in maxing out the CPU: ![image](https://user-images.githubusercontent.com/42591237/106055046-23daf580-60ed-11eb-9c09-70282f88449a.png) It looks like this until all pictures are loaded which slows down the server and NC by a lot because no other process is able to work until this is done. --- ### After applying that fix : it only generates 4 previews at a time which doesn't kill the server anymore: (the CPU usages slowly raises from the idle usage while generating the previews and tops out at a load of around 3) (the screenshot shows the top-out state) ![image](https://user-images.githubusercontent.com/42591237/106055034-1e7dab00-60ed-11eb-83b7-893c09454ced.png) --- ## Take-aways: With this fix, the preview generation finally works on the fly and I no longer need to run a cronjob with the previewgenerator and don't need to waste a lot of space only for the previews since they are generated on the fly and only for folders that are really needed. ### So please merge this! --- Note: the Ram utilization is because of some other services that are runnin in the background and not due to NC

Do you mind sharing what the resources are on this device (CPU/RAM)?

ernolf · May 13, 2023, 10:30pm

You can check that with

mount | grep -qs "tmpfs on /tmp type tmpfs" && echo "/tmp is tmpfs" || echo "/tmp is no tmpfs"

In this context, I would be particularly interested to know whether you also have lots of directories in the /tmp directory with the name pattern /tmp/lu[a-z0-9]{8,10}.tmp
ls /tmp/lu* 2>/dev/null
For me, this happens with occ preview:generate-all and occ preview:pre-generate. The directories mainly contain temporary files for PDF and Office documents. They are not all deleted, which means that after a while the /tmp directory (which is on tmpfs in my case, see above) occupies entire RAM, which in turn leads to a server freeze.
I only use the occ preview:pre-generate commands with a wrapper script that removes the temporary files. Otherwise it doesn’t work for me at all

dlalonde · May 15, 2023, 11:06am

Thank you both for your time and help, it’s helping me understand how all of this works.

I don’t have tmpfs.

Saying that I’ve reinstalled several times since yesterday, changing configurations each time and the problem seems to come from PDF generation. If I remove PDF from the list and use occ preview:generate-all -vvv it works for every other file but if I keep it there, as soon as it hits the first file, it takes forever and never finishes and I have to reboot to kill a process. I have yet to find which one though.

dlalonde · May 16, 2023, 10:56am

OK I’m going to stop testing this. My whole machine has 2 Gb of RAM but it’s clearly not enough. Everytime I thought I found the culprit, I would reinstall to see if the problem would occur by removing or adding what I thought I found but, it always comes back no matter what I do, even with a fresh install and no additionnal configs for previews. So I tried upgrading my machine to 4 gb and everything was working really well. So I guess that’s the issue. 2GB is NOT enough for a machine to run Nextcloud.

Again thanks guys!