Gallery too slow or hanging for large image collections

Normally, yes. But I don’t upload photos in the usual way. I have attached an external drive with a few thousand images and need an efficient way to produce the thumbs. In my case, I need a way to generate the thumbnails “behind the scenes” or perhaps “on demand”.

1 Like

This seems plausible. I do not know how new files on your HD are discovered by the files app, but maybe the discover could be the point of time where you create such thumbnails!? This could also decrease the amount of thumbnails to be created each time.

I use the “External storage support” app, which works very well for me to access my files in File view. Accessing the external mount is transparent… only the folder icon of the new drive is somewhat different in Files view. Everything else feels the same so far as I can tell (although I haven’t explored all the options).

My problem is that after I attach the drive, I cannot use Gallery view at all… it hangs, or at least it appears to, and I cannot get past the root folder as the spinning wheel I’m sure is continuing to try (or failing) to create the thumbnails (there’s no feedback, so I don’t know what it’s really doing… I can’t find anything in the logs about it). I’ve let it continue with the “spinning circle” for hours, and still it hasn’t finished its work. Thus, I don’t think the moment of drive attachment would be the right point to generate the thumbs, at least not in the same manner as now.

This is why, at least in my case, I’d prefer to have the option of a separate or background or an “on demand” thumbnail creation. “Separate” would be something like the “create-thumbnails” feature mentioned in my earlier notes here. “Background” means it could create the thumbs behind the scenes, but still lets me go into Gallery view for those other folders which already have thumbs created, without interference by the thumbnail generation process. By “on demand” I mean the possibility to generate the thumbs only when the specific parent folder is clicked on. In my case, that would seldom involve more than 40 or 50 images, in contrast to having to bulk generate the other several thousand at one go.

The other thing which just occurred to me is that for all the images I would need to make thumbs for, after they are created the cache would probably be enormous. I’m running mysql which works well at the moment, but I imagine the database and php configurations would need to be considered, too, for handling such large numbers of images and their thumbnails. An additional consideration would be what happens if the external drive is ejected or unmounted. For me, it’s not an issue, since I am “permanently” mounting an separate partition on the server drive, and will not really ever think about ejecting it. But I’m sure other folks might attach a “real” separate hard drive.

Thanks for following up on this matter. It’s hardly a critical thing for the operation of Nextcloud/Owncloud, but solving it would facilitate the use of Gallery for those of us with mass quantities of images.

Mike

Quick question here. When using the occ command line to generate thumbnails… if the source picture(s) gets moved to a new folder within owncloud:

  1. Will a new thumbnail have to be generated for the source file after it’s moved?
  2. If a new thumbnail does indeed have to be generated, what happens to the old one?
  3. Where are these thumbnails stored and under what type of naming convention?

Thanks!
Mike

1 Like

No news?

I’m also trying to create thousands of thumbnails … no way, get this:

sudo -u www-data php ./occ gallery:create-thumbnails --a

[Symfony\Component\Console\Exception\CommandNotFoundException]
There are no commands defined in the “gallery” namespace.

NC 11.0, Gallery 16.0.0

Check a few posts above: Gallery too slow or hanging for large image collections

Hi

I’ve found that it’s not necessarily the thumbnail generation that causes all of the problems (although I am experiencing the same issues mentioned above)

My folder layout for my personal photography from the last 15 years is roughly (in a local “external folder”):
Pics/
Pics/photos/
Pics/photos//

For reference, I’m running NextCloud 11 on apache2.4, php7.1, on an i7-3770 with 16gb ram, and 2x raided(mirrored) 2tb spinny disks

Just loading the top-level Pics folder, which only includes a single folder, but no photos, takes about 15 seconds, even on subsequent loads (i.e. after loading thumbs)

Loading the photos folder then takes about 40 seconds; then the thumbnail generation starts, async from the browser.

At a guess, it’s doing a deep tree scan on every page load?

Thanks!

Seb

Thank you for this information. Can you please share when this should be ready in production?

1 Like

+1. I just have a few hundred pictures and this section of NextCloud is impossibly slow.

this feature is badly needed. i have hundreds of photos in every folder and previews are tooo slow to load.

Agreed. The mobile gallery apps are very fast. I would love to be able to ditch Apple for storing my photos.

i installed the preview-generator-program from the “store” and (re-)generate all previews with a nightly cronjob. i have ~100GB of pictures and the cronjob takes ~150mins. the previews take up ~83GB (!!!) in /data/appdata_ocjecx05wt96/preview but the gallery really is amazingly fast with it (before that, it wasn’t useable at all.)
so if you have the diskspace and can spare some computing cycles, i’d say it’s really worth a try.
GOOD LUCK!

Enhance your config.php:

'preview_max_x' => 512,
'preview_max_y' => 512, 
'preview_max_scale_factor' => 1,

Then follow this guide and your previews will jump up as popcorn.

~ 50GB of photos leads to 3,9GB of thumbs.

Hi!

Would you be so kind to tell where exactly mentioned lines of codes go in Generate.php and PreGenerate.php? I see a class (Generate and PreGenerate respectively) and can’t really decide in which function do they fit and what they will do (they possibly go into parseFile and processFile but I’m not a php wizard myself so not really sure).

Beside that, is it possible that you say a word about the difference between preview:generate-all and preview:pre-generate?

Thank you in advance!
hooger

Ok, for the second question I found this: https://github.com/rullzer/previewgenerator#how-to-use-the-app
It says:

preview:generate-all: Loop over all files and try to generate previews for them…

preview:pre-generate: Do the actual pregeneration. This means only for new or modified files (since the app was enabled or the last pregeneration was done).

the previews are very nice and fast once generated, but generation itself is pushing my system to its limits.

  1. the dir containing the previews has about the same size as the all the pictures together (110GB)
  2. generation takes 7+hrs/nite and increasing (cronjob) - even if there are no new pictures
  3. pre-generate does not work on my system (was finished after 2 secs but did not create any previews)
    is there any way this could be fixed? if i delete the huge preview-dir i am afraid re-genration will take DAYS)

sudo /snap/bin/nextcloud.occ preview:pre-generate
is returning immediately for me too (it doesn’t appear to do anything)
I tried adding -vvv and still nothing is shown and it exits right away.

sudo /snap/bin/nextcloud.occ preview:generate-all
is doing something :slight_smile:

You should make sure to use the update command in your crontab since it can’t take more then a second if there are no new images to generate.

This and opcache + local cache APCu are a must for big library’s.

Of course first time generation can take a while depending on library size and processing power.

After that it should be mega fast.

One issue I still walk in too is that sometimes the gallery will just stop loading images in half way. It gets stuck on the loader at the bottom and when clicked I get jumped back one directory. This might be a Ajax problem, I will soon try other update methods.

Both the apps for Android and iPad keep giving major issues loading content off any types, but especially the image heavy directories make the app freeze. This is horrible unusable, which is a shame. I decided to just use the web app in most cases.

I feel like this might be a WebDAV issue, and nextcloud might have to think of a different engine for syncing files over.

I’m not sure if I had a similar issue or the same. When I scrolled through the pictures maybe the tenth picture took about 1 minute to load. Some pictures were fast again and suddenly another one was extremely slow.
I figured I had error messages about file locks in the system logs/ journal and that this was caused by Redis not working. Once I got Redis fully working, this issue stopped. No more files were locked by mistake or too long.