Gallery too slow or hanging for large image collections

🍱 Features & apps
1

I would like to suggest there be a different method to initialise the Gallery file previews.

I have a few thousand images spread across a few hundred directories, and it’s impossible to wait so long for Gallery to initialise the previews each time I wish to go into the files. I’ve tried to wait overnight for the “spinning wheel” to stop so I can actually go into the directories, but even that doesn’t work… the spinning wheel doesn’t stop. I did create a small test directory at the top of the file tree with only a few images, and Gallery works fine. But for a deep directory list, it either hangs or is impossibly slow.

This limitation (large image collections will take a long time to load) was mentioned in the documentation, so I am not reporting a “bug”, but rather a request for improved features in future releases.

Thus, I suggest one or more changes (or perhaps make them options) in Gallery to improve handing of large image collections:
– prevent preview images for directories; allow only for the images themselves
– allow “on demand” initialising of a directory; that is, only create previews for the current directory and do not descend further to create previews until the next directory is clicked
– improve the caching so that previews are not newly initialised each time the file list is accessed
– suggest alternative php settings to facilitate the handling of many files.

Thanks for any suggestions how to handle this problem.
Nextcloud 10.0.1, Gallery 15.0.0, Ubuntu 16.04.3, mysql 5.0.12, php 7.0.8

1 Like
2

There are plans for an automatic background thumbnail generation:

Until then, you could use the occ-command to do this manually (or run a cronjob at night):

Does this solve your issue or do you want to have a different approach?

3

I think that should solve the problem of thumbnail generation. I’ll give it a try. Thanks for the advice and future notice of auto background generation.

Mike

4

Hi again,

I went into the wiki page you mentioned and it says “This feature is not yet available in production”. Indeed, I tried the occ command detailed on that page and got the error message:

[Symfony\Component\Console\Exception\CommandNotFoundException]
There are no commands defined in the "gallery" namespace.

I also went to the git page for gallery (https://github.com/nextcloud/gallery) and noticed this:

This version is for ownCloud 9.2 and Nextcloud 11.

However, I cannot find anywhere how to install Nextcloud 11 so I can try out this version of Gallery.

Do you have any suggestions what I might do to run the occ command, or upgrade to Nextcloud 11, or anything else that I am doing wrong?

Thanks again…

Mike
Nextcloud 10.0.1, Gallery 15.0.0

5

Try the galleryplus app, it’s the 9.1-version with backports of the new features:

6

Sorry to bother you again.

I successfully installed Gallery+ (it now reads version 15.1.1), but I still cannot run the occ “create-thumbnails” command. I tried the command as found in https://github.com/nextcloud/gallery/wiki/Create-Thumbnails using both:

./occ gallery:create-thumbnails (as indicated on the wiki page)

and
./occ galleryplus:create-thumbnails (thinking that the galleryplus directory might be the right one instead of gallery)

Both produced the same error:
"There are no commands defined in the “gallery” namespace."
and “There are no commands defined in the “galleryplus” namespace.”

I dug into both gallery and galleryplus directories and cannot find any mention of “create-thumbnails”. Do I need to install or access the backports in another way?

Thanks again for your help.

Mike

7

@oparoz can you help?

8

Thanks for the feedback. That thumbnail generating command never made it into production, but we’re talking about maybe integrating it into core as well as using a background job.

1 Like
9

OK, thanks for the information. I will look forward to seeing this feature in a future update!

Mike

10

Doesn’t it make sense to generate the preview after upload of file was successful?

1 Like
11

Normally, yes. But I don’t upload photos in the usual way. I have attached an external drive with a few thousand images and need an efficient way to produce the thumbs. In my case, I need a way to generate the thumbnails “behind the scenes” or perhaps “on demand”.

1 Like
12

This seems plausible. I do not know how new files on your HD are discovered by the files app, but maybe the discover could be the point of time where you create such thumbnails!? This could also decrease the amount of thumbnails to be created each time.

13

I use the “External storage support” app, which works very well for me to access my files in File view. Accessing the external mount is transparent… only the folder icon of the new drive is somewhat different in Files view. Everything else feels the same so far as I can tell (although I haven’t explored all the options).

My problem is that after I attach the drive, I cannot use Gallery view at all… it hangs, or at least it appears to, and I cannot get past the root folder as the spinning wheel I’m sure is continuing to try (or failing) to create the thumbnails (there’s no feedback, so I don’t know what it’s really doing… I can’t find anything in the logs about it). I’ve let it continue with the “spinning circle” for hours, and still it hasn’t finished its work. Thus, I don’t think the moment of drive attachment would be the right point to generate the thumbs, at least not in the same manner as now.

This is why, at least in my case, I’d prefer to have the option of a separate or background or an “on demand” thumbnail creation. “Separate” would be something like the “create-thumbnails” feature mentioned in my earlier notes here. “Background” means it could create the thumbs behind the scenes, but still lets me go into Gallery view for those other folders which already have thumbs created, without interference by the thumbnail generation process. By “on demand” I mean the possibility to generate the thumbs only when the specific parent folder is clicked on. In my case, that would seldom involve more than 40 or 50 images, in contrast to having to bulk generate the other several thousand at one go.

The other thing which just occurred to me is that for all the images I would need to make thumbs for, after they are created the cache would probably be enormous. I’m running mysql which works well at the moment, but I imagine the database and php configurations would need to be considered, too, for handling such large numbers of images and their thumbnails. An additional consideration would be what happens if the external drive is ejected or unmounted. For me, it’s not an issue, since I am “permanently” mounting an separate partition on the server drive, and will not really ever think about ejecting it. But I’m sure other folks might attach a “real” separate hard drive.

Thanks for following up on this matter. It’s hardly a critical thing for the operation of Nextcloud/Owncloud, but solving it would facilitate the use of Gallery for those of us with mass quantities of images.

Mike

14

Quick question here. When using the occ command line to generate thumbnails… if the source picture(s) gets moved to a new folder within owncloud:

  1. Will a new thumbnail have to be generated for the source file after it’s moved?
  2. If a new thumbnail does indeed have to be generated, what happens to the old one?
  3. Where are these thumbnails stored and under what type of naming convention?

Thanks!
Mike

1 Like
15

No news?

I’m also trying to create thousands of thumbnails … no way, get this:

sudo -u www-data php ./occ gallery:create-thumbnails --a

[Symfony\Component\Console\Exception\CommandNotFoundException]
There are no commands defined in the “gallery” namespace.

NC 11.0, Gallery 16.0.0

16

Check a few posts above: Gallery too slow or hanging for large image collections - #8 by oparoz

17

Hi

I’ve found that it’s not necessarily the thumbnail generation that causes all of the problems (although I am experiencing the same issues mentioned above)

My folder layout for my personal photography from the last 15 years is roughly (in a local “external folder”):
Pics/
Pics/photos/
Pics/photos//

For reference, I’m running NextCloud 11 on apache2.4, php7.1, on an i7-3770 with 16gb ram, and 2x raided(mirrored) 2tb spinny disks

Just loading the top-level Pics folder, which only includes a single folder, but no photos, takes about 15 seconds, even on subsequent loads (i.e. after loading thumbs)

Loading the photos folder then takes about 40 seconds; then the thumbnail generation starts, async from the browser.

At a guess, it’s doing a deep tree scan on every page load?

Thanks!

Seb

18

Thank you for this information. Can you please share when this should be ready in production?

1 Like
19

+1. I just have a few hundred pictures and this section of NextCloud is impossibly slow.

20

this feature is badly needed. i have hundreds of photos in every folder and previews are tooo slow to load.