Gallery too slow or hanging for large image collections

I think that should solve the problem of thumbnail generation. I’ll give it a try. Thanks for the advice and future notice of auto background generation.

Mike

Hi again,

I went into the wiki page you mentioned and it says “This feature is not yet available in production”. Indeed, I tried the occ command detailed on that page and got the error message:

[Symfony\Component\Console\Exception\CommandNotFoundException]
There are no commands defined in the "gallery" namespace.

I also went to the git page for gallery (https://github.com/nextcloud/gallery) and noticed this:

This version is for ownCloud 9.2 and Nextcloud 11.

However, I cannot find anywhere how to install Nextcloud 11 so I can try out this version of Gallery.

Do you have any suggestions what I might do to run the occ command, or upgrade to Nextcloud 11, or anything else that I am doing wrong?

Thanks again…

Mike
Nextcloud 10.0.1, Gallery 15.0.0

Try the galleryplus app, it’s the 9.1-version with backports of the new features:

Sorry to bother you again.

I successfully installed Gallery+ (it now reads version 15.1.1), but I still cannot run the occ “create-thumbnails” command. I tried the command as found in https://github.com/nextcloud/gallery/wiki/Create-Thumbnails using both:

./occ gallery:create-thumbnails (as indicated on the wiki page)

and
./occ galleryplus:create-thumbnails (thinking that the galleryplus directory might be the right one instead of gallery)

Both produced the same error:
"There are no commands defined in the “gallery” namespace."
and “There are no commands defined in the “galleryplus” namespace.”

I dug into both gallery and galleryplus directories and cannot find any mention of “create-thumbnails”. Do I need to install or access the backports in another way?

Thanks again for your help.

Mike

@oparoz can you help?

Thanks for the feedback. That thumbnail generating command never made it into production, but we’re talking about maybe integrating it into core as well as using a background job.

1 Like

OK, thanks for the information. I will look forward to seeing this feature in a future update!

Mike

Doesn’t it make sense to generate the preview after upload of file was successful?

1 Like

Normally, yes. But I don’t upload photos in the usual way. I have attached an external drive with a few thousand images and need an efficient way to produce the thumbs. In my case, I need a way to generate the thumbnails “behind the scenes” or perhaps “on demand”.

1 Like

This seems plausible. I do not know how new files on your HD are discovered by the files app, but maybe the discover could be the point of time where you create such thumbnails!? This could also decrease the amount of thumbnails to be created each time.

I use the “External storage support” app, which works very well for me to access my files in File view. Accessing the external mount is transparent… only the folder icon of the new drive is somewhat different in Files view. Everything else feels the same so far as I can tell (although I haven’t explored all the options).

My problem is that after I attach the drive, I cannot use Gallery view at all… it hangs, or at least it appears to, and I cannot get past the root folder as the spinning wheel I’m sure is continuing to try (or failing) to create the thumbnails (there’s no feedback, so I don’t know what it’s really doing… I can’t find anything in the logs about it). I’ve let it continue with the “spinning circle” for hours, and still it hasn’t finished its work. Thus, I don’t think the moment of drive attachment would be the right point to generate the thumbs, at least not in the same manner as now.

This is why, at least in my case, I’d prefer to have the option of a separate or background or an “on demand” thumbnail creation. “Separate” would be something like the “create-thumbnails” feature mentioned in my earlier notes here. “Background” means it could create the thumbs behind the scenes, but still lets me go into Gallery view for those other folders which already have thumbs created, without interference by the thumbnail generation process. By “on demand” I mean the possibility to generate the thumbs only when the specific parent folder is clicked on. In my case, that would seldom involve more than 40 or 50 images, in contrast to having to bulk generate the other several thousand at one go.

The other thing which just occurred to me is that for all the images I would need to make thumbs for, after they are created the cache would probably be enormous. I’m running mysql which works well at the moment, but I imagine the database and php configurations would need to be considered, too, for handling such large numbers of images and their thumbnails. An additional consideration would be what happens if the external drive is ejected or unmounted. For me, it’s not an issue, since I am “permanently” mounting an separate partition on the server drive, and will not really ever think about ejecting it. But I’m sure other folks might attach a “real” separate hard drive.

Thanks for following up on this matter. It’s hardly a critical thing for the operation of Nextcloud/Owncloud, but solving it would facilitate the use of Gallery for those of us with mass quantities of images.

Mike

Quick question here. When using the occ command line to generate thumbnails… if the source picture(s) gets moved to a new folder within owncloud:

  1. Will a new thumbnail have to be generated for the source file after it’s moved?
  2. If a new thumbnail does indeed have to be generated, what happens to the old one?
  3. Where are these thumbnails stored and under what type of naming convention?

Thanks!
Mike

1 Like

No news?

I’m also trying to create thousands of thumbnails … no way, get this:

sudo -u www-data php ./occ gallery:create-thumbnails --a

[Symfony\Component\Console\Exception\CommandNotFoundException]
There are no commands defined in the “gallery” namespace.

NC 11.0, Gallery 16.0.0

Check a few posts above: Gallery too slow or hanging for large image collections - #8 by oparoz

Hi

I’ve found that it’s not necessarily the thumbnail generation that causes all of the problems (although I am experiencing the same issues mentioned above)

My folder layout for my personal photography from the last 15 years is roughly (in a local “external folder”):
Pics/
Pics/photos/
Pics/photos//

For reference, I’m running NextCloud 11 on apache2.4, php7.1, on an i7-3770 with 16gb ram, and 2x raided(mirrored) 2tb spinny disks

Just loading the top-level Pics folder, which only includes a single folder, but no photos, takes about 15 seconds, even on subsequent loads (i.e. after loading thumbs)

Loading the photos folder then takes about 40 seconds; then the thumbnail generation starts, async from the browser.

At a guess, it’s doing a deep tree scan on every page load?

Thanks!

Seb

Thank you for this information. Can you please share when this should be ready in production?

1 Like

+1. I just have a few hundred pictures and this section of NextCloud is impossibly slow.

this feature is badly needed. i have hundreds of photos in every folder and previews are tooo slow to load.

Agreed. The mobile gallery apps are very fast. I would love to be able to ditch Apple for storing my photos.

i installed the preview-generator-program from the “store” and (re-)generate all previews with a nightly cronjob. i have ~100GB of pictures and the cronjob takes ~150mins. the previews take up ~83GB (!!!) in /data/appdata_ocjecx05wt96/preview but the gallery really is amazingly fast with it (before that, it wasn’t useable at all.)
so if you have the diskspace and can spare some computing cycles, i’d say it’s really worth a try.
GOOD LUCK!