Is it possible to back up generated previews?

Hello,
I am currently exploring if NextCloud is a viable solution for my needs.

I plan to back up a lot of photos and would like to access them in a Google photos like manner. I have established that this is possible.

Here’s the deal: I have got all the photos on network storage (NAS), which I successfully mounted.

Upon opening one of the folders NextCloud started generating thumbnail previews for the photos, which takes a long time, given the number of files.

Is it possible to back those generated previews up in some way in case I mess up my instance, or something goes wrong so I don’t have to go through the process again?

Best was is to generate previews via cron, it will only than download generated one. Check this out: How to know if Preview Generator is working? - #6 by gas85

Also here is an old tread with config How to clean up nextcloud-data/appdata_* and /updater-* - #7 by gas85 and explanation to understand previews: How to clean up nextcloud-data/appdata_* and /updater-* - #9 by j0hn

1 Like

Therefore you must understand the mechanism used to store those previews.

The previews are located in your data directory:

  • "data/appdata_$instanceid/preview/*

First you should know that each file added to the nextcloud file cache receives its unique file_id.

Since the loading of directories with many files takes a long time and requires a lot of resources each time a contained file is opened, a mechanism is used for things like this where many files are created, that ensures that the (preview) files are distributed across many small, evenly distributed directories.
For this performance reasons, these are created in a hex tree. This hex tree has a directory depth of 7, which is created with the first 7 digits of the md5 hash of the file_id.

Let’s take a file with file ID 1234 as an example. The 128 bit long 32 digit hexadecimal number of the md-5 sum is then 81dc9bdb52d04dc20036dbd8313ed055

so this file path is then created (with the first 7 digits): 8/1/d/c/9/b/d

and because this type of path assignment represents a pseudo-random distribution with 268,435,456 different folder paths, which ensures a good, balanced distribution of the tree, it is at risk of collision. But because the file_id is also appended to this path as a folder, the path is absolutely uniq and a large, evenly distributed tree is possible with relatively short paths.

So the previews of the image with the file_id 1234 will be located inside of this folder:

  • "data/appdata_$instanceid/preview/8/1/d/c/9/b/d/1234

As you can easily see here, the entire path is no longer valid as soon as the images get a different file ID.
But when making a backup, you should definitely save all files together. That means files, database and appdata folder.
Then the preview images will also be retained.

This app does the generation of the hex tree before you open the file the first time.

App-Id previewgenerator
App-Name Preview Generator
Summary A preview pre-generation app
Categories multimedia
App-Version 5.4.0
Repository GitHub - nextcloud/previewgenerator: Nextcloud app to do preview generation in the background.
Issue-Tracker Issues · nextcloud/previewgenerator · GitHub
PHP min/max 8.0 / 8.3
NC min/max 26 / 28
Not-shipped (not included) App available in appstore
Appstore Preview Generator - Apps - App Store - Nextcloud

another posibility is to use imaginay for the preview generation. You can run it as a docker image like Nexctloud AIO does.


Much and good luck,
ernolf

2 Likes

Thanks @gas85 for the links and the scripts, and @ernolf for the awesome explanation of how preview generation works!

I have not had time to work on this more but wanted to say thank you in case this topic gets closed automatically.

With the information the two of you provided I should be able to get my NextCloud running nicely.

:100: