Backup Restore takes forever because of previews

ℹ️ Support 📦 Appliances (Docker, Snappy, VM, NCP, AIO)
,
1

I have been using nextcloud for quiet a while now and i love it. Yesterday, i had to move the data to a different drive. I used the opportunity to test if my backup would work. So i tried to restore the backup to the empty nfs drive.

The issue im facing:

during the restoration of my backup, i noticed that it takes forever to restore the huuge preview folder structure. It took at least 3 Hours (!!) to restore <5G of Preview data.

Why is the preview data organized this shitty way? its up to 11 folders deep.

a small excerpt from the log:

2024-10-06T21:22:50.369674438Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/9/d/
2024-10-06T21:22:50.369677848Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/9/d/0/
2024-10-06T21:22:50.369681587Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/9/d/0/d/
2024-10-06T21:22:50.369685227Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/9/d/0/d/e/
2024-10-06T21:22:50.369688567Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/9/d/0/d/e/76687/
2024-10-06T21:22:50.369691958Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/9/d/0/d/e/76687/1024-768.jpg
2024-10-06T21:22:50.424101292Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/9/d/0/d/e/76687/2048-1536-max.jpg
2024-10-06T21:22:50.510766187Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/9/d/0/d/e/76687/256-256-crop.jpg
2024-10-06T21:22:50.523123705Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/9/d/0/d/e/76687/341-256.jpg
2024-10-06T21:22:50.538018890Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/9/d/0/d/e/76687/64-64-crop.jpg
2024-10-06T21:22:50.538046751Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/9/e/
2024-10-06T21:22:50.547132755Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/9/e/8/
2024-10-06T21:22:50.565905055Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/9/e/8/2/
2024-10-06T21:22:50.639582021Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/9/e/8/2/0/
2024-10-06T21:22:50.663158809Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/9/e/8/2/0/225959/
2024-10-06T21:22:50.663187370Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/9/e/8/2/0/225959/2048-2048-max.png
2024-10-06T21:22:50.815917115Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/9/e/8/2/0/225959/256-256.png
2024-10-06T21:22:50.962313323Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/
2024-10-06T21:22:50.962345384Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/0/
2024-10-06T21:22:50.962349234Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/0/3/
2024-10-06T21:22:50.982961321Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/0/3/f/
2024-10-06T21:22:51.006445259Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/0/3/f/f/
2024-10-06T21:22:51.017936202Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/0/3/f/f/227022/
2024-10-06T21:22:51.026861305Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/0/3/f/f/227022/2048-2048-max.png
2024-10-06T21:22:51.151544045Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/0/3/f/f/227022/256-256.png
2024-10-06T21:22:51.167174083Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/0/9/
2024-10-06T21:22:51.211820131Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/0/9/f/
2024-10-06T21:22:51.211846891Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/0/9/f/d/
2024-10-06T21:22:51.446209062Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/0/9/f/d/260491/
2024-10-06T21:22:51.470426893Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/0/9/f/d/260491/1024-1536.jpg
2024-10-06T21:22:51.531629031Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/0/9/f/d/260491/1365-2048-max.jpg
2024-10-06T21:22:51.595027950Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/0/9/f/d/260491/256-384.jpg
2024-10-06T21:22:51.613770071Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/0/9/f/d/260491/43-64.jpg
2024-10-06T21:22:51.641113506Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/0/9/f/d/260491/683-1024.jpg
2024-10-06T21:22:51.641141365Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/2/
2024-10-06T21:22:51.641145145Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/2/0/
2024-10-06T21:22:51.641148615Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/2/0/6/
2024-10-06T21:22:51.698918712Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/2/0/6/1/
2024-10-06T21:22:51.735973041Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/2/0/6/1/261639/
2024-10-06T21:22:51.757942633Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/2/0/6/1/261639/1365-2048-max.jpg
2024-10-06T21:22:51.815068439Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/2/0/6/1/261639/171-256.jpg
2024-10-06T21:22:51.815097569Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/5/
2024-10-06T21:22:51.815101319Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/5/8/
2024-10-06T21:22:51.815104779Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/5/8/8/
2024-10-06T21:22:51.815108119Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/5/8/8/1/
2024-10-06T21:22:51.843446094Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/6/
2024-10-06T21:22:51.911351069Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/6/5/
2024-10-06T21:22:51.942034205Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/6/5/6/
2024-10-06T21:22:51.942062676Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/6/5/6/f/
2024-10-06T21:22:52.046128626Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/9/
2024-10-06T21:22:52.046157925Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/9/d/
2024-10-06T21:22:52.046161855Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/9/d/1/
2024-10-06T21:22:52.046165335Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/9/d/1/0/
2024-10-06T21:22:52.066856593Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/b/
2024-10-06T21:22:52.083337405Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/b/b/
2024-10-06T21:22:52.121613249Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/b/b/c/
2024-10-06T21:22:52.165616985Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/b/b/c/2/
2024-10-06T21:22:52.165646196Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/b/b/c/2/235061/
2024-10-06T21:22:52.215547873Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/b/c/
2024-10-06T21:22:52.283228858Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/b/c/b/
2024-10-06T21:22:52.283256737Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/b/c/b/d/
2024-10-06T21:22:52.338415135Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/c/
2024-10-06T21:22:52.338454085Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/c/4/
2024-10-06T21:22:52.338459436Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/c/4/b/
2024-10-06T21:22:52.338464156Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/c/4/b/d/
2024-10-06T21:22:52.338468756Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/c/4/b/d/236737/
2024-10-06T21:22:52.338473225Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/d/
2024-10-06T21:22:52.338477575Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/d/e/
2024-10-06T21:22:52.338482055Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/d/e/5/
2024-10-06T21:22:52.383414804Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/d/e/5/0/
2024-10-06T21:22:52.383440364Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/d/e/5/0/211203/
2024-10-06T21:22:52.414590560Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/e/
2024-10-06T21:22:52.454913253Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/e/8/
2024-10-06T21:22:52.508271503Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/e/8/1/
2024-10-06T21:22:52.508306194Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/e/8/1/c/
2024-10-06T21:22:52.650831270Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/e/8/1/c/93702/
2024-10-06T21:22:52.661524921Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/e/8/1/c/93702/1536-2048-max.jpg
2024-10-06T21:22:52.661555550Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/e/8/1/c/93702/256-256-crop.jpg
2024-10-06T21:22:52.661575581Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/e/8/1/c/93702/256-341.jpg
2024-10-06T21:22:52.661579181Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/e/8/1/c/93702/64-64-crop.jpg
2024-10-06T21:22:52.718320963Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/e/8/1/c/93702/768-1024.jpg
2024-10-06T21:22:52.730397519Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/f/
2024-10-06T21:22:52.730429640Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/f/5/
2024-10-06T21:22:52.730433740Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/f/5/1/
2024-10-06T21:22:52.730437289Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/f/5/1/5/
2024-10-06T21:22:52.749765292Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/f/5/1/5/463731/
2024-10-06T21:22:52.749795462Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/f/5/1/5/463731/1536-2048-max.jpg
2024-10-06T21:22:53.242853155Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/f/5/1/5/463731/192-256.jpg
2024-10-06T21:22:53.269492014Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/f/5/1/5/463731/256-256-crop.jpg
2024-10-06T21:22:53.269519785Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/f/5/1/5/463731/256-341.jpg
2024-10-06T21:22:53.304587648Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/f/5/1/5/463731/48-64.jpg
2024-10-06T21:22:53.327833864Z nextcloud_aio_nextcloud_data/appdata_occny3ajwvk3/preview/3/7/a/f/5/1/5/463731/768-1024.jpg

i can’t even calculate the exact size of this preview/ directory because its so gigantic (Not the size in GB, i mean the ammount of folders and subfolders) that it takes forever to calculate.

right now, ncdu is running for approx. 1h and it counted only 590 MiB…

Could someone explain to me why this is organised this way? or is it just a dumb configuration error?

As i am using AIO, it is not possible to exclude the previews from the backup right? what would you guys suggest?

2

Hi, I guess we would need to add a flag to the aio interface that allows to skip the preview directory when restoring a backup. Can you please create a feature request here: Sign in to GitHub · GitHub. Thanks!

3

It’s not a configuration error. The folder structure is deliberate. It reduces the quantity of files and folders that would otherwise be in the parent preview folder and ensures that each underlying folder is kept rather small.

So i tried to restore the backup to the empty nfs drive.

Most of your performance problems are likely due to NFS being in the mix.

The many small files + traversing the deep hierarchy is not ideal for NFS (to put it mildly).

There probably isn’t much downside to excluding previews from restore processes. I suppose the same could be said for backing previews up to start with…

1 Like
4

That would be great but i have read a couple of times that the preview folder of some folks exceed 50+ GB. Wouldnt it be better to add an exclude from the backup option?

5

Basically, I don’t find Nextcloud’s preview function ideal. There are settings options, see here, but every image is saved in its original size, for example. It would be nice if you could only create the two previews for the file view. That would be perfectly adequate for the most users.

Perhaps the simplest solution would be to simply not backup the previews and therefore not restore them. Of course, you can also directly disable previews.

As someone once said so beautifully: Nextcloud is not a photo programme. But I think the preview software could also load at least the original image directly from the original folder. Maybe someone would like to create an issue.

6

yes, that would be nice. But it would be great to set a maximum quota for the image previews e.g. 20GB.

7

I think that is not possible. But maybe you like to set e.g.:

  'preview_max_x' => 1024,
  'preview_max_y' => 1024,

I think that is a good restriction.

Also you can use to update:

sudo -u www-data php occ preview:delete_old
sudo -u www-data php occ preview:generate-all

You can also read this thread. I do not fully unterstand it. I also find this comment in an issue.