Large delay & confusion in Thumbnail Generation for Photos in External Storage

JackDinn · May 4, 2024, 10:21am

Hello Nextcloud Community,

I’m encountering an issue with delayed thumbnail generation for photos added via SMB and stored in an externally mounted storage directory that is mounted & accessed via Nextcloud. Despite several attempts to troubleshoot and correct the problem, thumbnails for newly added photos either take a very long time to appear or don’t show up at all without manual intervention (basically me tinkering around with all the commands i can think of, refreshing things, logging out/in etc. Eventually the photo MIGHT show up, it might not).

Environment:

Nextcloud version: 28.0.5.1
External Storage: Mounted at /mnt/cavern
Access Method: SMB (Samba) on Linux
Server OS: Truenas, FreeBSD
Web server: Caddy v2.6.4
PHP version: PHP 8.1.17

Configuration:

External storage is configured through Nextcloud with settings to check for changes "Every time accessed."

What I’ve Tried:

I tried both Photos app and Memories app

Manual Scanning:
    I've used the occ files:scan command for specific paths and the entire external storage, but it does not recognize new files immediately, if at all.
    Example command: sudo -u www php /usr/local/www/nextcloud/occ files:scan --path="admin/files/cavern"

Thumbnail Generation:
    Attempted to force thumbnail generation using occ preview:generate-all, specifically targeting new photo directories.
    The logs show the commands execute but do not lead to immediate thumbnail availability.

Log Analysis:
    Checked nextcloud.log for errors. I see no errors, it never actually mentions the file/s in question.

Permissions and Cache:
    Ensured that file permissions are correct and tried clearing both Nextcloud and Redis caches.
    Restarted the web server post-configuration changes.

All the commands i have tried:

su -m www -c 'php /usr/local/www/nextcloud/occ maintenance:repair'
su -m www -c 'php /usr/local/www/nextcloud/occ preview:generate-all'
su -m www -c 'php /usr/local/www/nextcloud/occ files:cleanup'
su -m www -c 'php /usr/local/www/nextcloud/occ preview:generate-all --path="admin/files/cavern"'
su -m www -c 'php /usr/local/www/nextcloud/occ files:scan --path="admin/files/cavern"'

Despite these efforts, the issue persists. Thumbnails for new images either don’t generate or take days to appear in the Nextcloud interface. I’m looking for insights or suggestions on how to ensure that thumbnails are generated more reliably and quickly after new files are added to the external storage.

Has anyone else experienced similar issues, or does anyone have recommendations on how to troubleshoot this further? Any help or guidance would be greatly appreciated!

Thank you!

Cisco30 · May 4, 2024, 10:48am

hi
you need to install the preview generator app
then you carry out a first manual scan

sudo -u www-data php /var/www/nextcloud/occ preview:generate-all -vvv

and then added a CRON task
crontab -e
and add the following line to refresh to generate new thumbnails every 10 min
*/10 * * * * sudo -u www-data php -f /var/www/nextcloud/occ preview:pre-generate

JackDinn · May 4, 2024, 11:56am

I have tried.

I put a photo into the “odd asortments” folder.

I ran the occ preview:generate-all on just that specific folder (just for time saving)

preview:generate-all --path="/admin/files/cavern/PROGRAMDATA/MY Docs/My Photos/odd asortments

I grep’ed the specific photo file name and you can see it says it has indexed it.

But photo is not there, tried refreshing page , restaring browser, login/out of NC, clear cache etc. even restarted NC jail. Still not there.

It does show in the NC files though, just not memories.

I try the occ preview:pre-generate

Its still not there.

root@nextcloud:~ su -m www -c 'php /usr/local/www/nextcloud/occ preview:generate-all --path="/admin/files/cavern/PROGRAMDATA/MY Docs/My Photos/odd asortments" -vvv' | grep '0024'
2024-05-04T12:02:46+01:00 Generating previews for /admin/files/cavern/PROGRAMDATA/MY Docs/My Photos/odd asortments/20240504_120024.jpg
root@nextcloud:~ su -m www -c 'php /usr/local/www/nextcloud/occ preview:pre-generate -vvv'

There is no point in setting up a cron until i can see that whatever commands are going to work manually first.

What should i try next?

P.S.

I just refreshed the memories page again before posting this and now it is there, but why? I refreshed it 20 times after the last occ command and it didn’t show up. Probably an hour after i ran the occ commands.

Its been doing this for days, it just seems to randomly appear?

Cisco30 · May 4, 2024, 2:34pm

have you enabled “enable previews” and “enabledPreviewProviders” in config.php? I posted my config.php to you.

{
    "instanceid": "***REMOVED SENSITIVE VALUE***",
    "passwordsalt": "***REMOVED SENSITIVE VALUE***",
    "secret": "***REMOVED SENSITIVE VALUE***",
    "trusted_domains": [
        "nextcloud.domain.cloud",
        "www.nextcloud.domain.cloud"
    ],
    "trusted_proxies": "***REMOVED SENSITIVE VALUE***"
    "datadirectory": "***REMOVED SENSITIVE VALUE***",
    "dbtype": "mysql",
    "version": "28.0.5.1",
    "overwrite.cli.url": "https:\/\/nextcloud.domain.cloud",
    "dbname": "***REMOVED SENSITIVE VALUE***",
    "dbhost": "***REMOVED SENSITIVE VALUE***",
    "dbport": "",
    "dbtableprefix": "oc_",
    "mysql.utf8mb4": true,
    "dbuser": "***REMOVED SENSITIVE VALUE***",
    "dbpassword": "***REMOVED SENSITIVE VALUE***",
    "installed": true,
    "default_phone_region": "FR",
    "htaccess.RewriteBase": "\/",
    "trashbin_retention_obligation": "auto, 30",
    "filelocking.enabled": "true",
    "knowledgebaseenabled": false,
    "activity_expire_days": 7,
    "skeletondirectory": "\/var\/www\/extra_nextcloud\/skeleton\/",
    "redis": {
        "host": "***REMOVED SENSITIVE VALUE***",
        "port": "0",
        "timeout": "0.0"
    },
    "memcache.distributed": "\\OC\\Memcache\\Redis",
    "memcache.local": "\\OC\\Memcache\\Redis",
    "memcache.locking": "\\OC\\Memcache\\Redis",
    "mail_smtpmode": "smtp",
    "mail_smtpsecure": "ssl",
    "mail_sendmailmode": "smtp",
    "mail_from_address": "***REMOVED SENSITIVE VALUE***",
    "mail_domain": "***REMOVED SENSITIVE VALUE***",
    "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpport": "465",
    "mail_smtpauth": 1,
    "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
    "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
    "default_language": "fr",
    "default_locale": "fr_FR",
    "updater.release.channel": "stable",
    "maintenance_window_start": 1,
    "maintenance": false,
    "enable_previews": true,
    "enabledPreviewProviders": {
        "1": "OC\\Preview\\BMP",
        "2": "OC\\Preview\\GIF",
        "3": "OC\\Preview\\JPEG",
        "4": "OC\\Preview\\Krita",
        "5": "OC\\Preview\\MarkDown",
        "6": "OC\\Preview\\MP3",
        "7": "OC\\Preview\\OpenDocument",
        "8": "OC\\Preview\\PNG",
        "9": "OC\\Preview\\TXT",
        "10": "OC\\Preview\\XBitmap",
        "11": "OC\\Preview\\Image",
        "12": "OC\\Preview\\HEIC",
        "13": "OC\\Preview\\TIFF",
        "14": "OC\\Preview\\Movie",
        "15": "OC\\Preview\\MP4",
        "16": "OC\\Preview\\PDF"
    },
    "preview_max_x": 1024,
    "preview_max_y": 1024,
    "preview_concurrency_all": 32,
    "preview_concurrency_new": 16,
    "preview_max_filesize_image": 200,
    "preview_max_memory": 1024,
    "preview_ffmpeg_path": "\/usr\/bin\/ffmpeg",
    "theme": "",
    "loglevel": 2,
    "defaultapp": "",
}

JackDinn · May 4, 2024, 2:51pm

Hmm, interesting. This is my partial config.php with all the sections invloved with Preview

'enable_previews' => true,
'enabledPreviewProviders' => 
  array (
    1 => 'OC\\Preview\\PDF',
    2 => 'OC\\Preview\\Image',
    3 => 'OC\\Preview\\Photoshop',
    4 => 'OC\\Preview\\TIFF',
    5 => 'OC\\Preview\\SVG',
    6 => 'OC\\Preview\\Movie',
    7 => 'OC\\Preview\\MKV',
    8 => 'OC\\Preview\\MP4',
    9 => 'OC\\Preview\\AVI',
  ),

I have seen that i dont have jpeg, jpg or bmp , quite a few missing for some reason. is this going to be the problem? if so then why do the pics show up eventually at all?

EDIT: Ok i tried adding the JPEG and other file types but it didn’t help

JackDinn · May 6, 2024, 7:46am

I found that if you edit a file then its even worse.

To test i created 2 image files:

– & –

Firstly, just for the hell of it i did run occ preview:generate-all
Then added this cron */15 * * * * su -m www -c ‘php /usr/local/www/nextcloud/occ preview:pre-generate -vvv’
I added the “OLD” file to the external storage folder.
I tinkered around with occ commands, refreshing the page etc etc until that image file showed up in the thumbnails. (it was about 8 hours before it showed up).
Then i edited/replaced the “OLD” file with the “NEW” file to test altering a file in the storage folder.
I then just checked NC every few hours to see if the NEW files thumbnail had been generated yet.
I tried all the usual things mentioned in the thread above to try to get it to recognise the new file.

It has been left for a couple of days now.

The thumbnail is still showing the OLD file in both the files and memories.

However, if i click on the OLD thumbnail and bring up the picture it will firstly show the OLD file but then if i zoom in it then suddenly switched to the NEW file, then if i close that view it goes back to showing the OLD file.

Heres a screen grab

So thats nuts ¯\(ツ)/¯

So we are back to my original question. How do i force it to recognise that the file has changed?