Slow PHP cron job with SMB share (more than 300 hours)

[Encourage4525]

Support intro

Sorry to hear you’re facing problems

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can

Nextcloud version (eg, 20.0.5): 28.0.0
Operating system and version (eg, Ubuntu 20.04): Debian 12
Apache or nginx version (eg, Apache 2.4.25): apache
PHP version (eg, 7.4): 8.2.11

The issue you are facing:
I have an SMB storage attached to the nextcloud. And since the attache the CRON jobs takes a long time. The longest I’ve waited for the end of the run 300+ something hours. But I had to kill that job. It causes high CPU usage on the Nextcloud server. If I mount the same storage into the LXC with mountpoint there is no problem, everything runs perfectly.

Is this the first time you’ve seen this error? (Y/N):N

Steps to replicate it:

1. Install Nextcloud
2. Mount an SMB share (size: 12TB, rights: 775, auth: passwd, filled with a lot of small file) with the webUI admin interface use global stored password
3. Assign the mounted storage to a Group with 5-10 user
4. Wait for the cron jobs

The output of your Nextcloud log in Admin > Logging:

SRY MY LOGGING IS MESSED UP THE PAGE JUST CRASH ON LOAD

If anyone can point me to a direction how to fix this problem.

Best regards,
Mate

EDIT 1

I’ve found the occ command to export the reducted config:

    "system": {
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "cloud.zsolyahome.com",
            "172.16.1.89"
        ],
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "default_phone_region": "HU",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "28.0.0.11",
        "overwrite.cli.url": "https:\/\/cloud.zsolyahome.com",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "filelocking.enabled": true,
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "memcache.local": "\\OC\\Memcache\\Redis",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379,
            "timeout": 0,
            "password": "***REMOVED SENSITIVE VALUE***"
        },
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "log_type": "file",
        "logfile": "\/var\/log\/nextcloud.log",
        "loglevel": 1,
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_sendmailmode": "smtp",
        "mail_smtpsecure": "ssl",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpauth": 1,
        "mail_smtpport": "465",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "theme": "",
        "app_install_overwrite": [
            "activitylog",
            "files_bpm",
            "spreed",
            "files_texteditor",
            "talk_simple_poll",
            "files_markdown",
            "breezedark",
            "socialsharing_facebook",
            "files_fulltextsearch_tesseract",
            "files_fulltextsearch",
            "fulltextsearch",
            "fulltextsearch_elasticsearch",
            "event_update_notification",
            "audioplayer",
            "talk_matterbridge",
            "workflow_media_converter"
        ],
        "enable_previews": true,
        "enabledPreviewProviders": [
            "OC\\Preview\\Imaginary"
        ],
        "preview_imaginary_url": "http:\/\/172.16.1.94:5648",
        "updater.release.channel": "stable",
        "data-fingerprint": "f73dc831633a092bcdaa54861dd227fd",
        "remember_login_cookie_lifetime": 172800,
        "session_lifetime": 7200,
        "session_relaxed_expiry": false,
        "session_keepalive": true,
        "auto_logout": true,
        "token_auth_enforced": false,
        "token_auth_activity_update": 10,
        "onlyoffice": {
            "jwt_header": "AuthorizationJwt"
        }
    }

[tflidd]

Do you use the php-smbclient module?
https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/external_storage/smb.html

Can you check the size of your filecache-table? There are reports when a lot of changes happen from outside Nextcloud (which is expected on external storage), this can blow up the size of the table:

File cache table excessively large (and does not shrink after data removal) / Nextcloud should defragment file cache table

Also during the cronjob, can you check if the CPU is at the max limit, or the network connection, or database?

[jtr]

Please check your nextcloud.log directly for any clues.

I wonder if this is related to S3 external storage excessive network traffic - #4 by bjo ?

Is this new behavior after upgrading to NC28 or do you not have a comparison (I.e. new installation)?

The logreader (Web based access to logs) matter is likely either an ad blocker (browser extension) or misconfiguration in your environment for handling .mjs files (search elsewhere on the forum or refer to the NC27 release notes which provided updated web server configs for this).

[Encourage4525]

Hi!
Here is the filecache table size:

The CPU on the nextcloud server is at 100% on single thread.
The this configuration has 12 thread so after 12 days the server becomes unusable. (Can this behavior connected to the reddis server, because on the nextcloud I have it turned on.)
CPU usage of cloud server:

Memory usage of cloud server:

MYSQL database size:

I don’t see any unusual activity in mysql in terms of connectivity or the number of commands/queries/questions.

[Encourage4525]

Hi @jtr!

I’ve encountered with this behavior before even in NC26 but back then I could resolve it by mount the share through a simple folder pass-through into the LXC.
But now I try to add some migration feature to the LXC container (for improve reliability) and on the other node I cannot attach my share with folder mount, only by SMB.

I checked the logs, I had redis crash for the last entry on 2023.12.13. after that nothing.

THX for the

[tflidd]

@Encourage4525

Can this behavior connected to the reddis server,

If it is working, it should rather reduce the load and handle the caching tasks more efficiently. So if you turn it off, it should get worse.

@Encourage4525

Here is the filecache table size:

Looks not extremely large, you should have around 4 million entries (files).

@Encourage4525

CPU usage of cloud server:

The iowait time is something you can try to reduce by adjusting the mysql caches. If it is too small, the processing is often limited by the database server accessing the hard drive.
I can’t judge how the 1% average is enough to explain this alone.

Just by looking through the database structure, did you check the oc_jobs table, it lists all the jobs and also the time it took to run them, so you can identify the critical task.

[Encourage4525]

The iowait time is something you can try to reduce by adjusting the mysql caches.

Sry it looks like I forget to mention the DB is on different LXC and even node. So the usage of the cloud server CPU only include the PHP tasks and everything else what is running on that server, like redis. But nothing else. This LXC dedicated only for the NC server.

In the meantime I figured out I needed to run the following commands:

sudo -u www-data php occ files:cleanup
sudo -u www-data php occ files:scan --all

After that this concurrent running PHP cronjobs ended and it can be finished in correct time.

I’m assume the cronjob wanted to update the file cache and It’s not finnished when the next job started, and because the first not set some flag or some variable, it started to scan, or update the filetable…
THIS IS JUST ASSUMPTIONS. I’m not NC developer, I don’t know the inner architecture.

Thanks for the help, but this seems to be solved with the 2 commands above.
Br,
Mate Zsolya

[tflidd]

@Encourage4525

After that this concurrent running PHP cronjobs ended

In theory, I thought to have seen that there is a lock mechanism that should prevent this, but I’m not too much in the inner workings either to be sure.

If you don’t have to run this command to fix this issue again and again, I’d considered this to be fixed. If it happens on a regular basis, since it is external storage, there are perhaps to be addressed. So for the moment, I mark you last response as answer.