Nextcloud database server 100% load (on all cores) on multiple files upload!

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can. :heart:

The Basics

  • Nextcloud Server version (e.g., 29.x.x):
    • Nextcloud Hub 8 (29.0.7)
  • Operating system and version (e.g., Ubuntu 24.04):
    • Centos7. Also tried on Rocky 9.5
  • Web server and version (e.g, Apache 2.4.25):
    • Apache 2
  • PHP version (e.g, 8.3):
    • PHP 8.2.20
  • Is this the first time you’ve seen this error? (Yes / No):
    • Yes
  • When did this problem seem to first start?
    • Right after a clean installation (migration from owncloud)
  • Installation method (e.g. AlO, NCP, Bare Metal/Archive, etc.)
    • Native install on virtual server, with a separate database virtual server
  • Are you using CloudfIare, mod_security, or similar? (Yes / No)
    • No

Summary of the issue you are facing:

[Nextcloud creates 100% load on all 16 cores of the database server during uploading of a folder that contains multiple files (280) ]

Steps to replicate it (hint: details matter!):

  1. Fresh nextcloud install (actually previous migration from owncloud) without any addons other than “forms” and “onlyoffice”. No files encryption enabled. The database is on a similar virtual server with 16 cores and 16Gb of RAM.
  2. Try to upload through the web interface, a folder (88Mb with 280files in it)
  3. Nextcloud first pauses to create the folder tree. At that time there is no huge CPU load.
  4. When the actual uploading of the files begins, the load on all CPU cores in the database, is 100%. This causes the nextcloud instance for all users to be extremely slow. The nextcloud server is not loaded at that time, only the database server does.
  5. As a test Redis was installed and configured according to this Transactional file locking — Nextcloud latest Administration Manual latest documentation but again no luck!

Log entries

Nextcloud

Please provide the log entries from your Nextcloud log that are generated during the time of problem (via the Copy raw option from Administration settings->Logging screen or from your nextcloud.log located in your data directory). Feel free to use a pastebin/gist service if necessary.

tail -f <dir>/nextcloud.log does not have any entries during the upload operation.

Configuration

Nextcloud

The output of occ config:list system or similar is best, but, if not possible, the contents of your config.php file from /path/to/nextcloud is fine (make sure to remove any identifiable information!):

sudo -u apache php occ config:list system
{
    "system": {
        "debug": false,
        "maintenance": false,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            ....
        ],
        "roundcube_owncloud_des_key": "...",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:...
        "overwriteprotocol": "https",
        "dbtype": "mysql",
        "version": "29.0.7.1",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "ldapIgnoreNamingRules": false,
        "preview_libreoffice_path": "\/usr\/bin\/libreoffice -display :0  --headless",
        "loglevel": 3,
        "mail_smtpmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpauth": true,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "25",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379,
            "timeout": 0,
            "password": "***REMOVED SENSITIVE VALUE***"
        },
        "theme": "...",
        "skeletondirectory": "...",
        "xframe_restriction": false,
        "logfile": "...ud.log",
        "default_language": "el",
        "default_phone_region": "GR",
        "maintenance_window_start": 1,
        "enable_previews": false,
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trashbin_retention_obligation": "auto",
        "singleuser": false,
        "hide_login_form": true,
        "login.alternatives": [
            {
                "href": "\/index.php\/apps\/user_cas\/login",
                "name": "CAS Login",
                "img": "\/apps\/user_cas\/img\/cas-logo.png"
            }
        ],
        "allow_user_to_change_mail_address": "",
        "app_install_overwrite": [
            "user_cas"
        ],
        "mysql.utf8mb4": true
    }
}

Apps

sudo -u apache php occ app:list
Enabled:

  • activity: 2.21.1
  • admin_audit: 1.19.0
  • calendar: 4.7.16
  • circles: 29.0.0-dev
  • cloud_federation_api: 1.12.0
  • comments: 1.19.0
  • contacts: 6.0.0
  • contactsinteraction: 1.10.0
  • dashboard: 7.9.0
  • dav: 1.30.1
  • federatedfilesharing: 1.19.0
  • files: 2.1.1
  • files_external: 1.21.0
  • files_pdfviewer: 2.10.0
  • files_reminders: 1.2.0
  • files_sharing: 1.21.0
  • files_trashbin: 1.19.0
  • files_versions: 1.22.0
  • forms: 4.3.1
  • logreader: 2.14.0
  • lookup_server_connector: 1.17.0
  • mail: 3.7.8
  • notes: 4.11.0
  • notifications: 2.17.0
  • oauth2: 1.17.1
  • onlyoffice: 9.4.0
  • password_policy: 1.19.0
  • photos: 2.5.0
  • provisioning_api: 1.19.0
  • related_resources: 1.4.0
  • serverinfo: 1.19.0
  • settings: 1.12.0
  • sharebymail: 1.19.0
  • systemtags: 1.19.0
  • text: 3.10.1
  • theming: 2.4.0
  • twofactor_backupcodes: 1.18.0
  • updatenotification: 1.19.1
  • user_cas: 1.10.0
  • user_ldap: 1.20.0
  • user_status: 1.9.0
  • viewer: 2.3.0
  • workflowengine: 2.11.0
    Disabled:
  • bruteforcesettings: 2.9.0
  • encryption: 2.17.0
  • federation: 1.19.0 (installed 1.19.0)
  • files_downloadlimit: 2.0.0 (installed 2.0.0)
  • firstrunwizard: 2.18.0 (installed 1.1)
  • nextcloud_announcements: 1.18.0 (installed 1.18.0)
  • privacy: 1.13.0 (installed 1.13.0)
  • recommendations: 2.1.0 (installed 2.1.0)
  • support: 1.12.0 (installed 1.12.0)
  • survey_client: 1.17.0 (installed 1.17.0)
  • suspicious_login: 7.0.0
  • twofactor_totp: 11.0.0-dev
  • weather_status: 1.9.0 (installed 1.9.0)

When the actual uploading of the files begins, the load on all CPU cores in the database, is 100%. This causes the nextcloud instance for all users to be extremely slow. The nextcloud server is not loaded at that time, only the database server does.

Does the upload eventually complete / do things eventually calm down?

What specific version of either MySQL or MariaDB are you using?

Do you have any warnings or errors under Administration settings->Overview?

Nextcloud Hub 8 (29.0.7)

Any particular reason you’re not running the latest maintenance release?

"loglevel": 3,

You might try setting this to 2 (the default) just in case there

Can’t say user_cas is a contributing factor, but it’s notable given this app isn’t compatible with >Nc24

As a test Redis was installed and configured according to this Transactional file locking — Nextcloud latest Administration Manual latest documentation but again no luck!

I suggest also setting up APCu for memcache.local and using the Redis you set-up also for memcache.distributed while you’re at it as described here.

Have you optimized your MariaDB-settings? This can make quite a difference. There should be something in the admin manual and a lot in these forums.
GOOD LUCK!

΅When the uploading of the files is completed, everything returns to normal. All CPU cores in the database server are very lightly loaded.

I am using 10.11.11-MariaDB

I get only warnings in the Administration settings->Overview and they are are as follows:

The database is used for transactional file locking. To enhance performance, please configure memcache, if available. (* note that I still get this message no matter if I have enabled Redis. I verify the Redis activity by redis-cli MONITOR)

No memory cache has been configured. To enhance performance, please configure a memcache, if available.

The reason I am running Nextcloud Hub 8 (29.0.7) is that our live servers run this one (2 servers behind a load balancer and a third database server)
We usually update every few months the live servers, following a successful update of the development server (the one referenced on this thread). Our live installation has more than 70.000 users and we can only use the community edition due to limited budget.

user_cas is patched for the current version (available on the internet). It works fine but there are these incompatibility messages.

At least in the config.php you posted it’s only enabled for 'memcache.locking', but not for 'memcache.local'.

However please read the warning in the documentation before enabling it. It might be better to use APCu as a local file cache, instead of Redis: Memory caching — Nextcloud latest Administration Manual latest documentation

A complete configuration with APCu as a local memcache would then look like this:

'memcache.local' => '\OC\Memcache\APCu',
'memcache.locking' => '\OC\Memcache\Redis',
'memcache.distributed' => '\OC\Memcache\Redis',
 "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379,
            "timeout": 0,
            "password": "***REMOVED SENSITIVE VALUE***"
        },

Furthermore, if you’re using PHP-FPM, the PHP process manager comes to mind. I have no experience with instances the size of yours, but I had a similar problem some time ago and it helped me to set it to:

pm = ondemand

In Debian/Ubuntu, this option can be found in:

/etc/php/"$PHPVERSION"/fpm/pool.d/www.conf

And in general, Carsten Rieger’s instructions helped me a lot when it came to PHP and database configuration. The instructions are in German, but you might still want to take a look at his PHP settings and the database config, even if you are not a native speaker: :slight_smile:

Apache:

nginx: