High CPU usage all of a sudden (NC 30)

The Basics

  • Nextcloud Server version (e.g., 29.x.x):
    • nextcloud:30-fpm-alpine
  • Operating system and version (e.g., Ubuntu 24.04):
    • Ubuntu 22.04
  • Web server and version (e.g, Apache 2.4.25):
    • nginx-mainline
  • Reverse proxy and version _(e.g. nginx 1.27.2)
    • nginx-mainline
  • PHP version (e.g, 8.3):
    • as supplied in Docker image
  • Is this the first time you’ve seen this error? (Yes / No):
    • yes
  • When did this problem seem to first start?
    • one week ago, exactly at 10.30am
  • Installation method (e.g. AlO, NCP, Bare Metal/Archive, etc.)
    • Docker-Compose setup: fpm-alpine, nginx, MariaDB as supported and listed on that GitHub page.
  • Are you using CloudfIare, mod_security, or similar? (Yes / No)
    • no

Summary of the issue you are facing:

Everything was running fine until last week, 10:30am, when the CPU usage suddenly spiked and made the application almost unusable.

I looked into the application logs but there’s no clear reason for this behaviour.

There was one entry for the database with an integerity constraint violation for a file within a groupfolder (which we use heavily) but I solved that by deleting the file in question und purging the trashbin of that groupfolder.

I’m looking for a hint where to look. What can possibly cause this? There must be something wrong with the database because it happened so suddenly and when I put the application into maintenance mode, the CPU load drops heavily.

I should add this is a VM running on a potent server: four CPU cores, 16 GB RAM, fast SSD.

Steps to replicate it (hint: details matter!):

  1. There was no change in the setup whatsoever
  2. no updates
  3. no change in application usage
  4. I suspect sync problems between Groupfolders and maybe the desktop app.

Log entries

Nextcloud

Please provide the log entries from your Nextcloud log that are generated during the time of problem (via the Copy raw option from Administration settings->Logging screen or from your nextcloud.log located in your data directory). Feel free to use a pastebin/gist service if necessary.

There's nothing clear even with debug on. But I wil provide the logs if requested.

Web Browser

If the problem is related to the Web interface, open your browser inspector Console and Network tabs while refreshing (reloading) and reproducing the problem. Provide any relevant output/errors here that appear.

PASTE

Web server / Reverse Proxy

The output of your Apache/nginx/system log in /var/log/____:

PASTE HERE

Configuration

Nextcloud

The output of occ config:list system or similar is best, but, if not possible, the contents of your config.php file from /path/to/nextcloud is fine (make sure to remove any identifiable information!):

{
    "system": {
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "forwarded_for_headers": [
            "HTTP_X_FORWARDED",
            "HTTP_FORWARDED_FOR"
        ],
        "trusted_domains": [
            "instance.domain.topdomain.de",
            "10.31.23.3",
            "localhost"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "30.0.11.1",
        "overwrite.cli.url": "https:\/\/localhost",
        "overwriteprotocol": "https",
        "overwritehost": "instance.domain.topdomain.de",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "skeletondirectory": "",
        "default_language": "de",
        "default_locale": "de_DE",
        "installed": true,
        "default_phone_region": "DE",
        "trashbin_retention_obligation": "30, 35",
        "auth.bruteforce.protection.enabled": false,
        "app_install_overwrite": [
            "files_clipboard",
            "files_lock",
            "dropit",
            "admin_notifications",
            "ocr",
            "customproperties",
            "defaultgroup",
            "uploaddetails",
            "groupfolders"
        ],
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "25",
        "maintenance": false,
        "loglevel": 2,
        "maintenance_window_start": 1,
        "upgrade.disable-web": true
    }
}

Apps

Enabled:

  • activity: 3.0.0
  • admin_audit: 1.20.0
  • app_api: 4.0.6
  • auto_groups: 1.6.2
  • bruteforcesettings: 3.0.0
  • cloud_federation_api: 1.13.0
  • comments: 1.20.1
  • dav: 1.31.1
  • federatedfilesharing: 1.20.0
  • files: 2.2.0
  • files_downloadlimit: 3.0.0
  • files_pdfviewer: 3.0.0
  • files_reminders: 1.3.0
  • files_sharing: 1.22.0
  • files_trashbin: 1.20.1
  • files_versions: 1.23.0
  • groupfolders: 18.1.2
  • impersonate: 1.17.1
  • logreader: 3.0.0
  • lookup_server_connector: 1.18.0
  • nextcloud_announcements: 2.0.0
  • oauth2: 1.18.1
  • password_policy: 2.0.0
  • provisioning_api: 1.20.0
  • related_resources: 1.5.0
  • serverinfo: 2.0.0
  • settings: 1.13.0
  • sharebymail: 1.20.0
  • theming: 2.6.0
  • theming_customcss: 1.18.0
  • twofactor_backupcodes: 1.19.0
  • twofactor_totp: 12.0.0-dev
  • updatenotification: 1.20.0
  • user_retention: 1.14.1
  • viewer: 3.0.0
  • webhook_listeners: 1.1.0-dev
  • whiteboard: 1.0.5
  • workflowengine: 2.12.0

Disabled:

  • admin_notifications: 1.0.2 (installed 1.0.2)
  • breezedark: 29.0.0 (installed 29.0.0)
  • circles: 30.0.0 (installed 22.1.0)
  • contactsinteraction: 1.11.0 (installed 1.2.0)
  • customproperties: 2.0.4 (installed 2.0.4)
  • dashboard: 7.10.0 (installed 7.1.0)
  • data_request: 2.0.1 (installed 2.0.1)
  • defaultgroup: 0.3.0 (installed 0.3.0)
  • dropit: 0.4.0 (installed 0.4.0)
  • encryption: 2.18.0
  • extract: 1.3.6 (installed 1.3.6)
  • federation: 1.20.0 (installed 1.11.0)
  • files_accesscontrol: 1.20.1 (installed 1.20.1)
  • files_antivirus: 5.6.3 (installed 5.6.3)
  • files_clipboard: 0.7.1 (installed 0.7.1)
  • files_downloadactivity: 1.18.1 (installed 1.18.1)
  • files_external: 1.22.0
  • files_retention: 1.19.1 (installed 1.19.1)
  • files_rightclick: 0.15.1 (installed 1.6.0)
  • firstrunwizard: 3.0.0 (installed 2.10.0)
  • flow_notifications: 1.10.2 (installed 1.10.2)
  • metadata: 0.22.0 (installed 0.22.0)
  • notifications: 3.0.0 (installed 2.13.1)
  • photos: 3.0.2 (installed 1.3.0)
  • privacy: 2.0.0 (installed 1.7.0)
  • quickaccesssorting: 3.0.0 (installed 3.0.0)
  • quota_warning: 1.21.0 (installed 1.21.0)
  • recommendations: 3.0.0 (installed 1.0.0)
  • socialsharing_email: 3.3.0 (installed 3.3.0)
  • support: 2.0.0 (installed 1.6.0)
  • survey_client: 2.0.0 (installed 1.9.0)
  • suspicious_login: 8.0.0
  • systemtags: 1.20.0 (installed 1.11.0)
  • text: 4.1.0 (installed 3.2.0)
  • twofactor_nextcloud_notification: 4.0.0
  • uploaddetails: 0.1.3 (installed 0.1.3)
  • user_ldap: 1.21.0
  • user_status: 1.10.0 (installed 1.1.1)
  • weather_status: 1.10.0 (installed 1.1.0)
  • workflow_script: 1.15.0 (installed 1.15.0)
  • workspace: 4.0.4 (installed 4.0.4)

I’m new to Nextcloud myself, but I do notice those spikes and it becomes more predominant on a Raspberry Pi so I am curious how you would resolve it.

In my case I tried to break apart the work from NextCloud as much as possible such as using Imaginary , PgBouncer and Redis. I even configured some static content caching on Caddy for the JS, and CSS files along with avatars.

But it does lag a bit on heavy load of one user with a family album of 30+ years of content. (Which is to be expected)

Unfortunately, I’m not experiencing spikes but a persistent high CPU usage that wasn’t there before and started probably when the cron job ran? The sudden spike started at precisely 10:30:00 am.

I should add I’m using the setup from https://github.com/nextcloud/docker/tree/master/.examples/docker-compose/insecure/mariadb/fpm

After more investigation and stopping all external clients from polling the server, the load dropped instantly. Even though there is barely any load on the server the uploads and the UI (as far as database calls are involved) are sluggish.

What can cause this in the database?

Hmm in my case I used Postgresql rather than MariaDB. When I originally did it I kept on hitting connection issues and found out that PHP doesn’t really do “connection pooling” like I am used to in Java land.

In which case I modified my architecture to have PgBouncer to handle the connection pooling. I wonder if MariaDB/MySQL has something similar otherwise you just have to increase the capacity of the database.

I am almost sure that my problem is caused by a database issue in conjunction with the Group Folders app.

I suspect there are database querys which are extremely slow.

SELECT `f`.`folder_id`, `mount_point`, `quota`, `acl`, `c`.`fileid`, `c`.`storage`, `c`.`path`, `c`.`name`, `c`.`mimetype`, `c`.`mimepart`, `c`.`size`, `c`.`mtime`, `c`.`storage_mtime`, `c`.`etag`, `c`.`encrypted`, `c`.`parent`, `a`.`permissions` AS `group_permissions`, `c`.`permissions` AS `permissions` FROM `oc_group_folders` `f` INNER JOIN `oc_group_folders_groups` `a` ON `f`.`folder_id` = `a`.`folder_id` LEFT JOIN `oc_filecache` `c` ON (`c`.`name` = CONCAT(`f`.`folder_id`, '')) AND (`c`.`parent` = '687') AND (`c`.`storage` = '1') WHERE `a`.`group_id` IN ('AVA-Formularbibliothek-Lesend', 'Jeder')                           | 1596.749 |     0 |         0 |    0.000 |      164512 |         2258920 |             0 |  6596798 | SELECT `f`.`folder_id`, `mount_point`, `quota`, `acl`, `c`.`fileid`, `c`.`storage`, `c`.`path`, `c`.`name`, `c`.`mimetype`, `c`.`mimepart`, `c`.`size`, `c`.`mtime`, `c`.`storage_mtime`, `c`.`etag`, `c`.`encrypted`, `c`.`parent`, `a`.`permissions` AS `group_permissions`, `c`.`permissions` AS `permissions` FROM `oc_group_folders` `f` INNER JOIN `oc_group_folders_groups` `a` ON `f`.`folder_id` = `a`.`folder_id` LEFT JOIN `oc_filecache` `c` ON (`c`.`name` = CONCAT(`f`.`folder_id`, '')) AND (`c`.`parent` = '687') AND (`c`.`storage` = '1') WHERE `a`.`group_id` IN ('One-Folder-Lesend', 'Jeder')

Is there a way to detect a problem in the group folder tables or probably in the oc_filecache table?

since you’re using mariadb you can use the slow logs capability of the DB

SET global slow_query_log = 1;
SET global log_output = 'table';
SET global long_query_time= 0.5;

select * from mysql.slow_log;

I want to provide some feedback concerning the solution to our problem: We solved it by adjusting several parameters of the MariaDB. InnoDB’s innodb_buffer_pool_size parameter for example and some others as suggested by the usual calculators for optimal database settings.

Suddenly our problem disappeared. The database has been running with the standard settings for years. It was confusing that all of a sudden it started to create problems.

1 Like

normally with databases, as data situation changes the tuning parameters tend to need updates. Also with better hardware you can afford more RAM at times.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.