Nextcloud-aio-database pegged CPU on DB query

Support intro

Sorry to hear you’re facing problems. :slightly_frowning_face:

The community help forum (help.nextcloud.com) is for home and non-enterprise users. Support is provided by other community members on a best effort / “as available” basis. All of those responding are volunteering their time to help you.

If you’re using Nextcloud in a business/critical setting, paid and SLA-based support services can be accessed via portal.nextcloud.com where Nextcloud engineers can help ensure your business keeps running smoothly.

Getting help

In order to help you as efficiently (and quickly!) as possible, please fill in as much of the below requested information as you can.

Before clicking submit: Please check if your query is already addressed via the following resources:

(Utilizing these existing resources is typically faster. It also helps reduce the load on our generous volunteers while elevating the signal to noise ratio of the forums otherwise arising from the same queries being posted repeatedly).

The Basics

  • Nextcloud Server version (e.g., 29.x.x):
    • (32.0.3)
  • Operating system and version (e.g., Ubuntu 24.04):
    • Debian 13
  • Web server and version (e.g, Apache 2.4.25):
    • Nextcloud AIO
  • Reverse proxy and version _(e.g. nginx 1.27.2)
    • 1.29
  • PHP version (e.g, 8.3):
    • Nextcloud AIO
  • Is this the first time you’ve seen this error? (Yes / No):
    • Yes
  • When did this problem seem to first start?
    • About a month ago
  • Installation method (e.g. AlO, NCP, Bare Metal/Archive, etc.)
    • AIO
  • Are you using CloudfIare, mod_security, or similar? (Yes / No)
    • CloudFlare

Summary of the issue you are facing:

After about 10 minutes, the nextcloud database container gets stuck on a query. If I execute this

root@nextcloud:~# docker exec -it nextcloud-aio-database psql -U oc_nextcloud -d nextcloud_database -c "
SELECT pid, now()-query_start AS age, client_addr, state,
left(query, 300) AS query
FROM pg_stat_activity
WHERE state <> ‘idle’
ORDER BY query_start;"

I get this query which never seems to stop

pid  |       age       | client_addr | state  |                                                                                                                                                    query
------±----------------±------------±-------±-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
559 | 00:23:17.081032 | 172.19.0.8  | active | SELECT “filecache”.“fileid”, “storage”, “path”, “path_hash”, “filecache”.“parent”, “filecache”.“name”, “mimetype”, “mimepart”, “size”, “mtime”, “storage_mtime”, “encrypted”, “filecache”.“etag”, “filecache”.“permissions”, “checksum”, “unencrypted_size”, “metadata_etag”, “creation_time”, “upload_time”

If I then run

root@nextcloud:\~# docker exec -it nextcloud-aio-database psql -U oc_nextcloud -d nextcloud_database -c "
SELECT id, class, argument, reserved_at, last_run
FROM oc_jobs
WHERE reserved_at <> 0
ORDER BY reserved_at DESC
LIMIT 50;"

I get this job which also never seems to go away

id |               class               | argument | reserved_at |  last_run
----±----------------------------------±---------±------------±-----------
10 | OCA\\Files\\BackgroundJob\\ScanFiles | null     |  1768599827 | 1768599827
(1 row)

This problem gets worse over time since cron will spawn another job later on and they start to build up

Steps to replicate it (hint: details matter!)

I don’t know what to describe since this job automatically starts after about 10 minutes by itself.

Log entries

Nextcloud

Please provide the log entries from your Nextcloud log that are generated during the time of problem (via the Copy raw option from Administration settings->Logging screen or from your nextcloud.log located in your data directory). Feel free to use a pastebin/gist service if necessary.

There are no log entries for this, the DB query is stuck is the problem

Configuration

Nextcloud

The output of occ config:list system or similar is best, but, if not possible, the contents of your config.php file from /path/to/nextcloud is fine (make sure to remove any identifiable information!):

{
    "system": {
        "one-click-instance": true,
        "one-click-instance.user-limit": 100,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "check_data_directory_permissions": false,
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "overwritehost": "cloud.xxx.com.au",
        "overwriteprotocol": "https",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost",
            "cloud.xxx.com.au",
            "cloud.xxx.home.lan",
            "192.168.1.159",
            "111.56.91.111",
            "192.168.1.89"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "pgsql",
        "version": "32.0.3.2",
        "overwrite.cli.url": "https:\/\/cloud.xxx.com.au\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "default_locale": "en_AU",
        "force_locale": "en_AU",
        "default_language": "en",
        "force_language": "en",
        "loglevel": 2,
        "log_type": "file",
        "logfile": "\/var\/www\/html\/data\/nextcloud.log",
        "log_rotate_size": "10485760",
        "log.condition": {
            "apps": [
                "admin_audit"
            ]
        },
        "preview_max_x": 2048,
        "preview_max_y": 2048,
        "jpeg_quality": 60,
        "enable_previews": true,
        "enabledPreviewProviders": [
            "OC\\Preview\\JPEG",
            "OC\\Preview\\PNG",
            "OC\\Preview\\GIF",
            "OC\\Preview\\BMP",
            "OC\\Preview\\TXT",
            "OC\\Preview\\MarkDown",
            "OC\\Preview\\MP3",
            "OC\\Preview\\OpenDocument",
            "OC\\Preview\\Krita",
            "OC\\Preview\\Movie"
        ],
        "upgrade.disable-web": true,
        "mail_smtpmode": "smtp",
        "trashbin_retention_obligation": "auto, 30",
        "versions_retention_obligation": "auto, 30",
        "activity_expire_days": "30",
        "simpleSignUpLink.shown": false,
        "share_folder": "\/Shared",
        "one-click-instance.link": "https:\/\/nextcloud.com\/all-in-one\/",
        "upgrade.cli-upgrade-link": "https:\/\/github.com\/nextcloud\/all-in-one\/discussions\/2726",
        "updatedirectory": "\/nc-updater",
        "davstorage.request_timeout": 3600,
        "htaccess.RewriteBase": "\/",
        "dbpersistent": false,
        "files_external_allow_create_new_local": true,
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "forwarded_for_headers": [
            "HTTP_X_FORWARDED_FOR"
        ],
        "mail_smtpsecure": "ssl",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "allow_local_remote_servers": true,
        "appsallowlist": false,
        "updater.release.channel": "stable",
        "maintenance_window_start": 100,
        "auth.bruteforce.protection.enabled": true,
        "ratelimit.protection.enabled": true,
        "memories.db.triggers.fcu": true,
        "memories.exiftool": "\/var\/www\/html\/custom_apps\/memories\/bin-ext\/exiftool-amd64-musl",
        "memories.vod.path": "\/var\/www\/html\/custom_apps\/memories\/bin-ext\/go-vod-amd64",
        "memories.vod.ffmpeg": "\/usr\/bin\/ffmpeg",
        "memories.vod.ffprobe": "\/usr\/bin\/ffprobe",
        "documentation_url.server_logs": "https:\/\/github.com\/nextcloud\/all-in-one\/discussions\/5425",
        "updatechecker": false,
        "config_preset": 2,
        "defaultapp": "files",
        "DOMAIN": "cloud.xxx.com.au"
    }
}

Apps

The output of occ app:list (if possible).

root@nextcloud:\~# docker exec -u www-data -it nextcloud-aio-nextcloud php occ app:list
Enabled:

* admin_audit: 1.22.0
* assistant: 2.12.0
* bruteforcesettings: 5.0.0-dev.0
* calendar: 6.1.2
* cloud_federation_api: 1.16.0
* comments: 1.22.0
* contacts: 8.1.2
* contactsinteraction: 1.13.1
* dashboard: 7.12.0
* dav: 1.34.2
* encryption: 2.20.0
* federatedfilesharing: 1.22.0
* federation: 1.22.0
* files: 2.4.0
* files_downloadlimit: 5.0.0-dev.0
* files_pdfviewer: 5.0.0-dev.0
* files_reminders: 1.5.0
* files_sharing: 1.24.1
* files_trashbin: 1.22.0
* files_versions: 1.25.0
* firstrunwizard: 5.0.0-dev.0
* limit_login_to_ip: 4.2.0
* logreader: 5.0.0-dev.0
* lookup_server_connector: 1.20.0
* nextcloud-aio: 0.8.0
* notes: 4.12.4
* notifications: 5.0.0-dev.0
* notify_push: 1.3.0
* oauth2: 1.20.0
* password_policy: 4.0.0-dev.0
* photos: 5.0.0-dev.1
* privacy: 4.0.0-dev.0
* profile: 1.1.0
* provisioning_api: 1.22.0
* recommendations: 5.0.0-dev.0
* related_resources: 3.0.0-dev.0
* richdocuments: 9.0.2
* serverinfo: 4.0.0-dev.0
* settings: 1.15.1
* sharebymail: 1.22.0
* support: 4.0.0-dev.0
* systemtags: 1.22.0
* text: 6.0.1
* theming: 2.7.0
* twofactor_backupcodes: 1.21.0
* twofactor_totp: 14.0.0
* updatenotification: 1.22.0
* user_status: 1.12.0
* viewer: 5.0.0-dev.0
* weather_status: 1.12.0
* webhook_listeners: 1.3.0
* whiteboard: 1.5.2
* workflowengine: 2.14.0
  Disabled:
* activity: 5.0.0-dev.0 (installed 2.21.1)
* app_api: 32.0.0 (installed 4.0.5)
* circles: 32.0.0 (installed 27.0.1)
* files_accesscontrol: 3.0.2 (installed 3.0.2)
* files_external: 1.24.0
* files_rightclick: 0.15.1 (installed 1.6.0)
* integration_openai: 3.9.1 (installed 3.9.1)
* mail: 5.6.5 (installed 5.6.5)
* maps: 1.6.0 (installed 1.6.0)
* memories: 7.7.0 (installed 7.7.0)
* nextcloud_announcements: 4.0.0-dev.0 (installed 1.16.0)
* stt_whisper: 1.0.8 (installed 1.0.8)
* survey_client: 4.0.0-dev.0 (installed 1.17.0)
* suspicious_login: 10.0.0-dev.0 (installed 5.0.0)
* twofactor_nextcloud_notification: 6.0.0-dev.0
* user_ldap: 1.23.0

Tips for increasing the likelihood of a response

  • Use the preformatted text formatting option in the editor for all log entries and configuration output.
  • If screenshots are useful, feel free to include them.
    • If possible, also include key error output in text form so it can be searched for.
  • Try to edit log output only minimally (if at all) so that it can be ran through analyzers / formatters by those trying to help you.

Do you have many users?

Do you have many files?

Did you previously have External Storage with a bunch of files and remove it?

Etc.

What happens if you manually run a occ files:scan --unscanned <user> again some of your accounts?

I have in total 1.3 Million files across 4 user accounts. I have done as you suggested and done occ files:scan --unscanned for all users and for some odd reason this has fixed the problem!

Thank you for your help. Much appreciated, I’ve been working on this problem for days.

Did it take a while to scan from the command line? The background job should work. The main thing I recall it does differently is batch into groups of no more than 500 users per run, but that doesn’t sound applicable.

It also logs when the prior scan doesn’t finish typically.

One of the accounts took a long time. I suspect that was the problem, not sure.

But to make it more interesting, I’m back at the same problem again. The only thing I did differently recently is install the nextcloud desktop client for a user. That same night, CPU’s stuck at 50% again on the same queries.

Doing your solution this time around didn’t seem to fix the problem. Any other suggestions? :slight_smile:

This topic was automatically closed 8 days after the last reply. New replies are no longer allowed.