High Postgresql CPU usage on Nextcloud server

Support intro

Sorry to hear you’re facing problems :slightly_frowning_face:

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can :heart:

Nextcloud version (eg, 20.0.5): 20.04
Operating system and version (eg, Ubuntu 20.04): CentOS Linux release 8.2.2004 (Core)
Apache or nginx version (eg, Apache 2.4.25): Apache/2.4.37 (centos)
PHP version (eg, 7.4): PHP 7.4.12

The issue you are facing:
We are experiencing high cpu usage by postgresql processes.

Output from top command:


    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 894213 postgres  20   0  293508 150652 144556 R 100.0   1.9  15:47.92 postmaster
 894214 postgres  20   0  287972 142616 141356 R 100.0   1.8  15:48.46 postmaster
 894215 postgres  20   0  287972 142744 141484 R 100.0   1.8  15:48.00 postmaster

Is this the first time you’ve seen this error? (Y/N): N

Steps to replicate it:

The output of your Nextcloud log in Admin > Logging:

{"reqId":"YV1B9guLLH0FEAZF28ib7AAAAEc","level":3,"time":"2021-10-06T06:28:06+00:00","remoteAddr":"*.*.*.*","user":"******","app":"PHP","method":"MOVE","url":"/remote.php/dav/files/******/FZOEU/New%20folder","message":{"Exception":"Error","Message":"fread(): read of 8192 bytes failed with errno=21 Is a directory at /var/www/nextcloud/lib/private/Files/Storage/Wrapper/Encryption.php#862","Code":0,"Trace":[{"function":"onError","class":"OC\\Log\\ErrorHandler","type":"::"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Encryption.php","line":862,"function":"fread"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Encryption.php","line":930,"function":"readFirstBlock","class":"OC\\Files\\Storage\\Wrapper\\Encryption","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Encryption.php","line":961,"function":"getHeader","class":"OC\\Files\\Storage\\Wrapper\\Encryption","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Encryption.php","line":294,"function":"getEncryptionModule","class":"OC\\Files\\Storage\\Wrapper\\Encryption","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":278,"function":"rename","class":"OC\\Files\\Storage\\Wrapper\\Encryption","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":278,"function":"rename","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/View.php","line":822,"function":"rename","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/apps/dav/lib/Connector/Sabre/Node.php","line":142,"function":"rename","class":"OC\\Files\\View","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Tree.php","line":157,"function":"setName","class":"OCA\\DAV\\Connector\\Sabre\\Node","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":641,"function":"move","class":"Sabre\\DAV\\Tree","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpMove","class":"Sabre\\DAV\\CorePlugin","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":474,"function":"emit","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":251,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":319,"function":"start","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/apps/dav/lib/Server.php","line":332,"function":"exec","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/apps/dav/appinfo/v2/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/remote.php","line":167,"args":["/var/www/nextcloud/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/nextcloud/lib/private/Log/ErrorHandler.php","Line":91,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (Windows) mirall/3.3.5stable-Win64 (build 20210930) (Nextcloud, windows-10.0.19042 ClientArchitecture: x86_64 OsArchitecture: x86_64)","version":"20.0.4.0","id":"615d7bb96367f"}

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => '******',
  'passwordsalt' => '*****',
  'secret' => '*******',
  'trusted_domains' =>
  array (
    0 => '*****',
    1 => '*****',
  ),
  'trusted_proxies' =>
  array (
    0 => '******',
  ),
  'datadirectory' => '/opt/data/nextcloud',
  'filelocking.enabled' => true,
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
    'host' => '******',
    'port' => 6379,
  ),
  'dbtype' => 'pgsql',
  'version' => '20.0.4.0',
  'overwrite.cli.url' => 'https://*****',
  'dbname' => '******',
  'dbhost' => '******',
  'dbport' => '5432',
  'dbtableprefix' => 'oc_',
  'dbuser' => '******',
  'dbpassword' => '******',
  'installed' => true,
  'social_login_auto_redirect' => true,
  'overwriteprotocol' => 'https',
  'mail_from_address' => 'nextcloud',
  'mail_smtpmode' => 'smtp',
  'mail_sendmailmode' => 'smtp',
  'mail_domain' => '*****',
  'maintenance' => false,
  'theme' => '',
  'loglevel' => 2,
  'encryption.key_storage_migrated' => false,
  'mail_smtphost' => '*******',
  'mail_smtpport' => '25',
);

The output of your Apache/nginx/system log in /var/log/____:

**** - - [06/Oct/2021:14:00:00 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 255
**** - - [06/Oct/2021:13:59:40 +0200] "GET /index.php/apps/logreader/poll?lastReqId=YV1B9guLLH0FEAZF28ib7AAAAEc HTTP/1.1" 200 22
**** - - [06/Oct/2021:14:00:03 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 254
**** - - [06/Oct/2021:14:00:04 +0200] "PROPFIND /remote.php/webdav/Documents/***/****/***.pdf HTTP/1.1" 207 777
**** - - [06/Oct/2021:14:00:04 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 253
**** - - [06/Oct/2021:14:00:05 +0200] "GET /ocs/v2.php/apps/user_status/api/v1/user_status?format=json HTTP/1.1" 200 150
**** - - [06/Oct/2021:14:00:05 +0200] "GET /ocs/v2.php/apps/notifications/api/v2/notifications?format=json HTTP/1.1" 304 -
**** - - [06/Oct/2021:14:00:05 +0200] "GET /ocs/v2.php/apps/user_status/api/v1/user_status?format=json HTTP/1.1" 200 150
**** - - [06/Oct/2021:14:00:05 +0200] "GET /ocs/v2.php/apps/notifications/api/v2/notifications?format=json HTTP/1.1" 304 -
**** - - [06/Oct/2021:14:00:05 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 250
**** - - [06/Oct/2021:14:00:06 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 253
**** - - [06/Oct/2021:14:00:06 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 252
**** - - [06/Oct/2021:14:00:06 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 255
**** - - [06/Oct/2021:14:00:07 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 252
**** - - [06/Oct/2021:14:00:07 +0200] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 304 -
**** - - [06/Oct/2021:14:00:07 +0200] "GET /ocs/v2.php/apps/user_status/api/v1/user_status?format=json HTTP/1.1" 200 153
**** - - [06/Oct/2021:14:00:07 +0200] "GET /ocs/v2.php/apps/notifications/api/v2/notifications?format=json HTTP/1.1" 304 -
**** - - [06/Oct/2021:14:00:08 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 254
**** - - [06/Oct/2021:14:00:09 +0200] "PROPFIND /remote.php/webdav/Documents/****/****/****.pdf HTTP/1.1" 207 777
**** - - [06/Oct/2021:14:00:09 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 254
**** - - [06/Oct/2021:14:00:09 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 250
**** - - [06/Oct/2021:14:00:13 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 252
**** - - [06/Oct/2021:14:00:14 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 250
**** - - [06/Oct/2021:14:00:14 +0200] "PROPFIND /remote.php/webdav/Documents/****/****/****.pdf HTTP/1.1" 207 777
**** - - [06/Oct/2021:14:00:14 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 248
**** - - [06/Oct/2021:14:00:14 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 251
**** - - [06/Oct/2021:14:00:15 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 253
**** - - [06/Oct/2021:14:00:17 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 253
**** - - [06/Oct/2021:14:00:19 +0200] "PROPFIND /remote.php/webdav/Documents/****/****/****.pdf HTTP/1.1" 207 777
**** - - [06/Oct/2021:14:00:19 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 251
**** - - [06/Oct/2021:14:00:00 +0200] "GET /index.php/apps/logreader/poll?lastReqId=YV1B9guLLH0FEAZF28ib7AAAAEc HTTP/1.1" 200 22
**** - - [06/Oct/2021:14:00:21 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 248
**** - - [06/Oct/2021:14:00:21 +0200] "GET /ocs/v2.php/core/navigation/apps?absolute=true&format=json HTTP/1.1" 304 -
**** - - [06/Oct/2021:14:00:21 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 248
**** - - [06/Oct/2021:14:00:22 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 257
**** - - [06/Oct/2021:14:00:24 +0200] "PROPFIND /remote.php/webdav/Documents/****/****/****.pdf HTTP/1.1" 207 777
**** - - [06/Oct/2021:14:00:24 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 252
**** - - [06/Oct/2021:14:00:25 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 251
**** - - [06/Oct/2021:14:00:25 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 253
**** - - [06/Oct/2021:14:00:27 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 252
**** - - [06/Oct/2021:14:00:27 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 250
**** - - [06/Oct/2021:14:00:28 +0200] "GET /ocs/v2.php/apps/notifications/api/v2/notifications?format=json HTTP/1.1" 304 -
**** - - [06/Oct/2021:14:00:28 +0200] "GET /ocs/v2.php/apps/user_status/api/v1/user_status?format=json HTTP/1.1" 200 153
**** - - [06/Oct/2021:14:00:29 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 253
**** - - [06/Oct/2021:14:00:29 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 251
**** - - [06/Oct/2021:14:00:29 +0200] "PROPFIND /remote.php/webdav/Documents/****/****/****.pdf HTTP/1.1" 207 777
**** - - [06/Oct/2021:14:00:29 +0200] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 401 106
**** - - [06/Oct/2021:14:00:29 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 252
**** - - [06/Oct/2021:14:00:30 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 255
**** - - [06/Oct/2021:14:00:30 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 248
**** - - [06/Oct/2021:14:00:33 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 254
**** - - [06/Oct/2021:14:00:34 +0200] "PROPFIND /remote.php/webdav/Documents/****/****/****.pdf HTTP/1.1" 207 777
**** - - [06/Oct/2021:14:00:34 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 253
**** - - [06/Oct/2021:14:00:34 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 253
**** - - [06/Oct/2021:14:00:35 +0200] "GET /ocs/v2.php/apps/user_status/api/v1/user_status?format=json HTTP/1.1" 200 151
**** - - [06/Oct/2021:14:00:35 +0200] "GET /ocs/v2.php/apps/notifications/api/v2/notifications?format=json HTTP/1.1" 304 -
**** - - [06/Oct/2021:14:00:36 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 253
**** - - [06/Oct/2021:14:00:36 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 252
**** - - [06/Oct/2021:14:00:36 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 255
**** - - [06/Oct/2021:14:00:36 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 250
**** - - [06/Oct/2021:14:00:38 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 254
**** - - [06/Oct/2021:14:00:39 +0200] "PROPFIND /remote.php/webdav/Documents/****/****/****.pdf HTTP/1.1" 207 777
**** - - [06/Oct/2021:14:00:39 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 254
**** - - [06/Oct/2021:14:00:39 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 250
**** - - [06/Oct/2021:14:00:40 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 248
**** - - [06/Oct/2021:14:00:40 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 252
**** - - [06/Oct/2021:14:00:20 +0200] "GET /index.php/apps/logreader/poll?lastReqId=YV1B9guLLH0FEAZF28ib7AAAAEc HTTP/1.1" 200 22
**** - - [06/Oct/2021:14:00:42 +0200] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 304 -
**** - - [06/Oct/2021:14:00:42 +0200] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 304 -
**** - - [06/Oct/2021:14:00:42 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 248
**** - - [06/Oct/2021:14:00:43 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 252
**** - - [06/Oct/2021:14:00:44 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 247
**** - - [06/Oct/2021:14:00:44 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 250
**** - - [06/Oct/2021:14:00:44 +0200] "PROPFIND /remote.php/webdav/Documents/****/****/****.pdf HTTP/1.1" 207 777
**** - - [06/Oct/2021:14:00:44 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 251
**** - - [06/Oct/2021:14:00:45 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 253
**** - - [06/Oct/2021:14:00:46 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 251
**** - - [06/Oct/2021:14:00:47 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 253
**** - - [06/Oct/2021:14:00:49 +0200] "PROPFIND /remote.php/webdav/Documents/****/****/****.pdf HTTP/1.1" 207 777
**** - - [06/Oct/2021:14:00:52 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 257
**** - - [06/Oct/2021:14:00:53 +0200] "GET /ocs/v2.php/core/navigation/apps?absolute=true&format=json HTTP/1.1" 304 -
**** - - [06/Oct/2021:14:00:53 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 248
**** - - [06/Oct/2021:14:00:53 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 2097
**** - - [06/Oct/2021:14:00:54 +0200] "PROPFIND /remote.php/webdav/Documents/****/****/****.pdf HTTP/1.1" 207 777
**** - - [06/Oct/2021:14:00:54 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 252
**** - - [06/Oct/2021:14:00:54 +0200] "PUT /remote.php/dav/files/****/****.xlsx HTTP/1.1" 204 -
**** - - [06/Oct/2021:14:00:55 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 251
**** - - [06/Oct/2021:14:00:55 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 253
**** - - [06/Oct/2021:14:00:57 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 252
**** - - [06/Oct/2021:14:00:57 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 250
**** - - [06/Oct/2021:14:00:58 +0200] "GET /remote.php/dav/files/****/Joplin/info.json HTTP/1.1" 200 1310
**** - - [06/Oct/2021:14:00:59 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 251
**** - - [06/Oct/2021:14:00:58 +0200] "PROPFIND /remote.php/dav/files/****/Joplin/locks/ HTTP/1.1" 207 277
**** - - [06/Oct/2021:14:00:59 +0200] "PROPFIND /remote.php/dav/files/****/Joplin/locks/ HTTP/1.1" 207 277
**** - - [06/Oct/2021:14:00:59 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 252
**** - - [06/Oct/2021:14:01:00 +0200] "PROPFIND /remote.php/dav/files/****/ HTTP/1.1" 207 255
**** - - [06/Oct/2021:14:00:59 +0200] "PUT /remote.php/dav/files/****/Joplin/temp/timeCheck950812.txt HTTP/1.1" 201 -
**** - - [06/Oct/2021:14:01:00 +0200] "PROPFIND /remote.php/webdav/Documents/****/****/****.pdf HTTP/1.1" 207 777
**** - - [06/Oct/2021:14:00:40 +0200] "GET /index.php/apps/logreader/poll?lastReqId=YV1B9guLLH0FEAZF28ib7AAAAEc HTTP/1.1" 200 22

Hello,
we are having a problem that postgresql services are using around 100% of CPU.
When we run SELECT * FROM pg_stat_activity; we can see that postgresql services are trying to execute SQL statement:

SELECT “fileid”, “storage” FROM “oc_filecache” “fc” WHERE (“fileid” NOT IN (SELECT “fileid” FROM “oc_files_antivirus”)) AND (“mimetype” <> ‘2’) AND (“path” LIKE ‘files/%’) AND ((“fc”.“size” <> ‘0’)AND (“fc”.“size” < $1)) LIMIT 1000

We tried to execute EXPLAIN ANALYZE for the above SQL statement and it took 5176426.636 ms to finish.

I am not sure what to do, so any help is greatly appreciated.

Sorry i do not use postgres. But perhaps a problem with a large oc_filecache.
Read Large oc_filecache

Hmm… that “oc_files_antivirus” part doesn’t look familiar to me. Some kind of strange plugin?
The overall query is pretty straightforward. Try breaking it down until you find the part of the query that is making it slow. For example, see how long it takes to just run the select from oc_files_antivirus. Then see how long it takes to run the entire thing up to just before the first AND.

I suspect that postgresql will not be tested as well as mariadb, which is a lot more universal.

Other thoughts; it could be a limitation of your hardware – you didnt specify what kind of hardware you are running.

Also; CentOS Linux release 8.2.2004
It might be time to update;

  1. Current CentOS release version is 8.4,
  2. CentOS is going EOL in just TWO MONTHS from now (RH/IBM announced this about a year ago) – consider Rocky Linux, which is a project started by one of the original founders of CentOS and should therefore be thought of as the real CentOS. The official migration script can be found here: rocky-tools/migrate2rocky at main · rocky-linux/rocky-tools · GitHub