Php-fpm processes consuming a lot of CPU

[proton]

Nextcloud version (eg, 20.0.5): 28.0.3
Operating system and version (eg, Ubuntu 20.04): Ubuntu 20.04.6 LTS
Apache or nginx version (eg, Apache 2.4.25): nginx/1.24.0 Apache/2.4.41 (Ubuntu)
PHP version (eg, 7.4): 8.1

Hello,

The issue you are facing:

Nextcloud, or rather the php-fpm processes, consume a lot of CPU.

When trying to update via webui I get timeouts

I wanted to look into the matter and tried to read up on the subject. The first suggestion was to deactivate the apps, so I deactivated all apps and the performance did not really get better but after activating all apps again the logs started showing this (the url I blurred is wrong)

Now I’m torn because I don’t know if I should try to fix the problems or if there is a way to reinstall and restore the current state with reasonable effort.

Any advice is highly appreciated, many thanks in advance!

[tflidd]

Is this only for updates? If you have manual access via terminal, you can run the updater on the terminal, there you don’t have the usual time out and memory limits (and worst case, you could even stop the web server during the update). Some database operations can take quite some time.

Here the updater instructions on command line:
https://docs.nextcloud.com/server/latest/admin_manual/maintenance/update.html#using-the-command-line-based-updater

[proton]

I think it is a symptom, the main problem is the high cpu load.
I deleted the nextcloud folder and copied the latest version in it. Than changed config file and put db in place. CPU load is still very high and I guess it is something within the database.

What can I do?

[tflidd]

You can try to log slow queries, if you aren’t using any caches, you can do it to reduce the load on the database (redis, memcache), then you have tools to check the size of the database caches (tuning-primer/mysqltuner), …

[proton]

Thank for your help!
I must have expressed myself somewhat incorrectly. The database accesses are not the problem (as far as I can see, because the php-fpm processes are causing the load and not mysqld), I just assume that the reason for the high CPU load is somewhere in the database.

Now I tried to do a clean install just using the data directory (and hopefully copy task, calender etc later) but this also did not work.

[tflidd]

From outside, this is hard to tell, the CPU can also be high because it is waiting for something (the you should get a high i/o-wait, this is shown in some top versions, or use directly iotop to look for processes using the disk). Ideally the database uses quite a lot of cache in memory to reduce i/o operations.

@proton

Now I tried to do a clean install just using the data directory (and hopefully copy task, calender etc later) but this also did not work.

Well, if you are installing it the same way, there is a high chance that you get the same results…

[proton]

Sorry for my late reply.
jbd2 (journaling for ext4) is on top of iotop.

But I’m not sure how to determine which process is responsible for the high io.

Any ideas?

[tflidd]

Good question, might be journaling gets high when your drive is starting to die… or not sure how it manages extremely low storage on a partition or if you are starting to use your RAM cache (with free -m an all the tricks you should be able to check on the caching part)… or if you are on a RAID and it is syncing a lot in the background …

A good thing is perhaps to try to write a similar number/size of files independently of Nextcloud, if it does the same thing… And to a certain amount, it will be normal that is uses such resources.

[proton]

It seems like the problem was a hard drive that was about to fail in the near future. My provider change the disk and now it apparently works like normal again.

Thanks for help!

Best regards