High php-fpm cpu, slow response after upgrade

ℹ️ Support
,
1

I’ve been running NC15 on Debian9 with PHP7.0-fpm and Apache with HTTP/2, working fine. Two weeks ago, I upgraded the machine to Debian10 with PHP7.3-fpm and NC17, and since then everything takes a lot longer. I’ve got some long-term monitoring, so I can tell that the usual cpu usage on the web server (4 cores) was 1.5% avg before the update, and is up to 20% now (including night time). Simple changing directory in the files app will take about 1 second now, htop shows php-fpm processes creating high cpu usage.
settings/admin/overview shows no problems, memcache.local is APCu, Redis for memcache.locking, no fancy add-ons, nothing in the logs.
Any hints what to inspect next?

Regards,
Andreas

2

You should probably check if the opcache is active. It could be down to tuning the opcache.

3

Probably https://github.com/nextcloud/server/issues/17447 (infinite loop with searchdav requests)

Also good to read: https://github.com/nextcloud/server/issues/17241

I would try to patch the instance like in the first issue and check the monitoring. Feel free to join the other issue if that does not work.

4

Probably not a dav problem, because all requests that don’t involve dav are slow as well.
opcache is enabled and configured as recommended.

Checked: the searchdav-fix doesn’t fix this issue.

5

One more observation:
The memory usage was around 600MB for php-fpm processes, and is now down to 300MB. Total cpu around 100-200% (but not a single process at 100% stuck, load is coming and going), while it was avg 50% when users are busy.

A test file with phpinfo() shows me healtly opcache values (up&running, very few cache misses, 30% memory used).

6

Why do you stick to php-fpm and could you provide a rationale for this, if I may?

Please choose libapache2-mod-php7.3 over php-fpm as a home user certainly.

Happy hacking.
:sunflower:

7

mod-php is supported by mpm-prefork only. mpm-worker is required to support http/2

1 Like
8

Php-fpm is normally much faster than mod_php/cgi versions because the PHP engine remains loaded between each request.

1 Like
9

IMHO one could do some more thorough research and I posted some links to more background info in this forum already.
:lab_coat:

However, you are correct and this is a free world.
:smile:

One could make a connection, I presume.
:nerd_face:

Furthermore, one could reconsider if:

  1. the “much faster” is a requirement worth the apparent CPU load
  2. the apparent “easy to set up, requires less config” is a positive or a negative

BTW keep it safe, keep it simple is an accepted requirement for a stable system setup, I presume.
:lab_coat:

Some background info and discussion was provided in two post of same thread:

Hope this helps.
:nerd_face:

Happy hacking.
:sunflower:

10

Please provide your rationale. php-fpm is more flexible, the only reason to use mod php is because it’s easy to set up, requires less config.

1 Like
11

This thread has somewhat diverged from the original topic…
I’d still appreciate hints on how to trace down the problem of the NC installation/code.

12

Have you tried to debug php-fpm with gdb/strace? Check https://github.com/nextcloud/server/issues/17241 for some advice.

13

Didn’t try gdb/strace so far, but just tried the slowdebug with request_slowlog_timeout=1, and getting a lot of hits. Quite often:

password_verify() /var/www/nextcloud/lib/private/Security/Hasher.php:156

which is seems to be the same issue as https://github.com/nextcloud/server/issues/17241 , so I’ll close this and continue there.

3 Likes