Improve performance because of huge cpu load ( > 3)

[quentingrap]

Hello everyone,

After searching in lot of diffrebts topics, I’m asking some help.
For several weeks, we have been experiencing a lot of slowdowns on our Nextcloud.
In particular when creating files, loading pages etc.
I went to the admin page and the cpu load seems to be very high, and does not seem to be going down.
With htop we can see that php-fpm seems to take a lot of ressources.

I also see that cron job doesn’t go well. But when I launch it manually, it works. Maybe it’s just a drawback of php slowdowns.

Here are the server characteristics :

Nextcloud	Nextcloud 22.2.0
OS:	Linux 4.15.0-38-generic x86_64
CPU:	Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz (4 cores)
Memory:	15.63 GB

I have around 50 users at the same time.

Here are my apps

Enabled:
  - accessibility: 1.8.0
  - activity: 2.15.0
  - announcementcenter: 6.0.0
  - approval: 1.0.7
  - calendar: 2.3.4
  - circles: 22.1.1
  - cloud_federation_api: 1.5.0
  - comments: 1.12.0
  - contacts: 4.0.3
  - contactsinteraction: 1.3.0
  - dashboard: 7.2.0
  - dav: 1.19.0
  - deck: 1.5.3
  - federatedfilesharing: 1.12.0
  - federation: 1.12.0
  - files: 1.17.0
  - files_pdfviewer: 2.3.0
  - files_rightclick: 1.1.0
  - files_sharing: 1.14.0
  - files_trashbin: 1.12.0
  - files_versions: 1.15.0
  - files_videoplayer: 1.11.0
  - firstrunwizard: 2.11.0
  - forms: 2.3.0
  - guests: 2.0.2
  - impersonate: 1.9.0
  - integration_whiteboard: 0.0.14
  - logreader: 2.7.0
  - lookup_server_connector: 1.10.0
  - nextcloud_announcements: 1.11.0
  - notifications: 2.10.1
  - oauth2: 1.10.0
  - onlyoffice: 7.1.2
  - password_policy: 1.12.0
  - photos: 1.4.0
  - privacy: 1.6.0
  - provisioning_api: 1.12.0
  - recommendations: 1.1.0
  - serverinfo: 1.12.0
  - settings: 1.4.0
  - sharebymail: 1.12.0
  - spreed: 12.1.2
  - support: 1.5.0
  - systemtags: 1.12.0
  - tasks: 0.14.2
  - text: 3.3.0
  - theming: 1.13.0
  - twofactor_backupcodes: 1.11.0
  - updatenotification: 1.12.0
  - user_status: 1.2.0
  - viewer: 1.6.0
  - weather_status: 1.2.0
  - workflow_pdf_converter: 1.7.0
  - workflowengine: 2.4.0
Disabled:
  - admin_audit
  - caniupdate
  - encryption
  - files_external
  - survey_client
  - user_ldap

It still php7.4 but at the time, i tuned php with rieger advice
Here is the script I runned

sed -i "s/;env\[HOSTNAME\] = /env[HOSTNAME] = /" /etc/php/7.4/fpm/pool.d/www.conf
sed -i "s/;env\[TMP\] = /env[TMP] = /" /etc/php/7.4/fpm/pool.d/www.conf
sed -i "s/;env\[TMPDIR\] = /env[TMPDIR] = /" /etc/php/7.4/fpm/pool.d/www.conf
sed -i "s/;env\[TEMP\] = /env[TEMP] = /" /etc/php/7.4/fpm/pool.d/www.conf
sed -i "s/;env\[PATH\] = /env[PATH] = /" /etc/php/7.4/fpm/pool.d/www.conf
sed -i "s/pm.max_children = .*/pm.max_children = 240/" /etc/php/7.4/fpm/pool.d/www.conf
sed -i "s/pm.start_servers = .*/pm.start_servers = 20/" /etc/php/7.4/fpm/pool.d/www.conf
sed -i "s/pm.min_spare_servers = .*/pm.min_spare_servers = 10/" /etc/php/7.4/fpm/pool.d/www.conf
sed -i "s/pm.max_spare_servers = .*/pm.max_spare_servers = 20/" /etc/php/7.4/fpm/pool.d/www.conf
sed -i "s/;pm.max_requests = 500/pm.max_requests = 500/" /etc/php/7.4/fpm/pool.d/www.conf
sed -i "s/output_buffering =.*/output_buffering = 'Off'/" /etc/php/7.4/cli/php.ini
sed -i "s/max_execution_time =.*/max_execution_time = 1800/" /etc/php/7.4/cli/php.ini
sed -i "s/max_input_time =.*/max_input_time = 3600/" /etc/php/7.4/cli/php.ini
sed -i "s/post_max_size =.*/post_max_size = 10240M/" /etc/php/7.4/cli/php.ini
sed -i "s/;upload_tmp_dir =.*/upload_tmp_dir = \/upload_tmp/" /etc/php/7.4/cli/php.ini
sed -i "s/upload_max_filesize =.*/upload_max_filesize = 10240M/" /etc/php/7.4/cli/php.ini
sed -i "s/max_file_uploads =.*/max_file_uploads = 100/" /etc/php/7.4/cli/php.ini
sed -i "s/;date.timezone.*/date.timezone = Europe\/\Berlin/" /etc/php/7.4/cli/php.ini
sed -i "s/;session.cookie_secure.*/session.cookie_secure = True/" /etc/php/7.4/cli/php.ini
sed -i "s/;session.save_path =.*/session.save_path = \"N;700;\/usr\/local\/tmp\/sessions\"/" /etc/php/7.4/cli/php.ini
sed -i '$aapc.enable_cli = 1' /etc/php/7.4/cli/php.ini
sed -i "s/memory_limit = 128M/memory_limit = 512M/" /etc/php/7.4/fpm/php.ini
sed -i "s/output_buffering =.*/output_buffering = 'Off'/" /etc/php/7.4/fpm/php.ini
sed -i "s/max_execution_time =.*/max_execution_time = 1800/" /etc/php/7.4/fpm/php.ini
sed -i "s/max_input_time =.*/max_input_time = 3600/" /etc/php/7.4/fpm/php.ini
sed -i "s/post_max_size =.*/post_max_size = 10240M/" /etc/php/7.4/fpm/php.ini
sed -i "s/;upload_tmp_dir =.*/upload_tmp_dir = \/upload_tmp/" /etc/php/7.4/fpm/php.ini
sed -i "s/upload_max_filesize =.*/upload_max_filesize = 10240M/" /etc/php/7.4/fpm/php.ini
sed -i "s/max_file_uploads =.*/max_file_uploads = 100/" /etc/php/7.4/fpm/php.ini
sed -i "s/;date.timezone.*/date.timezone = Europe\/\Berlin/" /etc/php/7.4/fpm/php.ini
sed -i "s/;session.cookie_secure.*/session.cookie_secure = True/" /etc/php/7.4/fpm/php.ini
sed -i "s/;opcache.enable=.*/opcache.enable=1/" /etc/php/7.4/fpm/php.ini
sed -i "s/;opcache.enable_cli=.*/opcache.enable_cli=1/" /etc/php/7.4/fpm/php.ini
sed -i "s/;opcache.memory_consumption=.*/opcache.memory_consumption=128/" /etc/php/7.4/fpm/php.ini
sed -i "s/;opcache.interned_strings_buffer=.*/opcache.interned_strings_buffer=8/" /etc/php/7.4/fpm/php.ini
sed -i "s/;opcache.max_accelerated_files=.*/opcache.max_accelerated_files=10000/" /etc/php/7.4/fpm/php.ini
sed -i "s/;opcache.revalidate_freq=.*/opcache.revalidate_freq=1/" /etc/php/7.4/fpm/php.ini
sed -i "s/;opcache.save_comments=.*/opcache.save_comments=1/" /etc/php/7.4/fpm/php.ini
sed -i "s/;session.save_path =.*/session.save_path = \"N;700;\/usr\/local\/tmp\/sessions\"/" /etc/php/7.4/fpm/php.ini
sed -i "s/;emergency_restart_threshold =.*/emergency_restart_threshold = 10/" /etc/php/7.4/fpm/php-fpm.conf
sed -i "s/;emergency_restart_interval =.*/emergency_restart_interval = 1m/" /etc/php/7.4/fpm/php-fpm.conf
sed -i "s/;process_control_timeout =.*/process_control_timeout = 10s/" /etc/php/7.4/fpm/php-fpm.conf
sed -i '$aapc.enabled=1' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.file_update_protection=2' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.optimization=0' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.shm_size=256M' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.include_once_override=0' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.shm_segments=1' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.ttl=7200' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.user_ttl=7200' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.gc_ttl=3600' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.num_files_hint=1024' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.enable_cli=0' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.max_file_size=5M' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.cache_by_default=1' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.use_request_time=1' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.slam_defense=0' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.mmap_file_mask=/usr/local/tmp/apc.XXXXXX' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.stat_ctime=0' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.canonicalize=1' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.write_lock=1' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.report_autofilter=0' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.rfc1867=0' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.rfc1867_prefix =upload_' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.rfc1867_name=APC_UPLOAD_PROGRESS' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.rfc1867_freq=0' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.rfc1867_ttl=3600' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.lazy_classes=0' /etc/php/7.4/fpm/php.ini
sed -i '$aapc.lazy_functions=0' /etc/php/7.4/fpm/php.ini
sed -i "s/09,39.*/# &/" /etc/cron.d/php
(crontab -l ; echo "09,39 * * * * /usr/lib/php/sessionclean 2>&1") | crontab -u root -

Do you think my server is not enough performant or I can improve some parameters ??

Thanks for your advices

[tflidd]

Have you tried the tips from the documentation:
https://docs.nextcloud.com/server/latest/admin_manual/installation/server_tuning.html

you are only using a small part of your memory. If you set up your database caches correctly, and enable the caching with redis/apcu, this should improve a lot. There are a few topics about this here.

[quentingrap]

Hello,

Yes I’m already using mariadb, redis, php-fpm tuning and have the opcache parameters.

I notice that I have one user that cause the same error again and again because of error in synchronisation because of lack of space in this account. I don’t know if it can be a source of bad performance ?!

[tflidd]

It is a bit strange that you hardly use 2GB of your RAM. And you just have a swap of 1 GB which is fully used. Regarding the size of your database tables, for best performance, you should give large enough caches to the database.

NextCloud Performance Optimizations - Please Contribute!

About MySQL buffer tweak: I followed your links till the end and found this settings for the my.cnf: innodb_buffer_pool_size = 512M innodb_buffer_pool_instance = 1 innodb_flush_log_at_trx_commit = 2 innodb_log_buffer_size = 32M innodb_max_dirty_pages_pct = 90 innodb_buffer_pool_size defaults to 128M: source Increasing it should of course depend on your system and there should be enough memory left for the remaining to run safely. Also it doesn’t make sense to increase the buffer, if the dat…

Also worth checking is iotop in case processes slow down the system due to disk access. If it is the database, then this can be optimized with the cache settings especially with so much unused RAM in your case.

[quentingrap]

Here is the result of iotop (that i just discover thanks to you)

Seems to have an issue with jbd2/md1-8 !

For the nextcloud database, it’s 1016.87 MB

you should give large enough caches to the database.

I’m not sure, are you talking about this parameter innodb_buffer_pool_size ?
At the moment, it is set to 256M
Here are all this related parameters

# Read the manual for more InnoDB related options. There are many!
default_storage_engine  = InnoDB
# you can't just change log file size, requires special procedure
#innodb_log_file_size   = 50M
innodb_buffer_pool_size = 256M
innodb_log_buffer_size  = 8M
innodb_file_per_table   = 1
innodb_open_files       = 400
innodb_io_capacity      = 400
innodb_flush_method     = O_DIRECT

Also thanks for sharing this topic, i will check this as soon as possible

[quentingrap]

So I did several changes. Here is a summary.

At first, I update my server and reboot it (for the first time since I launch it 3 years ago --’)
Swap is now empty

MySQL logs
The logrotate for mysql was not handle correctly so I had too much Gigas of logs.
In /etc/logrotate.d/mysql-server, it was not logrotating my log so I change the first line adding /var/log/mysql/slow.log and one parameter size 1G and force logrotating logrotate -vf /etc/logrotate.conf to compress all these logs

PHP optimization

• I upgrade to PHP8.0 (from 7.4)
• Following Rieger script, I change parameter for php-fpm

cp /etc/php/8.0/fpm/pool.d/www.conf /etc/php/8.0/fpm/pool.d/www.conf.bak
cp /etc/php/8.0/fpm/php-fpm.conf /etc/php/8.0/fpm/php-fpm.conf.bak
cp /etc/php/8.0/cli/php.ini /etc/php/8.0/cli/php.ini.bak
cp /etc/php/8.0/fpm/php.ini /etc/php/8.0/fpm/php.ini.bak
cp /etc/php/8.0/fpm/php-fpm.conf /etc/php/8.0/fpm/php-fpm.conf.bak
cp /etc/php/8.0/mods-available/apcu.ini /etc/php/8.0/mods-available/apcu.ini.bak
cp /etc/ImageMagick-6/policy.xml /etc/ImageMagick-6/policy.xml.bak
AvailableRAM=$(awk '/MemAvailable/ {printf "%d", $2/1024}' /proc/meminfo)
AverageFPM=$(ps --no-headers -o 'rss,cmd' -C php-fpm8.0 | awk '{ sum+=$1 } END { printf ("%d\n", sum/NR/1024,"M") }')
FPMS=$((AvailableRAM/AverageFPM))
PMaxSS=$((FPMS*2/3))
PMinSS=$((PMaxSS/2))
PStartS=$(((PMaxSS+PMinSS)/2))
sed -i "s/;env\[HOSTNAME\] = /env[HOSTNAME] = /" /etc/php/8.0/fpm/pool.d/www.conf
sed -i "s/;env\[TMP\] = /env[TMP] = /" /etc/php/8.0/fpm/pool.d/www.conf
sed -i "s/;env\[TMPDIR\] = /env[TMPDIR] = /" /etc/php/8.0/fpm/pool.d/www.conf
sed -i "s/;env\[TEMP\] = /env[TEMP] = /" /etc/php/8.0/fpm/pool.d/www.conf
sed -i "s/;env\[PATH\] = /env[PATH] = /" /etc/php/8.0/fpm/pool.d/www.conf
sed -i 's/pm.max_children =.*/pm.max_children = '$FPMS'/' /etc/php/8.0/fpm/pool.d/www.conf
sed -i 's/pm.start_servers =.*/pm.start_servers = '$PStartS'/' /etc/php/8.0/fpm/pool.d/www.conf
sed -i 's/pm.min_spare_servers =.*/pm.min_spare_servers = '$PMinSS'/' /etc/php/8.0/fpm/pool.d/www.conf
sed -i 's/pm.max_spare_servers =.*/pm.max_spare_servers = '$PMaxSS'/' /etc/php/8.0/fpm/pool.d/www.conf
sed -i "s/;pm.max_requests =.*/pm.max_requests = 1000/" /etc/php/8.0/fpm/pool.d/www.conf
sed -i "s/allow_url_fopen =.*/allow_url_fopen = 1/" /etc/php/8.0/fpm/php.ini
sed -i "s/output_buffering =.*/output_buffering = 'Off'/" /etc/php/8.0/cli/php.ini
sed -i "s/max_execution_time =.*/max_execution_time = 3600/" /etc/php/8.0/cli/php.ini
sed -i "s/max_input_time =.*/max_input_time = 3600/" /etc/php/8.0/cli/php.ini
sed -i "s/post_max_size =.*/post_max_size = 10240M/" /etc/php/8.0/cli/php.ini
sed -i "s/upload_max_filesize =.*/upload_max_filesize = 10240M/" /etc/php/8.0/cli/php.ini
sed -i "s/;date.timezone.*/date.timezone = Europe\/\Berlin/" /etc/php/8.0/cli/php.ini
sed -i "s/memory_limit = 128M/memory_limit = 1024M/" /etc/php/8.0/fpm/php.ini
sed -i "s/output_buffering =.*/output_buffering = 'Off'/" /etc/php/8.0/fpm/php.ini
sed -i "s/max_execution_time =.*/max_execution_time = 3600/" /etc/php/8.0/fpm/php.ini
sed -i "s/max_input_time =.*/max_input_time = 3600/" /etc/php/8.0/fpm/php.ini
sed -i "s/post_max_size =.*/post_max_size = 10240M/" /etc/php/8.0/fpm/php.ini
sed -i "s/upload_max_filesize =.*/upload_max_filesize = 10240M/" /etc/php/8.0/fpm/php.ini
sed -i "s/;date.timezone.*/date.timezone = Europe\/\Berlin/" /etc/php/8.0/fpm/php.ini
sed -i "s/;session.cookie_secure.*/session.cookie_secure = True/" /etc/php/8.0/fpm/php.ini
sed -i "s/;opcache.enable=.*/opcache.enable=1/" /etc/php/8.0/fpm/php.ini
sed -i "s/;opcache.enable_cli=.*/opcache.enable_cli=1/" /etc/php/8.0/fpm/php.ini
sed -i "s/;opcache.memory_consumption=.*/opcache.memory_consumption=128/" /etc/php/8.0/fpm/php.ini
sed -i "s/;opcache.interned_strings_buffer=.*/opcache.interned_strings_buffer=8/" /etc/php/8.0/fpm/php.ini
sed -i "s/;opcache.max_accelerated_files=.*/opcache.max_accelerated_files=10000/" /etc/php/8.0/fpm/php.ini
sed -i "s/;opcache.revalidate_freq=.*/opcache.revalidate_freq=1/" /etc/php/8.0/fpm/php.ini
sed -i "s/;opcache.save_comments=.*/opcache.save_comments=1/" /etc/php/8.0/fpm/php.ini
sed -i "s|;emergency_restart_threshold.*|emergency_restart_threshold = 10|g" /etc/php/8.0/fpm/php-fpm.conf
sed -i "s|;emergency_restart_interval.*|emergency_restart_interval = 1m|g" /etc/php/8.0/fpm/php-fpm.conf
sed -i "s|;process_control_timeout.*|process_control_timeout = 10|g" /etc/php/8.0/fpm/php-fpm.conf
sed -i '$aapc.enable_cli=1' /etc/php/8.0/mods-available/apcu.ini
sed -i "s/rights=\"none\" pattern=\"PS\"/rights=\"read|write\" pattern=\"PS\"/" /etc/ImageMagick-6/policy.xml
sed -i "s/rights=\"none\" pattern=\"EPS\"/rights=\"read|write\" pattern=\"EPS\"/" /etc/ImageMagick-6/policy.xml
sed -i "s/rights=\"none\" pattern=\"PDF\"/rights=\"read|write\" pattern=\"PDF\"/" /etc/ImageMagick-6/policy.xml
sed -i "s/rights=\"none\" pattern=\"XPS\"/rights=\"read|write\" pattern=\"XPS\"/" /etc/ImageMagick-6/policy.xml

So now i have in /etc/php/8.0/fpm/pool.d/www.conf

pm.max_children = 540 (120 before)
pm.start_servers = 270 (12 before)
pm.min_spare_servers = 180 (6 before)
pm.max_spare_servers = 360 (18 before)

The difference is huge, I will quickly see if it’s not too much

I need more time to really now but at the moment

• Swap is now empty
• RAM is not stuck to 2Go
• Load average is around 2,5
• In iotop, jbd2/md1-8 is number one but not stuck at 99% IO but around 10 - 30%