Performance optimization

šŸ·ļø General
1

Hi guys,

I am running nextcloud for ~5yrs now and I was bored these days and went checking on some performance tweaks.

In the meantime i got apcu and redis running.
Still I am having with empty cache (ctrl+f5) 15s to load the dom according to chrome dev tools, i am the only user on the dedicated server, having VMs though.
I donā€™t want to go into more detail about sizing and hardware recommendations but more into finding bottlenecks by diagnosing crunching numbers. Donā€™t get me wrong, thats already 10s less than without any memcache.

So iā€™ve checked disc utilization with iotop as disks are slowest partā€¦ and that looks ok for me.
Checking context switches and CPU% utilization it looks like php is really taking the time for stuffā€¦
Iā€™ve followed the usual guides for performance enhancement and before I start to quote all my settingsā€¦ whats you fastest time with CTRL+F5 reload to load the DOM tree?

2

For me Nextcloud webGUI is slow on a first start even on dedicated server i built for it. It got 4core CPU, 8G DDR4 RAM, and two HDDā€™s (not SSD). There is nginx used for it, apcu (redis is turned off because made so many errors in logs), MariaDB got 1G cache configured, and second HDD is used ONLY for NC files with data directory (so it isnā€™t even used when LOGs are generated or database write something). There are only 4 users, while only one (me) is using it often.
I think using SSD would made it work faster, but two SSDā€™s with 4TB capacity would cost too much.
I believe NC is developing into more business level product, so instead of better performance and fixing things it will have more tools for cooperation in the future.

1 Like
3

Not muchā€¦
Although I havenā€™t run NC on bare metal (VMs only), giving it 8 cores, 64GB of RAM and 20+TB of all-flash vSAN storage does practically nothing perception-wise in GUI speedā€¦

That is definitely the trendā€¦

4

Your times seem to be quite high. I have DOMContentLoaded after 2100 ms.

My setup:

  • Ubuntu 20.04 with apache and php fpm (7.4) at home on a 200/4mbit connection
  • Proxy in the cloud with nginx (this alone adds ~40 ms to every request)
  • APCU for memcache-local, Redis for file locking
  • NC version 19.0.3.1
  • about 26 users, with 800GB storage used, usually about 5-10 user active
  • running on a server with IntelĀ® XeonĀ® CPU L3426, 8 GB of memory, OS + NC + Database are on a raid1 with an old patriot spark ssd and a Sandisk z400 (both 128GB), Actual data is on a ZFS raidz volume with 6x2TB HDDs (That machine has also a few other services running)
5

Ok, since there is somebody with way better performance, I think its worth looking into it.
i could imagine the version up in php is your benefit. for me it looks like really php needs the time on cpu utilization and not IO wait. Memory seems sufficient, swap not used.

My setup is a dedicated server in a real DC, so latency via internet is to be considered but peering is good.
The latency did not seem to cause the issue, a local firefox on the server was showing to similar timings. But now testing from home instead wifi its down to 9 seconds. puzzled
hypervisor:
Centos7, KVM (4 physical E3-1245 cores + hyperthreading)
2x 7200rpm disks in RAID1, XFS as filesystem

VM:
Centos7
8vCPU + 4GB memory in VM
nextcloud data dir is on nfs, hosted on the the RAID1 of hypervisor
nextcloud install dir is on VM own storage, XFS as filesystem
Memcaching configured as outlined here: https://www.linuxbots.com/tuning-nextcloud-for-better-performance/ besides ā€˜memcache.localā€™ -> thats apcu.

Troubleshooting, all run on the VM level.

PHP Version 7.3.22

Linux 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020 x86_64

  • iotop --only --accumulated shows a few KB read/write while i reload website. Looks really low on impact. also IOwait on cpu utilization is low.
  • iotops result seems to be confirmed by pidstat -r -> some minor faults, but no major faults (minflt/s vs majflt/s)
  • pidstat -w does not show any ā€œnvcswch/sā€ -> non voluntary context switches would be indicator of bottleneck in overall CPU performance (IMHO)
  • pidstat -u shows some CPU utilization only for PHP-fpm, postgres and http are very very low on CPU utilization.

pidstat -u 1 5 | egrep -i ā€œpostg|http|phpā€ --> i just pull a few lines from the output to give an idea, whole list would be toooo long.
08:00:46 PM UID PID %usr %system %guest %CPU CPU Command
08:01:03 PM 26 24144 2.00 0.00 0.00 2.00 6 postgres
08:01:04 PM 1734800015 10566 2.00 1.00 0.00 3.00 1 httpd
08:01:04 PM 1734800015 10628 3.00 0.00 0.00 3.00 1 httpd
08:01:04 PM 1734800015 10640 1.00 1.00 0.00 2.00 1 httpd
08:01:05 PM 1734800035 1448 34.00 3.00 0.00 37.00 4 php-fpm
08:01:05 PM 1734800035 1454 61.00 5.00 0.00 66.00 0 php-fpm

So for me the result shows, either I can reduce work PHP needs to do on compiling / working on code (caching) or just live with it.

6

This isnā€™t relevant.

Ctrl+F5 loads in 2 seconds for me.

Your issue is PHP configuration, webserver configuration, cache configuration, etc.

7

That sounds about rightā€¦
NC on Ubuntu 20.04 with php7.4 ā€œfeelsā€ faster than 18.04 with 7.3ā€¦

8

There weā€™re in common, its about PHP. Version or config.
the 2 seconds cache-empty reload i would really like to see though.

Thanks everybody.

9

A little extra could be HTTP/2 and mpm_event with fcgi / php-fpm

10

I donā€™t think PHP itself explains 15 sec vs. 2 sec differenceā€¦

My very first NC install on scrap Dell PE2950 (13 years old) with 2GB RAM, 2 CPUs and 500GB HDD
does the CTRL+F5 in under 10 seconds (Ubuntu 16.04, PHP 7.2)ā€¦

11

A few things that this installation differs from the ubuntu defaults

(read up on all of them, they can lead to some weird side effects)

php-fpm:

pm = dynamic
pm.max_children = 64
pm.start_servers = 5
pm.min_spare_servers = 4
pm.max_spare_servers = 10

php settings:

realpath_cache_size = 1M  # (https://www.php.net/manual/en/function.realpath-cache-size.php)
realpath_cache_ttl = 180

opcache.enable=1 # (https://www.php.net/manual/en/book.opcache.php)
opcache.enable_cli=1
opcache.memory_consumption=128
opcache.revalidate_freq=1

mysql (the nextcloud DB is 400MB):

innodb_buffer_pool_size        = 1G
key_buffer_size                = 64MB
tmp_table_size                 = 64M

What i am trying to achive here is, that everything that needs to be accessed , is already in memory, so that you have little to no disk access when using nextcloud. (Apart from accessing stored files or writes), since the SSDs i am using are super old and not really fast (and the disks are even worse).

The thing that i would investigate in your setup are the disks.

2x7200rpm on a Hypervisor is horrible for performance.
Spinning rust gives you about 150 IOps. (or in your case, 150 Writing, 300 read because of the raid 1).

Especially if you donā€™t use any opcode caching you have a lot of tiny io operations reading all the files without transfering much data