Diagnosing poor performance

ℹ️ Support
1

Nextcloud version (eg, 20.0.5): Nextcloud 24.0.4
Operating system and version (eg, Ubuntu 20.04): Debian 11
Web server: nginx 1.18
PHP version: 7.4.30

The issue you are facing: The web interface is incredibly slow. It takes several minutes for the first Files page to load.

The server load is huge (typically around 4, in ‘top’ on a 2 CPU, 4GB RAM server). There are thousands of PROPFIND requests for nearly-empty directories that take over 1s of PHP processing time to serve, for example.

The output of your Nextcloud log in Admin > Logging: Can’t. This page times out!

config.php: it’s mostly stuff I don’t want to share or is of little value. I think relevant parts are:

'memcache.local' => '\\OC\\Memcache\\Redis',       
'memcache.distributed' => '\\OC\\Memcache\\Redis', 
'memcache.locking' => '\\OC\\Memcache\\Redis',     
'redis' =>               
array (                  
 'host' => '127.0.0.1', 
 'port' => 6379,        
),

Output errors in nextcloud.log: nothing of much interest there. Every 5 minutes someone whose account is disabled tries a PROPFIND which generates an exception, but nothing else.

There’s a dozen or so users using 3.4.2 of the sync client for Linux.

I will update the clients to 3.6 once that’s released, but want to skip the whole 3.5 branch as it has some bugs that will be particularly annoying for users (e.g. constantly having to re-auth; being unable to access the main/settings windows) - I believe these are fixed in 3.6.

There’s several hundred GB of data on it, but the staff using sync clients are asked to keep their in-sync files to just what they need, this is typically much less than 1GB each.

Staff are not creating/deleting/editing lots and lots of documents; i.e. the load does not seem to make sense for the use.

Any tips gratefully received!

2

Hey,

can you share some insights into your php / webserver config ? What database are you using?
Also for single server apcu is recommended for local memcache: Memory caching — Nextcloud latest Administration Manual latest documentation Not sure if this is really something that might have an impact, though.

3

Thanks @SysKeeper

nginx is pretty straight forward, I believe I’m using the standard config.

php is running as FPM, with max_childern 200, start_servers: 10, min_spare_servers 5, max_spare_servers 30, max_requests 5000.

Though I rarely see more than half a dozen processes running. It seems to be slow at running.

Re ACPU:

If you are low on memory, use Redis for both.

is why I’m using redis for both. I only have 4GB. I’ll try switching to ACPU.

4

Good point! Missed that.
What about the DB you’re using?

5

Here’s a screenshot of top (after enabling APCU, ACPU or UCPA or whatever it is!) and me, one user, trying to load the main Files page. It took ~2mins to fully load (the README (blank!!) took the longest by far).

image
image1483×769 240 KB

The database , sorry missed that, is MariaDB 10.5.15. It’s using InnoDB, barracuda, DYNAMIC as the row format, has max connections 400,
and

key_buffer_size = 10485760 (~10MB)

innodb_buffer_pool_size = 318767104 (~319MB)

I set those values from some calculations based on giving it 1.5GB RAM, 400 connections, and it has a total index+data size of 3.5GB.

(The server also hosts a CRM and a couple of other websites; though those aren’t causing the nextcloud problems).

6

Which apps are enabled?

7

… and is opcache enabled?

8

Apps enabled

  • accessibility: 1.10.0
  • activity: 2.16.0
  • bruteforcesettings: 2.4.0
  • calendar: 3.4.2
  • circles: 24.0.1
  • cloud_federation_api: 1.7.0
  • comments: 1.14.0
  • contactsinteraction: 1.5.0
  • dav: 1.22.0
  • deck: 1.7.1
  • external: 4.0.0
  • federatedfilesharing: 1.14.0
  • federation: 1.14.0
  • files: 1.19.0
  • files_pdfviewer: 2.5.0
  • files_rightclick: 1.3.0
  • files_sharing: 1.16.2
  • files_trashbin: 1.14.0
  • files_versions: 1.17.0
  • files_videoplayer: 1.13.0
  • firstrunwizard: 2.13.0
  • impersonate: 1.11.0
  • logreader: 2.9.0
  • lookup_server_connector: 1.12.0
  • metadata: 0.16.0
  • nextcloud_announcements: 1.13.0
  • notifications: 2.12.0
  • oauth2: 1.12.0
  • password_policy: 1.14.0
  • photos: 1.6.0
  • polls: 3.7.0
  • privacy: 1.8.0
  • provisioning_api: 1.14.0
  • ransomware_protection: 1.13.0
  • recommendations: 1.3.0
  • serverinfo: 1.14.0
  • settings: 1.6.0
  • sharebymail: 1.14.0
  • spreed: 14.0.4
  • survey_client: 1.12.0
  • systemtags: 1.14.0
  • text: 3.5.1
  • theming: 1.15.0
  • twofactor_backupcodes: 1.13.0
  • updatenotification: 1.14.0
  • user_status: 1.4.0
  • viewer: 1.8.0
  • weather_status: 1.4.0
  • workflowengine: 2.6.0

OpCache: yes, but the “Overview” admin page does say

The OPcache buffer is nearly full. To assure that all scripts can be hold in cache, it is recommended to apply opcache.memory_consumption to your PHP configuration with a value higher than 128.

OPcache seems to be working at least:

image
image1808×1014 146 KB

9

I upped the opcache memory consumption to 512MB it does get rid of the ‘nearly full’ warning, but it doesn’t massively improve performance.

This vid shows a better experience than some times. Note that Reload was clicked right at the start of the vid.

nc