Web interface: Very slow/bad performance

Nextcloud version (eg, 20.0.5): 25.0.2
Operating system and version (eg, Ubuntu 20.04): Ubuntu 22.04
Apache or nginx version (eg, Apache 2.4.25): nginx/1.18.0 (Ubuntu)
Hosting: Netcup KVM Server VPS 2000 G9

More info PHP

 PHP

Version: 8.1.2

Memory limit: 2 GB

Max execution time: 3600

Upload max size: 10 GB

Extensions: Core, date, libxml, openssl, pcre, zlib, filter, hash, json, Reflection, SPL, session, standard, sodium, cgi-fcgi, mysqlnd, PDO, xml, apcu, bcmath, bz2, calendar, ctype, curl, dom, mbstring, FFI, fileinfo, ftp, gd, gettext, gmp, iconv, igbinary, imagick, intl, ldap, exif, mysqli, pdo_mysql, Phar, posix, readline, redis, shmop, SimpleXML, smbclient, sockets, sysvmsg, sysvsem, sysvshm, tokenizer, xmlreader, xmlwriter, xsl, zip, libsmbclient, Zend OPcache
Database

Type: mysql

Version: 10.6.11

Size: 506 MB

The issue you are facing:

Is this the first time you’ve seen this error? N

Steps to replicate it:

Right now I tend to believe it’s something with nginx/php settings or caching. Any idea or hint is very much welcomed.

  1. Web Interface is incredible slow, loading files app 30s for hard-refreshed load and 15s afterwards.

This is Chrome DevTools HAR File to import dashboard loading: https://easyupload.io/x73vii

Or here as a screenshot

Debug of DevTools

https://pastebin.com/H87rvp27

This is GTmetrix of my login page: https://easyupload.io/vi99pq (PDF)
And Pingdom HAR of my login page: https://easyupload.io/wa3v7l

The output of your Nextcloud log in Admin > Logging:

see at the end of post

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

$CONFIG = array (
  'instanceid' => '###',
  'passwordsalt' => '###',
  'secret' => '###',
  'trusted_domains' =>
  array (
    0 => 'cloud.###',
  ),
  'datadirectory' => '/home/nc_prod_files/data',
  'tempdirectory' => '/home/nc_prod_files/tmp',
  'default_language' => 'de',
  'force_locale' => 'de',
  'overwriteprotocol' => 'https',
  'overwrite.cli.url' => 'https://cloud.###',
  'lost_password_link' => 'https://xxx/?action=lostpassword',
  'dbtype' => 'mysql',
  'version' => '25.0.2.3',
  'dbname' => '###_cloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => '###_cloud',
  'dbpassword' => '###',
  'installed' => true,
  'mail_from_address' => 'automail',
  'mail_smtpmode' => 'smtp',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_domain' => 'xxx',
  'maintenance' => false,
  'default_phone_region' => 'DE',
  'theme' => '',
  'loglevel' => 3,
  'filelocking.enabled' => true,
  'debug' => true,
  'logfile' => '/var/log/nextcloud_prod/nextcloud.log',
  'log_type' => 'file',
  'maintenance_window_start' => 2,
  'logtimezone' => 'Europe/Berlin',
  'log_rotate_size' => '104857600',
  'activity_expire_days' => 14,
  'blacklisted_files' =>
  array (
    0 => '.htaccess',
    1 => 'Thumbs.db',
    2 => 'thumbs.db',
  ),
  'cron_log' => true,
  'skeletondirectory' => '',
  'reference_opengraph' => true,
  'enable_previews' => true,
  'memcache.local' => '\OC\Memcache\APCu',
  'memcache.locking' => '\OC\Memcache\Redis',
  'redis' =>
  array (
    'host' => '/var/run/redis/redis-server.sock',
    'dbindex' => 0,
    'port' => 0,
    'timeout' => 1.5,
  ),
  'preview_max_x' => 1024,
  'preview_max_y' => 768,
  'preview_max_scale_factor' => 1,
  'enabledPreviewProviders' =>
  array (
    0 => 'OC\\Preview\\PNG',
    1 => 'OC\\Preview\\JPEG',
    2 => 'OC\\Preview\\GIF',
    3 => 'OC\\Preview\\BMP',
    4 => 'OC\\Preview\\XBitmap',
    5 => 'OC\\Preview\\Movie',
    6 => 'OC\\Preview\\PDF',
    7 => 'OC\\Preview\\MP3',
    8 => 'OC\\Preview\\TXT',
    9 => 'OC\\Preview\\MarkDown',
  ),
  'updater.release.channel' => 'stable',
  'mail_smtpauth' => 1,
  'mail_smtphost' => 'mail.###',
  'mail_smtpport' => '465',
  'mail_smtpname' => 'automail@###',
  'mail_smtppassword' => '###',
  'mail_smtpsecure' => 'ssl',
  'auth.bruteforce.protection.enabled' => true,
  'app_install_overwrite' =>
  array (
    0 => 'apporder',
    1 => 'files_readmemd',
    2 => 'polls',
    3 => 'printer',
    4 => 'external',
    5 => 'calendar',
    6 => 'extract',
    7 => 'user_sql',
    8 => 'documentserver_community',
    9 => 'timetracker',
    10 => 'quickaccesssorting',
    11 => 'passman',
    12 => 'appointments',
    13 => 'flow_notifications',
    14 => 'forms',
    15 => 'hsts',
    16 => 'talk_simple_poll',
    17 => 'tasks',
    18 => 'telephoneprovider',
    19 => 'previewgenerator',
    20 => 'emlviewer',
    21 => 'side_menu',
    22 => 'zenodo',
    23 => 'geoblocker',
    24 => 'integration_whiteboard',
    25 => 'richdocuments',
    26 => 'pdfannotate',
    27 => 'spreed',
    28 => 'files_antivirus',
    29 => 'backup',
    30 => 'workflow_pdf_converter',
    31 => 'deck',
    32 => 'groupfolders',
    33 => 'impersonate',
    34 => 'keeweb',
    35 => 'issuetemplate',
    36 => 'ocr',
    37 => 'limit_login_to_ip',
    38 => 'transfer',
    39 => 'workflow_script',
  ),
  'integrity.check.disabled' => false,
  'mysql.utf8mb4' => true,
  'twofactor_enforced' => 'true',
  'twofactor_enforced_groups' =>
  array (
    0 => 'admin',
    1 => 'xxx-Admin',
  ),
  'twofactor_enforced_excluded_groups' =>
  array (
  ),
  'data-fingerprint' => '###',
  'log.condition' =>
  array (
    'apps' =>
    array (
      0 => 'admin_audit',
    ),
  ),
);

The output of your Apache/nginx/system log in /var/log/____:

nothing

PASTE HERE


Output errors in nextcloud.log in /var/www/ or as admin user in top right menu, filtering for errors. Use a pastebin service if necessary.

Bunch of those

Error	PHP	Return type of OCA\UserSQL\Properties::offsetExists($offset) should either be compatible with ArrayAccess::offsetExists(mixed $offset): bool, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice at /var/www/nextcloud_prod/apps/user_sql/lib/Properties.php#206	
2022-12-31T11:11:39+0100
Error	PHP	Return type of OCA\UserSQL\Properties::offsetExists($offset) should either be compatible with ArrayAccess::offsetExists(mixed $offset): bool, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice at /var/www/nextcloud_prod/apps/user_sql/lib/Properties.php#206	
2022-12-31T11:11:39+0100
Error	PHP	Return type of OCA\UserSQL\Properties::offsetExists($offset) should either be compatible with ArrayAccess::offsetExists(mixed $offset): bool, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice at /var/www/nextcloud_prod/apps/user_sql/lib/Properties.php#206	
2022-12-31T11:11:38+0100

There is a basic setup for the database that is better than the default setups:
https://docs.nextcloud.com/server/latest/admin_manual/configuration_database/linux_database_configuration.html#configuring-a-mysql-or-mariadb-database

However, the size of the caches depends also on your setup and can be adjusted with the help of scripts like tuning-primer.sh:

If that does not help, you can try to figure out which part is slowing down your responses, if there are resources runnin low (cpu, ram, i/o performance) and if the database is involved, the slow-sql-queries can help to reveal which part of the code it might be.
The logfiles itself can be a good hint as well.

The number of apps installed, I’d perhaps check a bit if one app might cause the issue (either find hints in the logs or just enable/disable trail-and-error).

If you say web-access is slow, access via webdav is faster?

Thank you very much!

Did that tuning script, rose table open cache and table definition cache, maybe that helped a little bit. Afterwards this is what that script echoed. https://pastebin.com/xCnAJ7xM

Honestly I haven’t had database in my mind, but it sounds reasonable when it comes to loading times of apps which is still bad. Any further ideas?

However, also the loading of the login page is slow. GTmetrix gives an E and says

Avoid enormous network payloads		
Large network payloads cost users real money and are highly correlated with long load times.

URL	TRANSFER SIZE
https://cloud.xxx.de/dist/core-common.js?v=92c47e36-37
3.13MB
https://cloud.xxx.de/apps/side_menu/js/sideMenu.js?v=92c47e36-37
630KB
https://cloud.xxx.de/apps/theming/image/background?v=37
622KB
https://cloud.xxx.de/dist/core-main.js?v=92c47e36-37
50.0KB
https://cloud.xxx.de/dist/icons.css
31.5KB
https://cloud.xxx.de/dist/core-login.js?v=92c47e36-37
31.5KB
https://cloud.xxx.de/core/css/server.css?v=92c47e36-37
28.0KB
https://cloud.xxx.de/login
7.75KB
https://cloud.xxx.de/apps/theming/image/logo?v=37
7.36KB
https://cloud.xxx.de/apps/theming/image/logo?useSvg=1&v=37
7.35KB

So, core-common is and should be 3.13MB?

Pingdom gives for login page 4.8MB and 42 requests but as the pingdom screenshot doesn’t show the full page I don’t think that’s all. But even that is quite big?!

F0	Make fewer HTTP requests
F0	Use cookie-free domains
F0	Add Expires headers

Page Speed insights gives

Registriert einen unload-Listener
Das `unload`-Ereignis wird nicht zuverlässig ausgelöst. Wenn der Listener darauf wartet, kann das Browseroptimierungen wie den Back-Forward-Cache beeinträchtigen. Verwenden Sie stattdessen `pagehide`- oder `visibilitychange`-Ereignisse. Weitere Informationen
Quelle
/dist/core-common.js?v=92c47e36-37:2:9934846(cloud.xxx.de)

and also

Empfehlung
Geschätzte Einsparung
Reduzieren Sie nicht verwendetes JavaScript
2,56 s
Sie können nicht verwendetes JavaScript reduzieren und das Laden von Skripts aufschieben, bis sie benötigt werden, um den Datenverbrauch durch Netzwerkaktivität zu senken. Weitere InformationenLCP
URL
Übertragungsgröße
Mögliche Einsparungen
/dist/core-common.js?v=92c47e36-37(cloud.xxx.de)
3.201,8 KiB
2.934,7 KiB
…js/sideMenu.js?v=92c47e36-37(cloud.xxx.de)
628,9 KiB
184,4 KiB
…./node_modules/@nextcloud/vue/dist/index.module.js
191,9 KiB
33,6 KiB
…./node_modules/@nextcloud/vue/node_modules/vue-color/dist/vue-color.min.js
21,0 KiB
14,2 KiB
…./node_modules/@nextcloud/vue/node_modules/ical.js/build/ical.js
16,7 KiB
13,5 KiB
…./node_modules/@nextcloud/vue/node_modules/vue/dist/vue.runtime.esm.js
14,6 KiB
12,4 KiB
…./node_modules/@nextcloud/vue/node_modules/@nextcloud/calendar-js/dist/index.esm.js
38,1 KiB
11,3 KiB
/dist/core-main.js?v=92c47e36-37(cloud.xxx.de)
50,1 KiB
27,6 KiB
/dist/core-login.js?v=92c47e36-37(cloud.xxx.de)
31,6 KiB
22,5 KiB
…/nextcloud/core/src/OC/dialogs.js
6,2 KiB
6,1 KiB
…/nextcloud/core/src/OC/contactsmenu/contact.handlebars
2,8 KiB
2,7 KiB
…/nextcloud/node_modules/moment/locale|sync|/^\.\/.*$
1,3 KiB
1,3 KiB
…/nextcloud/core/src/OC/contactsmenu.js
1,4 KiB
1,1 KiB
…/nextcloud/core/src/OC/backbone-webdav.js
0,9 KiB
0,9 KiB

Or maybe it’s something about my php-fpm settings? This is what I have in my www.conf

pm.max_children = 120

; The number of child processes created on startup.
; Note: Used only when pm is set to 'dynamic'
; Default Value: (min_spare_servers + max_spare_servers) / 2
pm.start_servers = 15

; The desired minimum number of idle server processes.
; Note: Used only when pm is set to 'dynamic'
; Note: Mandatory when pm is set to 'dynamic'
pm.min_spare_servers = 10

; The desired maximum number of idle server processes.
; Note: Used only when pm is set to 'dynamic'
; Note: Mandatory when pm is set to 'dynamic'
pm.max_spare_servers = 20

; The number of rate to spawn child processes at once.
; Note: Used only when pm is set to 'dynamic'
; Note: Mandatory when pm is set to 'dynamic'
; Default Value: 32
;pm.max_spawn_rate = 32

Had set it to static before, switched to dynamic and it feels a little bit faster now.

The interface is loaded much faster, but the data/values still needs time. So it’s still about the database?!

It’s fast enough and it’s not that important cause most of us only use the webinterface for files, calendar, talk, deck, mail.

Getting a good nc security score is soo easy compared to the performance scores.

Okay,

I had that setting in /etc/mysql/my.cnf

read_rnd_buffer_size = 1M 

from e.x. Nextcloud Installationsanleitung - Carsten Rieger IT-Services.

After some googling on mysql tuning for nextcloud I found https://haefelfinger.ch/posts/2021/2021-03-29-nextcloud-tuning/, and that person set

read_rnd_buffer_size = 4M 

I just overtook that setting, restarted services. For me with Ubuntu 22.04:

systemctl restart php8.1-fpm.service mariadb.service redis-server.service nginx.service

And it’s so much faster now. Loading of data is much faster, Nextcloud Talk App with Android getting the chats in a blink and not 10 seconds or so.

Gonna see if this has heavy downsides, therefore keeping this open for a bunch of days. Also maybe someone has some ideas for further tuning.

For the database performance tuning, it needs to run for some time (>=24h) not just 30 min.

The blog post you referred to, also says it got the suggestions from phpmyadmin based on their usage. Keep in mind that parameters can change a lot based on the use case (how many users, how much parallel usage, how much sharing, use of different type of app, …).

Can be, but you should be able to check the number of servers and children used. If you run close to the limits, then it should show in the logfiles of php-fpm as well.

That was considered less important by the developers because you usually visit the site very often so most of the static content is cached anyway. They tried to reduce the number of js/css/… files by combining an minimizing these files but because each app can use different js/css/… and that didn’t work well and they removed it.
When you navigate within the interface, you should see in the network analysis that a lot of content is cached.

You have set up system cron-jobs I suppose…