Collabora Online super slow - not all icons appearing

AurelienBISOTTI · November 16, 2020, 8:58pm

Nextcloud version (eg, 18.0.2): 20.0.1.1
Operating system and version (eg, Ubuntu 20.04): 20.04.1 LTS
Apache or nginx version (eg, Apache 2.4.25): nginx 1.18.0.1
PHP version (eg, 7.1): PHP 7.4 and PHP 7.3 (should I remove one ?)

The issue you are facing: I have a good server, yet Collabora is superslow and I’m missing some icons.

I use Plesk 18.0.31

CPU :
4 Core x 3.8 GHz (E3-1270 v6)
RAM :
64 Go
NVMe
2 x 1000 GB Hardware RAID 1

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:

Installed Nextcloud from Plesk
Install the “Collabora Online - Built-in CODE Server” from Nextcloud didn’t work
I did the command over SSH to install it "sudo -u MYNAME php -d memory_limit=512M occ app:install richdocumentscode " from the folder where “oc” was
It did not work the first time
I had to install all those modules to make the occ stuff work (maybe some aren’t mandatory)

apt-get install php-mysql
apt-get install php-pgsql
apt-get install php-sqlite3
apt-get install php-xml
apt-get install php-zip
apt-get install php-mbstring
apt-get install php-curl
apt-get install php-gd

The “sudo -u MYNAME php -d memory_limit=512M occ app:install richdocumentscode” worked
Installed Collabora Online from Nextcloud
I could open a .ods in Nextcloud, but the server crashed after a few seconds
Went into PHP7.4 logs, found a memory limit warning (limit was still 128MB)
Nano’ed every php.ini on the serveur, put every memory_limit = -1 (I’m a madlad, also I have 64Go RAM so that souldn’t be a problem I guess ?)
Now I can open a .ods from nexctloud/collabora without it crashing the server, yet it is suuuuuper slow.

The output of your Nextcloud log in Admin > Logging: basically those two repeating :

Error: Undefined offset: 3 at /var/www/vhosts/XXXXX.fr/nextcloud.XXXXX.fr/lib/private/legacy/OC_Helper.php#548

Module 'sqlite3' already loaded at Unknown#0

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'passwordsalt' => 'XXXXXX',
  'secret' => 'XXXXXXX',
  'trusted_domains' =>
  array (
    0 => 'localhost',
    1 => 'XXXXXXX',
  ),
  'datadirectory' => '/var/www/vhosts/XXXXXX/.nextcloud/data/XXXXXX',
  'dbtype' => 'mysql',
  'version' => '20.0.1.1',
  'overwrite.cli.url' => 'http://localhost',
  'dbname' => 'nextcloud_XXXXXX',
  'dbhost' => 'localhost:3306',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'ncuser_XXXXXX',
  'dbpassword' => 'XXXXXXX',
  'installed' => true,
  'instanceid' => 'oc1o0tzdxrwe',
  'mail_smtpmode' => 'smtp',
  'mail_smtpsecure' => 'ssl',
  'mail_sendmailmode' => 'smtp',
  'mail_from_address' => 'contact',
  'mail_domain' => 'XXXXXX',
  'mail_smtphost' => 'XXXXXX',
  'mail_smtpport' => '465',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpauth' => 1,
  'mail_smtpname' => 'XXXXXXXX',
  'mail_smtppassword' => 'XXXXXXl',
  'app_install_overwrite' =>
  array (
    0 => 'drawio',
  ),
);

The output of your Apache/nginx/system log in /var/log/____:

nginx error : 
2020/11/16 19:48:59 [alert] 1415#0: *1 open socket #3 left in connection 3   
2020/11/16 19:48:59 [alert] 1415#0: *3 open socket #16 left in connection 4   
2020/11/16 19:48:59 [alert] 1415#0: *749 open socket #20 left in connection 11         
2020/11/16 19:48:59 [alert] 1415#0: *748 open socket #19 left in connection 20  
2020/11/16 19:48:59 [alert] 1415#0: *130 open socket #18 left in connection 25   
2020/11/16 19:48:59 [alert] 1415#0: *750 open socket #21 left in connection 36
2020/11/16 19:48:59 [alert] 1415#0: *747 open socket #17 left in connection 48
2020/11/16 19:48:59 [alert] 1415#0: aborting  
2020/11/16 20:56:24 [alert] 16700#0: *3914 open socket #12 left in connection 3   
2020/11/16 20:56:24 [alert] 16700#0: aborting

apache error.log :

[Mon Nov 16 19:52:30.853259 2020] [core:notice] [pid 2223:tid 140227414735936] AH00094: Command line: '/usr/sbin/apache2'                                                                                             [Mon Nov 16 19:57:44.140525 2020] [mpm_event:notice] [pid 2223:tid 140227414735936] AH00493: SIGUSR1 received.  Doing graceful restart                                                                                [Mon Nov 16 19:57:47.300905 2020] [ssl:warn] [pid 2223:tid 140227414735936] AH01909: default-82_165_243_155:443:0 server certificate does NOT include an ID which matches the server name
[Mon Nov 16 19:57:47.303851 2020] [mpm_event:notice] [pid 2223:tid 140227414735936] AH00489: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1f mod_fcgid/2.3.9 configured -- resuming normal operations
[Mon Nov 16 19:57:47.303867 2020] [core:notice] [pid 2223:tid 140227414735936] AH00094: Command line: '/usr/sbin/apache2'                                                                                             [Mon Nov 16 19:58:11.144971 2020] [mpm_event:notice] [pid 2223:tid 140227414735936] AH00493: SIGUSR1 received.  Doing graceful restart
[Mon Nov 16 19:58:14.286249 2020] [ssl:warn] [pid 2223:tid 140227414735936] AH01909: default-82_165_243_155:443:0 server certificate does NOT include an ID which matches the server name
[Mon Nov 16 19:58:14.289176 2020] [mpm_event:notice] [pid 2223:tid 140227414735936] AH00489: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1f mod_fcgid/2.3.9 configured -- resuming normal operations
[Mon Nov 16 19:58:14.289191 2020] [core:notice] [pid 2223:tid 140227414735936] AH00094: Command line: '/usr/sbin/apache2'                                                                                             [Mon Nov 16 19:58:30.358702 2020] [mpm_event:notice] [pid 2223:tid 140227414735936] AH00493: SIGUSR1 received.  Doing graceful restart                                                                                [Mon Nov 16 19:58:33.497847 2020] [ssl:warn] [pid 2223:tid 140227414735936] AH01909: default-82_165_243_155:443:0 server certificate does NOT include an ID which matches the server name                             [Mon Nov 16 19:58:33.500803 2020] [mpm_event:notice] [pid 2223:tid 140227414735936] AH00489: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1f mod_fcgid/2.3.9 configured -- resuming normal operations
[Mon Nov 16 19:58:33.500817 2020] [core:notice] [pid 2223:tid 140227414735936] AH00094: Command line: '/usr/sbin/apache2'                                                                                             [Mon Nov 16 20:56:27.694319 2020] [mpm_event:notice] [pid 2223:tid 140227414735936] AH00491: caught SIGTERM, shutting down                                                                                            [Mon Nov 16 20:59:53.940057 2020] [ssl:warn] [pid 1190:tid 140654138039360] AH01909: default-82_165_243_155:443:0 server certificate does NOT include an ID which matches the server name
[Mon Nov 16 20:59:53.940608 2020] [:notice] [pid 1190:tid 140654138039360] ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/) configured.
[Mon Nov 16 20:59:53.940616 2020] [:notice] [pid 1190:tid 140654138039360] ModSecurity: APR compiled version="1.6.5"; loaded version="1.6.5"                                                                          [Mon Nov 16 20:59:53.940621 2020] [:notice] [pid 1190:tid 140654138039360] ModSecurity: PCRE compiled version="8.39 "; loaded version="8.39 2016-06-14"
[Mon Nov 16 20:59:53.940625 2020] [:notice] [pid 1190:tid 140654138039360] ModSecurity: LUA compiled version="Lua 5.1"                                                                                                [Mon Nov 16 20:59:53.940628 2020] [:notice] [pid 1190:tid 140654138039360] ModSecurity: LIBXML compiled version="2.9.10"                                                                                              [Mon Nov 16 20:59:53.940631 2020] [:notice] [pid 1190:tid 140654138039360] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
[Mon Nov 16 20:59:53.941098 2020] [suexec:notice] [pid 1190:tid 140654138039360] AH01232: suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec)                                                                 [Mon Nov 16 20:59:54.201591 2020] [ssl:warn] [pid 2074:tid 140654138039360] AH01909: default-82_165_243_155:443:0 server certificate does NOT include an ID which matches the server name                             [Mon Nov 16 20:59:54.209878 2020] [mpm_event:notice] [pid 2074:tid 140654138039360] AH00489: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1f mod_fcgid/2.3.9 configured -- resuming normal operations
[Mon Nov 16 20:59:54.209911 2020] [core:notice] [pid 2074:tid 140654138039360] AH00094: Command line: '/usr/sbin/apache2'                                                                                             [Mon Nov 16 21:26:42.435052 2020] [mpm_event:notice] [pid 2074:tid 140654138039360] AH00491: caught SIGTERM, shutting down                                                                                            [Mon Nov 16 21:30:07.923872 2020] [ssl:warn] [pid 1260:tid 140476904582208] AH01909: default-82_165_243_155:443:0 server certificate does NOT include an ID which matches the server name                             [Mon Nov 16 21:30:07.924399 2020] [:notice] [pid 1260:tid 140476904582208] ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/) configured.
[Mon Nov 16 21:30:07.924406 2020] [:notice] [pid 1260:tid 140476904582208] ModSecurity: APR compiled version="1.6.5"; loaded version="1.6.5"
[Mon Nov 16 21:30:07.924410 2020] [:notice] [pid 1260:tid 140476904582208] ModSecurity: PCRE compiled version="8.39 "; loaded version="8.39 2016-06-14"                                                               [Mon Nov 16 21:30:07.924414 2020] [:notice] [pid 1260:tid 140476904582208] ModSecurity: LUA compiled version="Lua 5.1"                                                                                                [Mon Nov 16 21:30:07.924417 2020] [:notice] [pid 1260:tid 140476904582208] ModSecurity: LIBXML compiled version="2.9.10"                                                                                              [Mon Nov 16 21:30:07.924421 2020] [:notice] [pid 1260:tid 140476904582208] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
[Mon Nov 16 21:30:07.924892 2020] [suexec:notice] [pid 1260:tid 140476904582208] AH01232: suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec)                                                                 [Mon Nov 16 21:30:08.176560 2020] [ssl:warn] [pid 2111:tid 140476904582208] AH01909: default-82_165_243_155:443:0 server certificate does NOT include an ID which matches the server name                             [Mon Nov 16 21:30:08.184810 2020] [mpm_event:notice] [pid 2111:tid 140476904582208] AH00489: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1f mod_fcgid/2.3.9 configured -- resuming normal operations
[Mon Nov 16 21:30:08.184833 2020] [core:notice] [pid 2111:tid 140476904582208] AH00094: Command line: '/usr/sbin/apache2'

Also I have those messages from PHP7.4 logs :


[16-Nov-2020 21:30:10] WARNING: [pool nextcloud.XXXX.fr] child 6358 said into stderr: "sh: 1: disown: not found"                                                                                                
[16-Nov-2020 21:30:10] WARNING: [pool nextcloud.XXXX.fr] child 6358 said into stderr: "Logging at warning level to file: /tmp/loolwsd.gk0L9YWUaa/loolwsd.log"                                                  
 [16-Nov-2020 21:30:10] WARNING: [pool nextcloud.XX.fr] child 6358 said into stderr: "Security: Check for the 'lool' username overridden on the command line."                                                 
[16-Nov-2020 21:30:11] WARNING: [pool nextcloud.XX.fr] child 6358 said into stderr: "Init vcl"                                                                                                                
[16-Nov-2020 21:30:11] WARNING: [pool nextcloud.XX.fr] child 6358 said into stderr: "preload: merged unordf ucpchelp1 msforms vbaobj pcr vbaswobj sw animcore hwp flash chartcore solver sc wpftcalc xof ucpc>
[16-Nov-2020 21:30:12] WARNING: [pool nextcloud.XX.fr] child 6358 said into stderr: " vbaevents pdfimport mysql_jdbc ucpext hyphen spell lnth for ctl passwordcontainer svgio"                               
 [16-Nov-2020 21:30:12] WARNING: [pool nextcloud.XX.fr] child 6358 said into stderr: "Absent (often optional): sm sm sm sm sm sm sm sm sm sm sm smd "                                                          
[16-Nov-2020 21:30:12] WARNING: [pool nextcloud.XX.fr] child 6358 said into stderr: "Disabled: ucpftp1 rptxml dbu rptui rpt mork odbc sdbc2 cmdmail syssh ucphier1 ucpimage PresenterScreen abp dbp updatefee>
[16-Nov-2020 21:30:12] WARNING: [pool nextcloud.XX.fr] child 6358 said into stderr: "Whitelisted languages: de_DE en_GB en_US es_ES fr_FR it nl pt_BR pt_PT ru "                                            
[16-Nov-2020 21:30:12] WARNING: [pool nextcloud.XX.fr] child 6358 said into stderr: "Preloading dictionaries: de_DE en_US fr_FR it_IT nl_NL pt_BR ru_RU en_GB nl_BE pt_PT es_ES "                             [16-Nov-2020 21:30:13] WARNING: [pool nextcloud.XX.fr] child 6358 said into stderr: "Preloading thesauri: de_DE en_US fr_FR it_IT nl_NL pt_BR ru_RU en_GB nl_BE pt_PT es_ES "                                 
[16-Nov-2020 21:30:13] WARNING: [pool nextcloud.XX.fr] child 6358 said into stderr: "Preload icons"                                                                                                          
[16-Nov-2020 21:30:13] WARNING: [pool nextcloud.XX.fr] child 6358 said into stderr: "Preload languages"                                                                                                       
[16-Nov-2020 21:30:13] WARNING: [pool nextcloud.XX.fr] child 6358 said into stderr: "Preload fonts"                                                                                                        
 [16-Nov-2020 21:30:14] WARNING: [pool nextcloud.XX.fr] child 6358 said into stderr: "Ready to accept connections on port 9982."                                                                             
 [16-Nov-2020 21:30:57] WARNING: [pool nextcloud.XX.fr] child 6358 said into stderr: "/tmp/.mount_Collab6QoWIK/usr/bin/loolmount: unmount failed to detach [/tmp/loolwsd.gk0L9YWUaa/jails/qzCMdRHOby4jSqy5]: O>
[16-Nov-2020 21:30:57] WARNING: [pool nextcloud.XX.fr] child 6358 said into stderr: "/tmp/.mount_Collab6QoWIK/usr/bin/loolmount: forced unmount of [/tmp/loolwsd.gk0L9YWUaa/jails/qzCMdRHOby4jSqy5] failed: O>[16-Nov-2020 21:31:00] WARNING: [pool nextcloud.XX.fr] child 6358 said into stderr: "/tmp/.mount_Collab6QoWIK/usr/bin/loolmount: unmount failed to detach [/tmp/loolwsd.gk0L9YWUaa/jails/icNkUiRb7swyxvgF]: O>
[16-Nov-2020 21:31:00] WARNING: [pool nextcloud.XX.fr] child 6358 said into stderr: "/tmp/.mount_Collab6QoWIK/usr/bin/loolmount: forced unmount of [/tmp/loolwsd.gk0L9YWUaa/jails/icNkUiRb7swyxvgF] failed: O>[16-Nov-2020 21:31:06] WARNING: [pool nextcloud.XX.fr] child 6358 said into stderr: "/tmp/.mount_Collab6QoWIK/usr/bin/loolmount: unmount failed to detach [/tmp/loolwsd.gk0L9YWUaa/jails/6zyFXqvu1DhIH8VX]: O>
[16-Nov-2020 21:31:06] WARNING: [pool nextcloud.XX.fr] child 6358 said into stderr: "/tmp/.mount_Collab6QoWIK/usr/bin/loolmount: forced unmount of [/tmp/loolwsd.gk0L9YWUaa/jails/6zyFXqvu1DhIH8VX] failed: O>[16-Nov-2020 21:31:10] WARNING: [pool nextcloud.XX.fr] server reached max_children setting (10), consider raising it                                                                                          
[16-Nov-2020 21:32:04] WARNING: [pool nextcloud.XX.fr] child 6358 said into stderr: "/tmp/.mount_Collab6QoWIK/usr/bin/loolmount: unmount failed to detach [/tmp/loolwsd.gk0L9YWUaa/jails/61N2RCwC460tNhs7]: O>
[16-Nov-2020 21:32:04] WARNING: [pool nextcloud.XX.fr] child 6358 said into stderr: "/tmp/.mount_Collab6QoWIK/usr/bin/loolmount: forced unmount of [/tmp/loolwsd.gk0L9YWUaa/jails/61N2RCwC460tNhs7] failed: O>
[16-Nov-2020 21:34:05] WARNING: [pool nextcloud.XX.fr] server reached max_children setting (10), consider raising it                                                                                         
[16-Nov-2020 21:47:28] WARNING: [pool nextcloud.XX.fr] server reached max_children setting (10), consider raising it

AurelienBISOTTI · December 5, 2020, 3:11pm

Up !

None else got this problem ?

dev0 · December 5, 2020, 9:09pm

hi,
what are your mpm_prefork_module, mpm_worker_module and mpm_event_module setting in the apache2.conf?

AurelienBISOTTI · December 8, 2020, 11:12am

Hello,

I don’t find any of these (mpm_prefork_module, mpm_worker_module and mpm_event_module) in apache2.conf.

I have raised max_children from Plesk, things went a little better. But 48 max_children seems already high ?

dev0 · December 8, 2020, 12:35pm

Ah ok, you are using php via fast-cgi, that’s why you don’t have the settings named as mine. I think there are similar settings for child process control.

One additional thing you can try is running a collabora document while doing a ps aux | grep apache | wc -l to see how much instances are running of it.

AurelienBISOTTI · December 8, 2020, 1:06pm

Did as you said while running a doc.

ps aux | grep apache | wc -l
6

Is it good ?

On this other thread, they say the built-in Collabora CODE inside Nextcloud is trash and we should use a separate serveur to host CODE. What do you think about it ? Collabora & NC19 - Slow Performance

Edit : switched to demo server and all runs smooth. So there definitely is a problem with the built in serveur Collabora CODE

dev0 · December 8, 2020, 3:59pm

Actually I am not a fan with the built in document servers, because if something is not working then you have everything in one place and there are some issues with save stuff and so on.

In your case it makes sense, because the integrated docserver needs additional apache processes, of you let running that by a 3rd party service they have to run the additional instances.

However only 6 are not that many, however I am not proficient in the plesk config, I alter them by hand. Perhaps the ondemand setting is not fast enough spinning up new instances?

AurelienBISOTTI · December 8, 2020, 5:20pm

I don’t know, but even in the Nextcloud doc they say the CODE integrated server in Nextcloud is slow. So I don’t think this has anything to do with my setup.

I’m running on test server right now, and I’ll make a dedicated server when I’ll feel it.

Thanks anyway

dev0 · December 8, 2020, 5:37pm

Yes but this is not only slow, but there is another issue too, anyway you will get happier with the docker version, so good luck with that.