Nextcloud Overload

dzidek23 · November 28, 2022, 3:44pm

sorry to hear that,
I hope you’ve looked at /etc/redis/redis.conf?

Look here for guides on how to setup redis:
https://docs.nextcloud.com/server/24/admin_manual/configuration_server/caching_configuration.html#id2

FSF · November 29, 2022, 8:26am

indeed I forgot to modify the redis.conf file

for confirmation, I modify the directives as follows ?

# Accept connections on the specified port, default is 6379 (IANA #815344).
# If port 0 is specified Redis will not listen on a TCP socket.
port 6379

# Unix socket.
#
# Specify the path for the Unix socket that will be used to listen for
# incoming connections. There is no default, so Redis will not listen
# on a unix socket when not specified.
#
#unixsocket /var/run/redis/redis-server.sock
#unixsocketperm 770

FSF · January 16, 2023, 10:19am

Hello and best wishes 2023 to all

On January 4th I made the modifications to use Redis in Nextcloud with a reboot.
Today January 16th, the server is so slow and overloaded that I have to reboot it
Here are the screenshots of Htop

https://zupimages.net/up/23/03/ms4d.jpg
https://zupimages.net/up/23/03/tuz8.jpg

We can see that it is collabora online which is responsible for this RAM overload.

How to solve this problem because it’s really disabling for a server in production.

Thanks for your help

FSF · January 23, 2023, 9:08am

Hello

And here it is, 6 days after the last reboot, the overload problem reappears.
For the moment the only solution I have is to program a reboot of the server every two days.

Now I really need help to find a solution

Thank you in advance.

wwe · January 23, 2023, 8:50pm

maybe it’s time to perform systematic troubleshooting. You left many questions about your system unanswered so far. Please start over and fill out the required support template.

From the screenshot it looks you use built-in CODE server - I think not many use such a system with 10+ active users… this is known to cause slow-down:

github.com/nextcloud/richdocuments

RichDocuments built-in CODE causes significant slowdown/timeout issues

opened 01:12PM - 07 Dec 20 UTC

eganonoa

bug 2. developing high performance integration: built-in code server

Over the last couple of weeks Snap and Docker versions of Nextcloud have updated… to version 2.0.2 (snap on Friday). After the upgrade the connection to Nextcloud, particularly via the web-interface, slows to a crawl and/or times out. It appears to be caused by richdocuments/collabora. What is strange about this is that you cannot see easily identify the problem: basic analytics show no major increase in load on the CPU, Ram or I/O, the logs don't immediately show anything out of the ordinary. If you open top/htop on the server, however, and observe for a while you can see big, temporary spikes where available CPU's are maxed out by richdocuments/the Appimage related to Collabora and the Code server. When you go to the admin settings under collabora you can see an error that says that it cannot establish a connection to the in-built CODE server. The performance problem can be fixed by disabling both the collabora and code server apps: either in the web interface if you can get there through the performance issues, or in the server terminal (sudo snap run nextcloud.occ app:disable richdocuments ; sudo snap run nextcloud.occ app:disable richdocumentscore). After doing that normal service is resumed entirely across the Nextcloud server. This indicates that the failure to establish a connection to the inbuilt CODE server is the underlying problem and that there is something that happens when upgrading from Nextcloud v. 2.0.1 to 2.0.2 that causes it to break. Strangely, however, this is an issue that only appears to be present in Nextcloud Snap (where slowdown occurs) and Docker (where timeouts occur) and doesn't appear to have been reported by users of the regular Nextcloud: see threads on Nextcloud Community Forum (https://help.nextcloud.com/t/nextcloud-20-0-2snap2-interface-incredibly-slow/100304), Nextcloud Snap Github (https://github.com/nextcloud/nextcloud-snap/issues/1569), and Nextcloud Reddit (https://www.reddit.com/r/NextCloud/comments/k7b1of/extremely_slow_interface_nextcloud_snap/).

you should try separate COOL installation (maybe as docker container) and in case the issue happens there as well report the issue at COllaboraOnLine forum/github.

FSF · January 24, 2023, 8:04am

if I didn’t fill in the form it’s because it hasn’t changed from the first post, but I can fill it up again no worries:

Support intro

Nextcloud version (eg, 20.0.5): 24.0.5
Operating system and version (eg, Ubuntu 20.04): Debian 11
Apache or nginx version (eg, Apache 2.4.25): Apache 2.4.54
PHP version (eg, 7.4): 8.1.11

The issue you are facing:

Is this the first time you’ve seen this error? (Y/N): N

Steps to replicate it: Leave the server in production without reboot between 1 and 2 weeks

The output of your Nextcloud log in Admin > Logging:

Nothing special (no error)

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => 'xxx',
  'passwordsalt' => 'xxx',
  'secret' => 'xxx',
  'trusted_domains' =>
  array (
    0 => 'xxx.xxx.xxx.xxx',
    1 => 'xxx.xxx.com',
    2 => 'xxx.xxx.lan',
  ),
  'datadirectory' => '/DataNextcloud',
  'dbtype' => 'mysql',
  'version' => '24.0.5.1',
  'overwrite.cli.url' => 'https://xxx.xxx.com',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud',
  'dbpassword' => 'xxx',
  'installed' => true,
  'default_phone_region' => 'FR',
  'htaccess.RewriteBase' => '/',
  'ldapProviderFactory' => 'OCA\\User_LDAP\\LDAPProviderFactory',
  'mail_from_address' => 'xxx',
  'mail_smtpmode' => 'smtp',
  'mail_sendmailmode' => 'smtp',
  'mail_domain' => 'xxx.com',
  'mail_smtpport' => '25',
  'mail_smtphost' => 'xxx.xxx.xxx.xxx',
  'lost_password_link' => 'disabled',
  'auth.webauthn.enabled' => false,
  'session_lifetime' => 72000,
  'session_keepalive' => false,
  'auto_logout' => true,
  'filelocking.enabled' => 'true',
  'skeletondirectory' => '',
  'redis' =>
  array (
    'host' => 'localhost',
    'port' => '6379',
    'timeout' => '0.0',
  ),
  'memcache.local' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'maintenance' => false,
  'logfile' => '/var/log/nextcloud/nextcloud.log',
  'loglevel' => 2,
  'logtimezone' => 'Europe/Paris',
  'log_rotate_size' => 104857600,
  'trashbin_retention_obligation' => 'auto, 60',
  'theme' => '',
  'app_install_overwrite' =>
  array (
    0 => 'integration_whiteboard',
  ),
);

The output of your Apache/nginx/system log in /var/log/____:

Nothing special in log file (no error)

Nothing special in log file (no error)

Output errors in nextcloud.log in /var/www/ or as admin user in top right menu, filtering for errors. Use a pastebin service if necessary.
nothing special in log file (no error)

I’m going to look at docker even if I’ve never used this system and I don’t know yet how to integrate it with nextcloud.

wwe · January 24, 2023, 6:54pm

doesn’t help… please review check apache log, increase log level etc… there must be some trace to the faulty activity…

FSF · January 25, 2023, 8:10am

Hello

No problem, I want us to find a solution.

Here is what I could extract from the apache logs (the logs are between 8 and 10 Mb)

in /var/log/apache2/error.log (many lines of this type)

::1 - - [25/Jan/2023:07:46:36 +0100] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.54 (Debian) OpenSSL/1.1.1n (internal dummy connection)"

in in /var/log/apache2/access.log (hundreds of lines of this type for exemple)

xxx.domain.lan:443 10.1.0.200 - - [24/Jan/2023:09:59:52 +0100] "POST /apps/richdocumentscode/proxy.php?req=/cool/https%3A%2F%2Fxxx.domain.com%2Findex.php%2Fapps%2Frichdocuments%2Fwopi%2Ffiles%2F127156_ocru8zmmu4cy%3Faccess_token%3DxWQiPOVgzHayI2gMkL7oZ8gY1dMcydp5%26access_token_ttl%3D0/ws?WOPISrc=https%3A%2F%2Fxxx.domain.com%2Findex.php%2Fapps%2Frichdocuments%2Fwopi%2Ffiles%2F127156_ocru8zmmu4cy&compat=/ws/2b86107c0b7126b25ebcd5e49c0f1f31/write/402 HTTP/1.1" 200 567 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0"

and

xxx.domain.lan:443 87.89.7.81 - - [24/Jan/2023:11:23:57 +0100] "POST /apps/richdocumentscode/proxy.php?req=/cool/https%3A%2F%2Fxxx.domain.com%2Findex.php%2Fapps%2Frichdocuments%2Fwopi%2Ffiles%2F2012_ocru8zmmu4cy%3Faccess_token%3DadgFUHTsJxooh68LCiMnW58XjAt7tybI%26access_token_ttl%3D0/ws?WOPISrc=https%3A%2F%2Fxxx.domain.com%2Findex.php%2Fapps%2Frichdocuments%2Fwopi%2Ffiles%2F2012_ocru8zmmu4cy&compat=/ws/6e89d65c1918c949410d0b44170538c2/write/138 HTTP/1.1" 200 567 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"

in in /var/log/apache2/nextcloud-error.log (many lines of this type for exemple)

[Wed Jan 25 00:00:10.851609 2023] [ssl:warn] [pid 692] AH01909: xxx.domain.lan:443:0 server certificate does NOT include an ID which matches the server name
[Wed Jan 25 08:47:35.337223 2023] [php:notice] [pid 18075] [client 172.16.10.15:60210] richdocumentscode (proxy.php) error exit, PID: 18075, Message: The param should be 'status' or 'req=...', but is: ''
[Wed Jan 25 08:47:36.675712 2023] [php:notice] [pid 18807] [client 172.16.10.15:60276] richdocumentscode (proxy.php) error exit, PID: 18807, Message: The param should be 'status' or 'req=...', but is: ''
[Wed Jan 25 08:47:37.925531 2023] [php:notice] [pid 18807] [client 172.16.10.15:60276] richdocumentscode (proxy.php) error exit, PID: 18807, Message: The param should be 'status' or 'req=...', but is: ''
[Wed Jan 25 08:47:39.167347 2023] [php:notice] [pid 18807] [client 172.16.10.15:60276] richdocumentscode (proxy.php) error exit, PID: 18807, Message: The param should be 'status' or 'req=...', but is: ''
[Wed Jan 25 08:47:57.671284 2023] [access_compat:error] [pid 26585] [client 87.89.7.81:45462] AH01797: client denied by server configuration: /var/www/nextcloud/config
[Wed Jan 25 08:48:25.884724 2023] [php:notice] [pid 18077] [client 172.16.10.15:60274] richdocumentscode (proxy.php) error exit, PID: 18077, Message: No content in reply from coolwsd. Is SSL enabled in error ?

Thank you for your help and your analysis

wwe · January 25, 2023, 8:37pm

for me it sounds like richdocuments wants to talk with CODE using local name… this connection fails which might result in additional tries… sounds like major config issue but starting from the point your installation works in general it doesn’t make sense.

please describe more details e.g. how did you install Nextcloud, Apache config, reverse proxy, DNS records, verify DNS resolution (e.g. if you mapped some public DNS to local IPs, is it same for server and clients?) etc…

did you try searching the errors?

seems to result from wrong Apache config. do you access the server on xxx.domain.lan or did you configure xxx.domain.lan as Apache hostname?

FSF · January 26, 2023, 8:44am

Hi

This the tuto i followed for install Nextcloud24 (But the owner has upgraded the page for nextcloud25).

This is my file /etc/apache2/sites-available/nextcloud.conf

<VirtualHost 172.16.0.114:80>
       ServerName xxx.domain.com
       Redirect permanent / https://xxx.domain.com
</VirtualHost>

<VirtualHost 172.16.0.114:443>
        ServerName xxx.domain.lan
        ServerAlias xxx.domain.com
        DocumentRoot "/var/www/nextcloud/"
        <Directory "/var/www/nextcloud/">
                Options Indexes FollowSymLinks MultiViews
                AllowOverride All
                Order allow,deny
                Allow from all
        </Directory>
        SSLEngine on
        SSLVerifyClient none
        SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0ean-shutdown downgrade-1.0 force-response-1.0
        SSLProtocol -all +TLSv1.2 +TLSv1.3
        SSLCipherSuite SSL ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        SSLCipherSuite TLSv1.3 TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256
        SSLHonorCipherOrder on
        SSLOpenSSLConfCmd Curves X25519:secp521r1:secp384r1:prime256v1
        SSLCertificateFile /etc/letsencrypt/live/xxx.domain.com/fullchain.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/xxx.domain.com/privkey.pem
        #Include /etc/letsencrypt/options-ssl-apache.conf
        #SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0
        ServerSignature Off
        <IfModule mod_dav.c>
                Dav off
        </IfModule>
        <IfModule mod_headers.c>
                Header always set Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"
        </IfModule>
        ErrorLog /var/log/apache2/nextcloud-error.log
        LogLevel warn

</virtualHost>

From the local network, you can connect to the server via the address .com or .lan

My local DNS server is under windows server, the local fqdn xxx.domain.lan is directed to the ip of the nextcloud server (172.16.0.114) and idem for the fqdn xxx.domain.com which is directed to 172.16.0.114.

I have a public IP dedicated to nextcloud which is directly directed to the private IP by my UTM (the server is self-hosted).

My .com domain is managed at OVH and the nextcloud fqdn xxx.domain.com is of course directed to my public IP.

I don’t have a reverseproxy on this IP address since it is dedicated to this Nextcloud server.

Thank you for your help

wwe · January 26, 2023, 11:56am

having both xxx.domain.com and xxx.domain.lan as A records pointing to the same IP (and reverse-lookup possibly pointing to .lan) might confuse the systems when they try to auto-detect it’s hostname. with local DNS in place you can always connect using xxx.domain.com I would recommend you get rid of the .lan hostname everywhere. This definitely simplifies configuration. you can keep the .lan DNS record as CNAME in the DNS just for the case somebody keeps using it.

Give it a try and check the logs if the number of connections decrease after change…

FSF · January 30, 2023, 2:06pm

so if I understand correctly what you mean, I have to remove all references to xxx.domain.lan in my apache2 configuration and in my windows server DNS?

wwe · January 30, 2023, 5:45pm

yes I would change to public DNS everywhere, maybe you need to restart a serve as well to pickup the changes and check if it helps…

crobarcro · March 15, 2024, 4:53pm

Did it help?