Nextcloud version: 15.0.10
Operating system and version: Debian 9
Apache or nginx version: Apache/2.4.29 (Ubuntu 18.0.4 LTS)
PHP version: PHP 7.0.33-0+deb9u3
The issue you are facing:
Timeouts or very slow connection between Apache Reverse Proxy and Nextcloud. Both are running on different VMs in a private cloud environment. The Apache VM has an external IP adress while Apache and Nextcloud communicate over their private IPs. After running perfectly fine for a few weeks it started to get very slow some hours ago. It runs fine for a few minutes, then gets very slow again.
We’re not sure if our cloud provider is having problems (they’re stating they’re not, but in our experience this isn’t that reliable) or if our configuration is wrong.
I was experimenting with HSTS earlier today, maybe this could be the cause?
Steps to replicate it:
- Connect to Nextcloud via URL
- Very long connection times in browser network view (20000 ms plus)
- Errors in Apache log.
Apache vhost configuration:
<IfModule mod_ssl.c>
<VirtualHost _default_:443>
ServerAdmin xy
ServerName xy
DocumentRoot /var/www/html
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
SSLEngine on
SSLCertificateFile /etc/apache2/ssl.crt/wildcard.domain.de-chain.crt
SSLCertificateKeyFile /etc/ssl/private/wildcard.domain.de.key
<FilesMatch "\.(cgi|shtml|phtml|php)$">
SSLOptions +StdEnvVars
</FilesMatch>
<Directory /usr/lib/cgi-bin>
SSLOptions +StdEnvVars
</Directory>
SSLProxyEngine on
SSLProxyProtocol +all
SSLProxyCipherSuite ALL
SSLProxyCheckPeerName off
SSLProxyCheckPeerCN off
SSLProxyCheckPeerExpire off
ProxyPass / http://172.18.0.81/ retry=1 acquire=3000 timeout=600 Keepalive=On
ProxyPassReverse / http://172.18.0.81/
ProxyPassReverseCookiePath / /
</VirtualHost>
</IfModule>
The output of your Nextcloud log in Admin > Logging:
{"reqId":"lc4NWq16dY7oZy1EGglk","level":3,"time":"2019-09-05T12:56:53+00:00","remoteAddr":"172.18.0.21","user":"--","app":"PHP","method":"GET","url":"\/cron.php","message":"Undefined index: all at \/var\/www\/nextcloud\/apps\/files_versions\/lib\/Storage.php#779","userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko\/20100101 Firefox\/70.0","version":"15.0.10.0"}
{"reqId":"lc4NWq16dY7oZy1EGglk","level":3,"time":"2019-09-05T12:56:53+00:00","remoteAddr":"172.18.0.21","user":"--","app":"PHP","method":"GET","url":"\/cron.php","message":"Undefined index: by_file at \/var\/www\/nextcloud\/apps\/files_versions\/lib\/Storage.php#781","userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko\/20100101 Firefox\/70.0","version":"15.0.10.0"}
{"reqId":"lc4NWq16dY7oZy1EGglk","level":3,"time":"2019-09-05T12:56:53+00:00","remoteAddr":"172.18.0.21","user":"--","app":"PHP","method":"GET","url":"\/cron.php","message":"Invalid argument supplied for foreach() at \/var\/www\/nextcloud\/apps\/files_versions\/lib\/Storage.php#781","userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko\/20100101 Firefox\/70.0","version":"15.0.10.0"}
{"reqId":"lc4NWq16dY7oZy1EGglk","level":3,"time":"2019-09-05T12:56:53+00:00","remoteAddr":"172.18.0.21","user":"--","app":"PHP","method":"GET","url":"\/cron.php","message":"ksort() expects parameter 1 to be array, null given at \/var\/www\/nextcloud\/apps\/files_versions\/lib\/Storage.php#805","userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko\/20100101 Firefox\/70.0","version":"15.0.10.0"}
{"reqId":"lc4NWq16dY7oZy1EGglk","level":3,"time":"2019-09-05T12:56:53+00:00","remoteAddr":"172.18.0.21","user":"--","app":"PHP","method":"GET","url":"\/cron.php","message":"reset() expects parameter 1 to be array, null given at \/var\/www\/nextcloud\/apps\/files_versions\/lib\/Storage.php#806","userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko\/20100101 Firefox\/70.0","version":"15.0.10.0"}
The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):
<?php
$CONFIG = array (
'instanceid' => '-',
'passwordsalt' => '-',
'secret' => '-',
'trusted_domains' =>
array (
0 => '172.18.0.81',
),
'datadirectory' => '/srv/data/',
'dbtype' => 'mysql',
'version' => '15.0.10.0',
'dbname' => 'nextclouddb',
'dbhost' => 'localhost',
'dbport' => '',
'dbtableprefix' => 'oc_',
'dbuser' => 'nextcloud',
'dbpassword' => '-',
'installed' => true,
'proxy' => 'http://172.18.0.250:3128/',
'mysql.utf8mb4' => true,
'maintenance' => false,
'memcache.local' => '\\OC\\Memcache\\APCu',
'skeletondirectory' => '',
'overwritehost' => 'portal.domain.de',
'overwriteprotocol' => 'https',
'overwritewebroot' => '/',
'overwritecondaddr' => '^172.18.0.21$',
'overwrite.cli.url' => 'http://172.18.0.81',
'updater.release.channel' => 'production',
'loglevel' => 2,
);
The output of your Apache/nginx/system log in /var/log/____:
[Thu Sep 05 14:17:51.983927 2019] [proxy:error] [pid 7684:tid 139741710243584] [client :55684] AH00898: Error reading from remote server returned by /index.php/apps/files/
[Thu Sep 05 14:17:51.983938 2019] [proxy:error] [pid 7685:tid 139741676672768] [client :48897] AH00898: Error reading from remote server returned by /index.php/apps/files/
[Thu Sep 05 14:17:51.986519 2019] [proxy_http:error] [pid 7685:tid 139740787504896] (20014)Internal error (specific information not available): [client :29655] AH01102: error reading status line from remote server 172.18.0.81:80
[Thu Sep 05 14:17:51.986528 2019] [proxy_http:error] [pid 7685:tid 139741693458176] (20014)Internal error (specific information not available): [client :36214] AH01102: error reading status line from remote server 172.18.0.81:80