Hello there!
A few months ago, maybe when upgrading to a new major version in the February 2023 timeframe, my Nextcloud instance ( cloud.hyperling.com ) started responding extremely slowly in web browsers without any change in my reverse proxy setup. it takes 30-90 seconds to start returning assets but once it starts it flows rather quickly. Logging in is possible but any change of the URI initiates another 30-90 second wait. If I get into Documents, Notes, Talk, etc then all on-screen functionality works fine. I can navigate between folders, message other users, etc.
The oddest part of this is that webdav, caldav, and cardav are all still fast and flow just fine through the reverse proxy. Other sites that the reverse proxy works for are hyperling.com and graphs.hyperling.com and they are fast like the *dav URIs, there is not a long delay like with the web UI of NC. Also odd is that if accessed directly through the LAN via IP:PORT over http (no s) it is plenty fast, no delay at all. So I feel like it definitely has something to do with the reverse proxy or the overwrite directives in the config file. I have all combinations of enabling and disabling them and the trusted proxies directive but no luck there either.
When navigating via a web browser I barely see any activity in the reverse proxy or nextcloud-app-1 logs until the asset return begins so I don’t think they’re looping between eachother (HTTP 502). Watching in Wireshark shows that a keep-alive is being sent and accepted during the pause. Using a web browser’s development tools and watching the Network tab shows a long initial wait and then once the first asset is returned all the rest start flowing in with normal response times. I had to increase the nginx timeouts otherwise an HTTP 504 would get returned.
I have been using Nextcloud for a few years, and last year migrated from a manual install on Debian to running the Docker version for ease of upgrades. After any maintenance I run this list of commands to make sure all is in the best possible order.
Upgrade:
docker compose down && docker compose pull && docker compose up -d
“Fixes”:
#!/bin/bash
# 2022-09-25 Hyperling
# Put fixes in a file so they do not need remembered.
docker exec -it nextcloud-app-1 apt update -y
docker exec -it nextcloud-app-1 apt install -y sudo libmagickcore-6.q16-6-extra
# 2022-10-30 More additions after moving to Nextcloud version 25.
docker exec -itu www-data nextcloud-app-1 ./occ db:add-missing-columns
docker exec -itu www-data nextcloud-app-1 ./occ db:add-missing-indices
docker exec -itu www-data nextcloud-app-1 ./occ db:add-missing-primary-keys
docker exec -itu www-data nextcloud-app-1 ./occ db:convert-filecache-bigint
docker exec -it nextcloud-app-1 chown -Rc www-data:www-data .
# 2023-02-12 Just for good measure.
docker exec -itu www-data nextcloud-app-1 ./occ app:update --all
# 2023-07-02
# This maybe used to exist, but make sure that Files app is correct.
docker exec -itu www-data nextcloud-app-1 ./occ files:scan --all
# This one takes a while.
docker exec -itu www-data nextcloud-app-1 ./occ files:scan-app-data
# Extras? Have used the commands in the past and may help in the future.
docker exec -itu www-data nextcloud-app-1 ./occ maintenance:theme:update
docker exec -itu www-data nextcloud-app-1 ./occ maintenance:repair
# May also be useful but do not have much experience with them.
docker exec -itu www-data nextcloud-app-1 ./occ versions:cleanup
docker exec -itu www-data nextcloud-app-1 ./occ files:cleanup
exit 0
Here is the docker compose file:
# Nextcloud configuration.
# This is a revised version of the original work here:
# https://hub.docker.com/_/nextcloud
version: '2'
services:
db:
image: mariadb:10.6
restart: always
command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
volumes:
- /opt/Docker/Volumes/Nextcloud/mariadb:/var/lib/mysql
environment:
- MYSQL_ROOT_PASSWORD=SENSITIVE
- MYSQL_DATABASE=nextcloud
- MYSQL_USER=nc
- MYSQL_PASSWORD=SENSITIVE
app:
image: nextcloud
restart: always
ports:
- 8080:80
links:
- db
volumes:
- /opt/Docker/Volumes/Nextcloud/nextcloud:/var/www/html
environment:
- MYSQL_DATABASE=nextcloud
- MYSQL_USER=nc
- MYSQL_PASSWORD=SENSITIVE
- MYSQL_HOST=db
- PHP_UPLOAD_LIMIT=5G
Here is my config/config.php:
<?php
$CONFIG = array (
'htaccess.RewriteBase' => '/',
'memcache.local' => '\\OC\\Memcache\\APCu',
'apps_paths' =>
array (
0 =>
array (
'path' => '/var/www/html/apps',
'url' => '/apps',
'writable' => false,
),
1 =>
array (
'path' => '/var/www/html/custom_apps',
'url' => '/custom_apps',
'writable' => true,
),
),
'instanceid' => 'SECRET?',
'passwordsalt' => 'SECRET?',
'secret' => 'SECRET?',
'trusted_domains' =>
array (
0 => 'prime:8080',
1 => 'cloud.hyperling.com',
2 => 'prime.hyperling.com',
3 => 'prime.hyperling.com:8080',
4 => 'cloud.hyperling.com:8080',
5 => '192.168.1.81:8080',
),
'datadirectory' => '/var/www/html/data',
'dbtype' => 'mysql',
'version' => '27.0.0.8',
'dbname' => 'nextcloud',
'dbhost' => 'db',
'dbport' => '',
'dbtableprefix' => 'oc_',
'mysql.utf8mb4' => true,
'dbuser' => 'nc',
'dbpassword' => 'SENSITIVE',
'installed' => true,
'maintenance' => false,
'theme' => 'hyperling',
'mail_from_address' => 'nextcloud',
'mail_smtpmode' => 'smtp',
'mail_sendmailmode' => 'smtp',
'mail_domain' => 'hyperling.com',
'mail_smtpauth' => 1,
'mail_smtpauthtype' => 'LOGIN',
'mail_smtphost' => 'mail.hyperling.com',
'mail_smtpport' => '587',
'mail_smtpname' => 'SENSITIVE',
'mail_smtppassword' => 'SENSITIVE',
'mail_smtpsecure' => 'tls',
'default_phone_region' => 'US',
'overwritehost' => 'cloud.hyperling.com',
'overwriteprotocol' => 'https',
'overwrite.cli.url' => 'https://cloud.hyperling.com',
'trusted_proxies' =>
array (
0 => 'reverse-proxy',
1 => 'reverse-proxy.hyperling.com',
2 => '192.168.1.80',
3 => '192.168.1.81',
4 => 'prime',
5 => 'prime.hyperling.com',
),
);
Last weekend (July 9th 2023) I migrated from a FreeBSD server with nginx built with ports to doing it through docker on Debian. I was hoping the issue was related to the aging VM and port build process but the migration did not change the issue at all. I went with a fresh and minimized config file rather than migrating all the configuration from the FreeBSD system. I have tried turning off any on any of the settings which start with proxy_ but it did not cause any improvements.
Here is the nginx.conf:
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log notice;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] $host "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
## Proxy Settings ##
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header HTTP_AUTHORIZATION $http_authorization;
proxy_hide_header X-Powered-By;
proxy_intercept_errors on;
proxy_http_version 1.1;
# Proxy Buffer settings - See http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
# Timeouts, give up to 5 minutes for slow apps.
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;
## LetsEncrypt Certbot Setup ##
# Allow nginx to fulfill LetsEncrypt Certbot challenges.
server {
listen 80;
location ^~ /.well-known/acme-challenge/ {
default_type "text/plain";
root /etc/nginx/letsencrypt;
}
}
## Upstream Configuration ##
include /etc/nginx/hosts/*;
## Reverse Proxied Website Configurations ##
include /etc/nginx/conf.d/*;
}
Here is the cloud.hyperling.com include file:
server {
listen 80;
server_name cloud.hyperling.com;
if ($host = cloud.hyperling.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
}
server {
listen 443 ssl; #proxy_protocol;
server_name cloud.hyperling.com;
ssl_certificate /etc/nginx/certs/cloud.hyperling.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/nginx/certs/cloud.hyperling.com/privkey.pem; # managed by Certbot
# https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/reverse_proxy_configuration.html#nginx
rewrite ^/\.well-known/carddav https://$server_name/remote.php/dav/ redirect;
rewrite ^/\.well-known/caldav https://$server_name/remote.php/dav/ redirect;
# Send traffic to upstream server
location / {
#proxy_set_header X-Forwarded-Proto https;
#proxy_set_header Host $http_host;
# 2023-07-09 Trying to fix the dang timeout issues.
#proxy_set_header X-Real-IP $remote_addr;
#proxy_http_version 1.1;
#proxy_set_header Connection "";
# Allow waiting up to 5 minutes for Nextcloud to respond. Grrr.
#proxy_read_timeout 600;
expires epoch;
add_header Pragma public;
add_header Cache-Control "private, no-store";
add_header Strict-Transport-Security "max-age=15552000; includeSubDomains; preload";
# 2022-11-26 Try to fix upload errors (HTTP Error: Request Entity Too Large).
client_max_body_size 0;
client_body_buffer_size 100M;
proxy_pass http://hyperling-cloud;
}
}
The hardware is doing fine, server is not anywhere near swapping, CPUs have plenty of availability. I tried watching the nextcloud-db-1 container’s log and it was not getting anything output while doing web tests. I looked through the Nextcloud and MariaDB/MySQL tuning guides but did not find anything obviously wrong with the DB situation. I did not end up making any changes inside the db-1 container.
I did notice while posting this that mariadb is pinned to 10.5, did not realize that was the case. It looks like it’s supported until 2025. The URL in the compose file has an example which now shows 10.6. I may give upgrading a try and report back. Would be great if it was the issue but I’m not expecting that it would be. I’ve upgraded to MariaDB 10.6 and there have been no improvements.
Feel free to ask anything, including basic/beginner questions. I have spent entire weekends on this and feel like I have exhausted all my sources, like it must be something so simple that I’m overlooking it, or something is wrong with a part of the system that I have not explored before.
Edit: Change MariaDB version.
