Nextcloud version (eg, 12.0.2): 17.0.1
Operating system and version (eg, Ubuntu 17.04): Debian GNU/Linux 9.11 (stretch)
Apache or nginx version (eg, Apache 2.4.25): nginx/1.17.6
PHP version (eg, 7.1):
PHP 7.3.12 with Zend OPcache v7.3.12
The issue you are facing: Very slow speed when using reverse proxy (download speed with reverse proxy is great). When not using reverse proxy upload and download speeds are great.
Is this the first time you’ve seen this error? (Y/N): Y
Steps to replicate it:
System resources are not a problem. Currently the platform is still in the testing phase and has no users (only me), the CPU (10 cores / 20 threads) remains very close to 0% even when a lot of data is being transferred.
Both the Nextcloud VM and the Nginx reverse proxy have resources available to spare (Nextcloud machine 4 vCPU and 8GB RAM; nginx reverse proxy 4vCPU and 6GB RAM).
I have observed that if I connect directly to Nextcloud without going through the reverse proxy I get upload and download speeds of about 900 Mbps, when I go through the reverse proxy the upload speed is 5Mbps per file (I explain this later), and The download is 70-120 Mbps.
I have discovered that if I upload a single file it goes up to ~ 5MMbps, but if I upload two files at the same time, they both go up at once to ~ 5Mbps each (therefore total upload speed = ~ 5Mbps), this happens like this Up to about 10 simultaneous files.
The reverse proxy is a nginx-reverse-proxy-manager built on docker in a VM with 4vCPU and 6GB RAM that never uses more than 2% CPU and never more than 2GB RAM.
The output of your Nextcloud log in Admin > Logging (I paste it, but I think all are old errors that are now resolved):
https://pastebin.com/j0Zs0BBj
The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):
<?php
$CONFIG = array (
'passwordsalt' => '***DELETED***',
'secret' => '***DELETED***',
'trusted_domains' =>
array (
0 => '***DELETED***(DOMAIN FROM WHICH THE USERS ACCESS) ',
),
'datadirectory' => '/nxc-data',
'dbtype' => 'mysql',
'version' => '17.0.1.1',
'overwrite.cli.url' => 'http://localhost',
'dbname' => '***DELETED***',
'dbhost' => 'localhost',
'dbport' => '',
'dbtableprefix' => 'oc_',
'mysql.utf8mb4' => true,
'dbuser' => '***DELETED***',
'dbpassword' => '***DELETED***',
'activity_expire_days' => 14,
'auth.bruteforce.protection.enabled' => true,
'theme' => 'nextcloud-breeze-dark',
'blacklisted_files' =>
array (
0 => '.htaccess',
1 => 'Thumbs.db',
2 => 'thumbs.db',
),
'cron_log' => true,
'enable_previews' => true,
'enabledPreviewProviders' =>
array (
0 => 'OC\\Preview\\PNG',
1 => 'OC\\Preview\\JPEG',
2 => 'OC\\Preview\\GIF',
3 => 'OC\\Preview\\BMP',
4 => 'OC\\Preview\\XBitmap',
5 => 'OC\\Preview\\Movie',
6 => 'OC\\Preview\\PDF',
7 => 'OC\\Preview\\MP3',
8 => 'OC\\Preview\\TXT',
9 => 'OC\\Preview\\MarkDown',
),
'filesystem_check_changes' => 0,
'filelocking.enabled' => 'true',
'htaccess.RewriteBase' => '/',
'integrity.check.disabled' => false,
'knowledgebaseenabled' => false,
'log_rotate_size' => 104857600,
'logfile' => '/nxc-data/nextcloud.log',
'logtimezone' => 'Europe/Berlin',
'memcache.local' => '\\OC\\Memcache\\APCu',
'memcache.locking' => '\\OC\\Memcache\\Redis',
'preview_max_x' => 1024,
'preview_max_y' => 768,
'preview_max_scale_factor' => 1,
'redis' =>
array (
'host' => '/var/run/redis/redis.sock',
'port' => 0,
'timeout' => 0.0,
),
'quota_include_external_storage' => false,
'share_folder' => '/Shares',
'skeletondirectory' => '',
'trashbin_retention_obligation' => 'auto, 7',
'instanceid' => '***DELETED***',
'installed' => true,
'mail_from_address' => '***DELETED***',
'mail_smtpmode' => 'smtp',
'mail_sendmailmode' => 'smtp',
'mail_domain' => '***DELETED***',
'mail_smtpauthtype' => 'LOGIN',
'mail_smtpauth' => 1,
'mail_smtphost' => '***DELETED***',
'mail_smtpport' => '***DELETED***',
'mail_smtpname' => '***DELETED***',
'mail_smtppassword' => '***DELETED***',
'mail_smtpsecure' => 'ssl',
'maintenance' => false,
);
The output of your Apache/nginx/system log in /var/log/____:
**error.log is empty**
access.log:
192.168.1.202 - - [18/Dec/2019:12:28:58 +0100] "GET /apps/logreader/poll?lastReqId=chWXCMUnr22IbCDJThKC HTTP/1.1" 200 2 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
192.168.1.202 - - [18/Dec/2019:12:29:18 +0100] "GET /apps/logreader/poll?lastReqId=chWXCMUnr22IbCDJThKC HTTP/1.1" 200 2 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
192.168.1.202 - - [18/Dec/2019:12:29:24 +0100] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 200 74 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
192.168.1.202 - - [18/Dec/2019:12:29:38 +0100] "GET /apps/logreader/poll?lastReqId=chWXCMUnr22IbCDJThKC HTTP/1.1" 200 2 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
192.168.1.202 - - [18/Dec/2019:12:29:54 +0100] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 200 74 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
192.168.1.202 - - [18/Dec/2019:12:29:58 +0100] "GET /apps/logreader/poll?lastReqId=chWXCMUnr22IbCDJThKC HTTP/1.1" 200 2 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
192.168.1.202 - - [18/Dec/2019:12:30:19 +0100] "GET /apps/logreader/poll?lastReqId=chWXCMUnr22IbCDJThKC HTTP/1.1" 200 2 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
192.168.1.202 - - [18/Dec/2019:12:30:24 +0100] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 200 74 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
192.168.1.202 - - [18/Dec/2019:12:30:39 +0100] "GET /apps/logreader/poll?lastReqId=chWXCMUnr22IbCDJThKC HTTP/1.1" 200 2 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
192.168.1.202 - - [18/Dec/2019:12:30:54 +0100] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 200 74 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
192.168.1.202 - - [18/Dec/2019:12:30:59 +0100] "GET /apps/logreader/poll?lastReqId=chWXCMUnr22IbCDJThKC HTTP/1.1" 200 2 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
192.168.1.202 - - [18/Dec/2019:12:31:19 +0100] "GET /apps/logreader/poll?lastReqId=chWXCMUnr22IbCDJThKC HTTP/1.1" 200 2 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
192.168.1.202 - - [18/Dec/2019:12:31:24 +0100] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 200 74 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
192.168.1.202 - - [18/Dec/2019:12:31:39 +0100] "GET /apps/logreader/poll?lastReqId=chWXCMUnr22IbCDJThKC HTTP/1.1" 200 2 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
192.168.1.202 - - [18/Dec/2019:12:31:54 +0100] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 200 74 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
192.168.1.202 - - [18/Dec/2019:12:31:59 +0100] "GET /apps/logreader/poll?lastReqId=chWXCMUnr22IbCDJThKC HTTP/1.1" 200 2 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
192.168.1.202 - - [18/Dec/2019:12:32:19 +0100] "GET /apps/logreader/poll?lastReqId=chWXCMUnr22IbCDJThKC HTTP/1.1" 200 2 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
192.168.1.202 - - [18/Dec/2019:12:32:24 +0100] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 200 74 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
192.168.1.202 - - [18/Dec/2019:12:32:39 +0100] "GET /apps/logreader/poll?lastReqId=chWXCMUnr22IbCDJThKC HTTP/1.1" 200 2 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
192.168.1.202 - - [18/Dec/2019:12:32:54 +0100] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 200 74 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
192.168.1.202 - - [18/Dec/2019:12:33:00 +0100] "GET /apps/logreader/poll?lastReqId=chWXCMUnr22IbCDJThKC HTTP/1.1" 200 2 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
192.168.1.202 - - [18/Dec/2019:12:33:20 +0100] "GET /apps/logreader/poll?lastReqId=chWXCMUnr22IbCDJThKC HTTP/1.1" 200 2 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
192.168.1.202 - - [18/Dec/2019:12:33:24 +0100] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 200 74 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
The output of the nginx.conf of the nginx reverse proxy machine:
[**THE DOCKER CONTAINER WHERE IS NGINX REVESE PROXY MANAGER**:/etc/nginx]# cat nginx.conf
# run nginx in foreground
daemon off;
user root;
# Set number of worker processes automatically based on number of CPU cores.
worker_processes auto;
# Enables the use of JIT for regular expressions to speed-up their processing.
pcre_jit on;
error_log /data/logs/error.log warn;
# Includes files with directives to load dynamic modules.
include /etc/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
sendfile on;
server_tokens off;
tcp_nopush on;
tcp_nodelay on;
client_body_temp_path /tmp/nginx/body 1 2;
keepalive_timeout 65;
ssl_prefer_server_ciphers on;
gzip on;
proxy_ignore_client_abort off;
client_max_body_size 2000m;
server_names_hash_bucket_size 64;
proxy_http_version 1.1;
proxy_set_header X-Forwarded-Scheme $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Accept-Encoding "";
proxy_cache off;
proxy_cache_path /var/lib/nginx/cache/public levels=1:2 keys_zone=public-cache:30m max_size=192m;
proxy_cache_path /var/lib/nginx/cache/private levels=1:2 keys_zone=private-cache:5m max_size=1024m;
# MISS
# BYPASS
# EXPIRED - expired, request was passed to backend
# UPDATING - expired, stale response was used due to proxy/fastcgi_cache_use_stale updating
# STALE - expired, stale response was used due to proxy/fastcgi_cache_use_stale
# HIT
# - (dash) - request never reached to upstream module. Most likely it was processed at Nginx-level only (e.g. forbidden, redirects, etc) (Ref: Mail Thread
log_format proxy '[$time_local] $upstream_cache_status $upstream_status $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Length $body_bytes_sent] [Gzip $gzip_ratio] [Sent-to $server] "$http_user_agent" "$http_referer"';
log_format standard '[$time_local] $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Length $body_bytes_sent] [Gzip $gzip_ratio] "$http_user_agent" "$http_referer"';
access_log /data/logs/default.log proxy;
# Dynamically generated resolvers file
include /etc/nginx/conf.d/include/resolvers.conf;
# Default upstream scheme
map $host $forward_scheme {
default http;
}
# Real IP Determination
# Docker subnet:
set_real_ip_from 172.0.0.0/8;
# NPM generated CDN ip ranges:
include conf.d/include/ip_ranges.conf;
# always put the following 2 lines after ip subnets:
real_ip_header X-Forwarded-For;
real_ip_recursive on;
# Files generated by NPM
include /etc/nginx/conf.d/*.conf;
include /data/nginx/default_host/*.conf;
include /data/nginx/proxy_host/*.conf;
include /data/nginx/redirection_host/*.conf;
include /data/nginx/dead_host/*.conf;
include /data/nginx/temp/*.conf;
# Custom
include /data/nginx/custom/http[.]conf;
}
stream {
# Files generated by NPM
include /data/nginx/stream/*.conf;
}
# Custom
include /data/nginx/custom/root[.]conf;
Any help will be greatly appreciated, I have not been sleeping for 3 days fighting with nginx and I am running out of ideas to try … 