File Uploads From Web Interface Limited to 10 Mbit/s

Support intro

Sorry to hear you’re facing problems :slightly_frowning_face:

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can :heart:

Nextcloud version : 15.0.2
Operating system and version : FreeBSD 12.0-RELEASE-p2
Apache or nginx version : nginx-1.14.2_6,2
PHP version : php72-7.2.14_1

The issue you are facing:
File uploads from the web do not transfer faster than ~10 Mbit/s per second despite plenty of bandwidth and system resources being available. Available bandwidth is around ~100 MBit/s (tested with iperf3, file transfer tested with sftp as well). System load average is around 0.2, over 1 GB of free memory, hard disks are largely idle (confirmed with gstat and systat -iostat 1).

Is this the first time you’ve seen this error?:
Fresh installation of Nextcloud 15 on a FreeBSD 12 server.

Steps to replicate it:

  1. Install Nextcloud 15 from FreeBSD ports tree, with all caching options enabled
  2. Configure server (& components) following instructions in documentation
  3. Upload a file through the web interface

The output of your Nextcloud log in Admin > Logging:
Only failed login attempts (leftover from previous installation as I reinstalled & configured again after finding poor web upload performance).

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'apps_paths' =>
  array (
    0 =>
    array (
      'path' => '/usr/local/www/nextcloud/apps',
      'url' => '/apps',
      'writable' => true,
    ),
    1 =>
    array (
      'path' => '/usr/local/www/nextcloud/apps-pkg',
      'url' => '/apps-pkg',
      'writable' => false,
    ),
  ),
  'logfile' => '/var/log/nextcloud/nextcloud.log',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'memcache.distributed' => '\OC\Memcache\Redis',
  'redis' =>
  array ( 
          'host' => '/tmp/redis.sock',
          'port'     => 0,
          'dbindex'  => 0,
  ),
  'instanceid' => '',
  'passwordsalt' => '',
  'secret' => '',
  'trusted_domains' =>
  array (
    0 => '$domain',
  ),
  'datadirectory' => '/NEXTCLOUD/data',
  'dbtype' => 'pgsql',
  'version' => '15.0.2.0',
  'overwrite.cli.url' => 'https://$domain',
  'dbname' => 'nextcloud_db2',
  'dbhost' => '$dbhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'nextcloud',
  'dbpassword' => '$dbpass',
  'installed' => true,
);

nginx config:

#user  nobody;
worker_processes  auto;

# This default error log path is compiled-in to make sure configuration parsing
# errors are logged somewhere, especially during unattended boot when stderr
# isn't normally logged anywhere. This path will be touched on every nginx
# start regardless of error log location configured here. See
# https://trac.nginx.org/nginx/ticket/147 for more info. 
#
#error_log  /var/log/nginx/error.log;
#

#pid        logs/nginx.pid;


events {
    worker_connections  1024;
    multi_accept on;
}


http {
	include       mime.types;
	default_type  application/octet-stream;

	#log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
	#                  '$status $body_bytes_sent "$http_referer" '
	#                  '"$http_user_agent" "$http_x_forwarded_for"';

	#access_log  logs/access.log  main;

	sendfile        on;
	#tcp_nopush     on;

	#keepalive_timeout  0;
	keepalive_timeout  65;

	#gzip  on;

	upstream php-handler {
		server unix:/tmp/php-fpm.sock;
	}

	server {
		listen 80;
		server_name $domain;

		# acme-client
		include acme.conf;

		return 301 https://$server_name$request_uri;
	}


	server {
		listen 443 ssl http2;
		server_name $domain;

		# Use Mozilla's guidelines for SSL/TLS settings
		# https://mozilla.github.io/server-side-tls/ssl-config-generator/
		# NOTE: some settings below might be redundant
		ssl_certificate 
		ssl_certificate_key 
		ssl_dhparam 

			# Add headers to serve security related headers
			# Before enabling Strict-Transport-Security headers please read into this
			# topic first.
			# add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";
			#
			# WARNING: Only add the preload option once you read about
			# the consequences in https://hstspreload.org/. This option
			# will add the domain to a hardcoded list that is shipped
			# in all major browsers and getting removed from this list
			# could take several months.
			add_header X-Content-Type-Options nosniff;
		add_header X-XSS-Protection "1; mode=block";
		add_header X-Robots-Tag none;
		add_header X-Download-Options noopen;
		add_header X-Permitted-Cross-Domain-Policies none;
		add_header Referrer-Policy no-referrer;

		# Remove X-Powered-By, which is an information leak
		fastcgi_hide_header X-Powered-By;

		# Path to the root of your installation
		root /usr/local/www/nextcloud;

		location = /robots.txt {
			allow all;
			log_not_found off;
			access_log off;
		}

		# The following 2 rules are only needed for the user_webfinger app.
		# Uncomment it if you're planning to use this app.
		#rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
		#rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json last;

		# The following rule is only needed for the Social app.
		# Uncomment it if you're planning to use this app.
		# rewrite ^/.well-known/webfinger /public.php?service=webfinger last;

		location = /.well-known/carddav {
			return 301 $scheme://$host/remote.php/dav;
		}
		location = /.well-known/caldav {
			return 301 $scheme://$host/remote.php/dav;
		}

		# set max upload size
		client_max_body_size 5120M;
		# following setting speeds things up by about 6x
		#client_body_buffer_size 4096k;
		fastcgi_buffers 128 4K;

		# Enable gzip but do not remove ETag headers
		gzip on;
		gzip_vary on;
		gzip_comp_level 4;
		gzip_min_length 256;
		gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
		gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;

		location / {
			rewrite ^ /index.php$request_uri;
		}

		location ~ ^\/(?:build|tests|config|lib|3rdparty|templates|data)\/ {
			deny all;
		}
		location ~ ^\/(?:\.|autotest|occ|issue|indie|db_|console) {
			deny all;
		}

		location ~ ^\/(?:index|remote|public|cron|core\/ajax\/update|status|ocs\/v[12]|updater\/.+|ocs-provider\/.+)\.php(?:$|\/) {
			fastcgi_split_path_info ^(.+?\.php)(\/.*|)$;
			include fastcgi_params;
			fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
			fastcgi_param PATH_INFO $fastcgi_path_info;
			fastcgi_param HTTPS on;
			#Avoid sending the security headers twice
			fastcgi_param modHeadersAvailable true;
			fastcgi_param front_controller_active true;
			fastcgi_pass php-handler;
			fastcgi_intercept_errors on;
			fastcgi_request_buffering off;
		}

		location ~ ^\/(?:updater|ocs-provider)(?:$|\/) {
			try_files $uri/ =404;
			index index.php;
		}

		# Adding the cache control header for js and css files
		# Make sure it is BELOW the PHP block
		location ~ \.(?:css|js|woff2?|svg|gif)$ {
			try_files $uri /index.php$request_uri;
			add_header Cache-Control "public, max-age=15778463";
			# Add headers to serve security related headers (It is intended to
					# have those duplicated to the ones above)
			# Before enabling Strict-Transport-Security headers please read into
			# this topic first.
			# add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";
			#
			# WARNING: Only add the preload option once you read about
			# the consequences in https://hstspreload.org/. This option
			# will add the domain to a hardcoded list that is shipped
			# in all major browsers and getting removed from this list
			# could take several months.
			add_header X-Content-Type-Options nosniff;
			add_header X-XSS-Protection "1; mode=block";
			add_header X-Robots-Tag none;
			add_header X-Download-Options noopen;
			add_header X-Permitted-Cross-Domain-Policies none;
			add_header Referrer-Policy no-referrer;

			# Optional: Don't log access to assets
			access_log off;
		}

		location ~ \.(?:png|html|ttf|ico|jpg|jpeg)$ {
			try_files $uri /index.php$request_uri;
			# Optional: Don't log access to other assets
			access_log off;
		}
	}
}

The output of your Apache/nginx/system log:

$ip - - [05/Feb/2019:16:19:32 +0000] "PUT /remote.php/dav/uploads/manas/web-file-upload-0084b32029626d70c8c797141505d19b-1549383530242/157286400 HTTP/2.0" 201 0 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36"
$ip - - [05/Feb/2019:16:19:35 +0000] "PUT /remote.php/dav/uploads/manas/web-file-upload-0084b32029626d70c8c797141505d19b-1549383530242/167772160 HTTP/2.0" 201 0 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36"
$ip - - [05/Feb/2019:16:19:37 +0000] "PUT /remote.php/dav/uploads/manas/web-file-upload-0084b32029626d70c8c797141505d19b-1549383530242/178257920 HTTP/2.0" 201 0 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36"
$ip - - [05/Feb/2019:16:19:40 +0000] "PUT /remote.php/dav/uploads/manas/web-file-upload-0084b32029626d70c8c797141505d19b-1549383530242/188743680 HTTP/2.0" 201 0 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36"
$ip - - [05/Feb/2019:16:19:42 +0000] "PUT /remote.php/dav/uploads/manas/web-file-upload-0084b32029626d70c8c797141505d19b-1549383530242/199229440 HTTP/2.0" 201 0 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36"
$ip - - [05/Feb/2019:16:19:45 +0000] "PUT /remote.php/dav/uploads/manas/web-file-upload-0084b32029626d70c8c797141505d19b-1549383530242/209715200 HTTP/2.0" 201 0 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36"
$ip - - [05/Feb/2019:16:19:47 +0000] "PUT /remote.php/dav/uploads/manas/web-file-upload-0084b32029626d70c8c797141505d19b-1549383530242/220200960 HTTP/2.0" 201 0 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36"
$ip - - [05/Feb/2019:16:19:50 +0000] "PUT /remote.php/dav/uploads/manas/web-file-upload-0084b32029626d70c8c797141505d19b-1549383530242/230686720 HTTP/2.0" 201 0 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36"
$ip - - [05/Feb/2019:16:19:52 +0000] "PUT /remote.php/dav/uploads/manas/web-file-upload-0084b32029626d70c8c797141505d19b-1549383530242/241172480 HTTP/2.0" 201 0 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36"
$ip - - [05/Feb/2019:16:19:55 +0000] "PUT /remote.php/dav/uploads/manas/web-file-upload-0084b32029626d70c8c797141505d19b-1549383530242/251658240 HTTP/2.0" 201 0 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36"

The only configuration setting that I found which made a difference was in nginx: client_body_buffer_size 4096k – this improves file uploads in the web interface by nearly 6x. However this seems like a bad setting (I am no nginx expert) according to their documentation:

https://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size

which suggests 8k or 16k

Looking for some insight or feedback.
Hoping to figure out how to run a Nextcloud server on FreeBSD with PostgreSQL & nginx with good performance.

Thanks

2 Likes

Hi,

Sorry for the late response. I needed some time for testing. And I found something very interesting.
I always had a very good upload speed on my server and while I always wanted to check the site load speed improvements with gzip on, I just enabled it for testing purposes and checked the upload speeds in addition.
With gzip on it drastically dropped!

gzip on:
Upload speed (server to client) varied from ~10 MBit/s (1.2 MB/s) to ~18 MBit/s (2.2 MB/s)

gzip off:
Upload speed (server to client) arround 625 Mbit/s (78 MB/s) - with no drops of the transfer speed

I obviously tested in 1 GBit environment :slight_smile:
As you can see the difference is really huge and I ran the tests twice, again with disabling and enabling gzip again.
The parameter client_body_buffer_size is default on my setup (didn’t define it anywhere) and I use standard nginx site configuration like suggested by Nextcloud docu (except the disable gzip).
Looks like we rather need to find a way to improve nginx performance for downloads with gzip enabled.

Regarding the site load speed I was quite impressed and would also like to enable gzip in the future. Site loads dropped from the range of 2.2s - 2.9s to 1.3s to 1.7s. So that’s massive, but I need a good upload speed and a drop in upload speed of 97% is a very bad trade-off.

As it caused some confusion I added the info above, that I wrote from the server’s perspective, meaning that “upload” is the file transfer from the server to the client, when the user hits the download button in the web GUI for example. So the client’s download speed drops with gzip enabled. That’s what I tried to say :stuck_out_tongue:

1 Like

Hi!

Thanks for the response.

Based on my limited knowledge, I’m quite surprised as I always thought gzip was used for compressing traffic from server to user and not the other way around.

Thanks for your insight, I will test speeds with gzip disabled.

Manas


Edit: Read your response again, a little confused now. I understand if gzip enabled would cause the server to send data slower to a client but would it also cause slowness when the client was sending data to the server? e.g. in this specific case when I am uploading a multi-gigabyte file over the Nextcloud web interface over the internet with HTTPS enabled.

Sorry it caused confusion, I was describing from the server side. So when I wrote “upload” I meant, that the server sends to data to the client. So from the client’s perspective, it is actually a download from the server.

I haven’t done any tests regarding uploading files from the client to the server and I wouldn’t expect any effects due to gzip there.