Very Slow upload speed

xattab · April 18, 2019, 12:56pm

Hi!
I have problem only with upload speed. Download speed fine.
I use nextcloud 15
OS Debian 9
RAM 128
CPU Intel xeon silver 4114
PHP 7.0
NGINX
Mysql (mariadb)

Hear my config.php for cache

‘memcache.local’ => ‘\OC\Memcache\APCu’,
‘memcache.distributed’ => ‘\OC\Memcache\Redis’,
‘memcache.locking’ => ‘\OC\Memcache\Redis’,
‘redis’ =>
array (
‘host’ => ‘/var/run/redis/redis.sock’,
‘port’ => 0,
‘dbindex’ => 0,
‘password’ => ‘secret’,
‘timeout’ => 1.5,
),

How i can fix this ?

Schmu · April 18, 2019, 1:06pm

Hi,

What exactly is slow, what speed would be fine? Where do you upload to, internal HDD/ SD-card or external server/ storage?
What kind of storage is it you are uploading to, what speeds does it support (like 5400rpm external HDD via USB2.0)? Is any kind of encryption enabled for data storage (NC server-side encryption for example)?

Could you post your web server config as well? It usually contains some parameters which can be tweaked to optimize web server speed.

xattab · April 18, 2019, 1:24pm

Hi Schmu!
I upload from web to self-hosted nextcloud (server use 4 x 1.6 TB SSD in Raid10.)
I use encryption on RAID10 (cryptsetup). Dont use nextcloud encryption

nginx config

upstream php-handler {
#server 127.0.0.1:9000;
server unix:/run/php/php7.0-fpm.sock;
}

server {
listen 80;
server_name nextcloud.tech;
return 301 https://$server_name$request_uri;
access_log off;
}

server {
listen 443 ssl http2;
add_header Strict-Transport-Security “max-age=31536000; includeSubDomains” always;
server_name nextcloud.tech;

ssl_certificate /etc/nginx/certs/tech.pem;
ssl_certificate_key /etc/nginx/certs/tech.key;

access_log /var/log/nginx/nextcloud.tech.access.log json;
error_log /var/log/nginx/nextcloud.tech.error.log;


add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
add_header X-Robots-Tag none;
add_header X-Download-Options noopen;
add_header X-Permitted-Cross-Domain-Policies none;

root /vhost/nextcloud/;

location = /robots.txt {
    allow all;
    log_not_found off;
    access_log off;
}

# The following 2 rules are only needed for the user_webfinger app.
# Uncomment it if you're planning to use this app.
rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json
last;

location = /.well-known/carddav {
  return 301 $scheme://$host/remote.php/dav;
}
location = /.well-known/caldav {
  return 301 $scheme://$host/remote.php/dav;
}

# set max upload size
client_max_body_size 4096M;
fastcgi_buffers 64 4K;

# Enable gzip but do not remove ETag headers
gzip on;
gzip_vary on;
gzip_comp_level 4;
gzip_min_length 256;
gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;


location / {
    rewrite ^ /index.php$uri;
}

location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {
    deny all;
}
location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) {
    deny all;
}

location ~ ^/(?:index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|ocs-provider/.+)\.php(?:$|/) {
    fastcgi_split_path_info ^(.+\.php)(/.*)$;
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_param PATH_INFO $fastcgi_path_info;
    fastcgi_param HTTPS on;
    #Avoid sending the security headers twice
    fastcgi_param modHeadersAvailable true;
    fastcgi_param front_controller_active true;
    fastcgi_pass php-handler;
    fastcgi_intercept_errors on;
    fastcgi_request_buffering off;
}

location ~ ^/(?:updater|ocs-provider)(?:$|/) {
    try_files $uri/ =404;
    index index.php;
}

# Adding the cache control header for js and css files
# Make sure it is BELOW the PHP block
location ~ \.(?:css|js|woff|svg|gif)$ {
    try_files $uri /index.php$uri$is_args$args;
    add_header Cache-Control "public, max-age=15778463";
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
    add_header Referrer-Policy "same-origin" always;
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";
    add_header X-Robots-Tag none;
    add_header X-Download-Options noopen;
    add_header X-Permitted-Cross-Domain-Policies none;
    # Optional: Don't log access to assets
    access_log off;
}

location ~ \.(?:png|html|ttf|ico|jpg|jpeg)$ {
    try_files $uri /index.php$uri$is_args$args;
    # Optional: Don't log access to other assets
    access_log off;
}

}

Schmu · April 18, 2019, 1:38pm

Information to the speeds are missing unfortunately

Just to rule out one thing, can you try to disable gzip in nginx for once and test again? Change

gzip on;

to

gzip off;

xattab · April 18, 2019, 3:01pm

I change
gzip on;
to
gzip on;
But nothing change.
I uploaded file 164MB for 6 min

Schmu · April 18, 2019, 3:38pm

Come on, some more technical details would be very helpful.
What network connections do you have? Do you upload the file in a local area network with 100MBit/s in both directions or do you upload files from a network with low upload speeds?

164MB in 6min are about 3.6 MBit/s. This could be the upload rate of Cable internet or slower DSL connections. So this could be the max speed. Therefor I asked about the speed you expect in my first post, which wasn’t answered.

xattab · April 18, 2019, 3:50pm

Here log from speed test

https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -

Retrieving speedtest.net configuration…
Testing from Servers.com B.V. (23.111.235.4)…
Retrieving speedtest.net server list…
Selecting best server based on ping…
Hosted by Pocos bv (Amsterdam) [1.39 km]: 6.335 ms
Testing download speed…
Download: 3645.88 Mbit/s
Testing upload speed…
Upload: 669.72 Mbit/s

xattab · April 18, 2019, 3:54pm

It s test from server

I can upload throw scp to server with more hire speed then 3.6 MBit/s

100% 140MB 8.7MB/s 00:16

xattab · April 22, 2019, 11:29am

@Schmu Do you have some ideas ?

Schmu · April 23, 2019, 7:48am

Hi @xattab

Sorry for the late answer, I have been on Easter vacation.
I just saw that you wrote, you have 128 RAM. You probably mean 128 GB, right?

You could try the following:

check this part of the guide and try the advices and settings to minimize timeout issues:
https://docs.nextcloud.com/server/15/admin_manual/configuration_files/big_file_upload_configuration.html#system-configuration
check if Redis is really working for Nextcloud. You should see messages in the journal/ system logs like redis-server[759]: 759:M 23 Apr 2019 09:34:50.029 * 10 changes in 300 seconds. Saving...
upload with different browsers (especially with IE for once) to see if you are affected by some strange issue like: https://github.com/nextcloud/server/issues/4438
watch the output of “top”/ “htop” during upload and check for CPU spikes (one core at 100% during upload?) and if DB, PHP-fpm or web server have a high load
set NC logging to Debug and see if there are any log messages during upload which indicate an issue somewhere
have a look at the system log and journal for the same

These are my ideas so far. We need to find out where exactly the bottleneck is.

xattab · April 24, 2019, 8:46am

xattab · April 24, 2019, 8:50am

@Schmu Hi!
I tried upload from EDGE and nothing different (slow speed)
Here my new nginx conf

upstream php-handler {
#server 127.0.0.1:9000;
server unix:/run/php/php7.0-fpm.sock;
}

server {
listen 80;
server_name nextcloud.tech;
return 301 https://$server_name$request_uri;
access_log off;
}

server {
listen 443 ssl http2;
add_header Strict-Transport-Security “max-age=31536000; includeSubDomains” always;
server_name nextcloud.tech;

ssl_certificate /etc/nginx/certs/araxio.tech.pem;
ssl_certificate_key /etc/nginx/certs/araxio.tech.key;

access_log /var/log/nginx/nextcloud.tech.access.log json;
error_log /var/log/nginx/nextcloud.tech.error.log;


add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
add_header X-Robots-Tag none;
add_header X-Download-Options noopen;
add_header X-Permitted-Cross-Domain-Policies none;

root /vhost/nextcloud/;

location = /robots.txt {
    allow all;
    log_not_found off;
    access_log off;
}

# The following 2 rules are only needed for the user_webfinger app.
# Uncomment it if you're planning to use this app.
rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json
last;

location = /.well-known/carddav {
  return 301 $scheme://$host/remote.php/dav;
}
location = /.well-known/caldav {
  return 301 $scheme://$host/remote.php/dav;
}

# set max upload size
client_max_body_size 16G;
fastcgi_buffers 64 4K;
client_body_temp_path /vhost/client_temp 1 2;
fastcgi_request_buffering off;

# Enable gzip but do not remove ETag headers
gzip on;
gzip_vary on;
gzip_comp_level 4;
gzip_min_length 256;
gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;


location / {
    rewrite ^ /index.php$uri;
}

location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {
    deny all;
}
location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) {
    deny all;
}

location ~ ^/(?:index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|ocs-provider/.+)\.php(?:$|/) {
    fastcgi_split_path_info ^(.+\.php)(/.*)$;
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_param PATH_INFO $fastcgi_path_info;
    fastcgi_param HTTPS on;
    #Avoid sending the security headers twice
    fastcgi_param modHeadersAvailable true;
    fastcgi_param front_controller_active true;
    fastcgi_pass php-handler;
    fastcgi_intercept_errors on;
    fastcgi_request_buffering off;
}

location ~ ^/(?:updater|ocs-provider)(?:$|/) {
    try_files $uri/ =404;
    index index.php;
}

# Adding the cache control header for js and css files
# Make sure it is BELOW the PHP block
location ~ \.(?:css|js|woff|svg|gif)$ {
    try_files $uri /index.php$uri$is_args$args;
    add_header Cache-Control "public, max-age=15778463";
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
    add_header Referrer-Policy "same-origin" always;
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";
    add_header X-Robots-Tag none;
    add_header X-Download-Options noopen;
    add_header X-Permitted-Cross-Domain-Policies none;
    # Optional: Don't log access to assets
    access_log off;
}

location ~ \.(?:png|html|ttf|ico|jpg|jpeg)$ {
    try_files $uri /index.php$uri$is_args$args;
    # Optional: Don't log access to other assets
    access_log off;
}

}

Schmu · April 24, 2019, 9:20am

Do you run an anti-virus tool which scans the files during upload?
Or is there a proxy server which scans the files passing through?
Did you try to upload files from different computers from different locations (LAN/ Internet)?
You always upload files via web browser, right? If so, is the upload speed the same with the NC desktop client?

With this big server you seem to be from a bigger company. Maybe you want to buy support from Nextcloud GmbH and ask them to optimize your server for performance regarding uploads.

holst · August 25, 2019, 4:48pm

So how did this end? How do Nextcloud users debug why uploads are slow after trying these steps above? I’m seeing O(10 Mbps) upload speeds for upload of an expected 1000 Mbps limit. Over a local switch.

Max_Papush · January 2, 2020, 10:18am

Try to remove http2 from nginx config.
Helped for me.

fredericksun.barlow · August 6, 2020, 5:12pm

I am new to nginx. Can you tell me how you remove http2 from nginx config? I have been having issues with my nextcloud upload speed too. I can’t get it to go over 25mb/s upload. I can transfer the same file to my windows server or ftp and I get 115mb/s+ for both upload and download. I have been using apache2 on ubuntu but I decided to give nginx a try.

lyallp · January 4, 2021, 10:21pm

I am having the same problem, slow upload speed, from my laptop running windows 10, fully up to date, via WiFi to my shared hosting NextCloud (cpanel/whm). NextCloud client 3.0.3 to NextCloud version 20.0.3.

Usually, uploads are fast, at the moment, it’s crawling along at around 1kB/s, yes, 1024 bytes per second. It eventually times out and fails. Only one file is at issue, my keepass database. I am unable to sync an update to the keepass database to the server and hence, on to other computers and my phone.
So, did anyone resolve their slow upload speed?

sharkbyte · February 22, 2021, 9:06pm

@lyallp did you ever figure out a resolution for this topic?

Paradox551 · February 22, 2021, 9:52pm

Remove http2 from your nginx config.