Slow uploads of many small files

ℹ️ Support
1

I’m getting slow (100-200kb/s) uploads when uploading a large number of files, yet the server does not seem to be under load! (Uploading the same files from the same source via SFTP gets 400-500kb/s.)

Nextcloud version : 18.0.7
Operating system and version: Debian 10
Apache or nginx version: nginx 1.14.2
PHP version (eg, 7.1): 7.3.19

The issue you are facing: Slow uploads

Is this the first time you’ve seen this error? (Y/N): N

Steps to replicate it:

  1. mount by webdav (client is Ubuntu 20:04 using nautilus/GIO)
  2. copy a bunch of files
  3. paste into the webdav view

There are no errors/warnings etc. in the Nextcloud log in Admin > Logging at the time of this upload.

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

$CONFIG = array (
    'instanceid' => 'xxxxxxxxxxxx',
    'passwordsalt' => 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
    'secret' => 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
    'trusted_domains' =>
    array (
      0 => 'xxxxxxxxxxxxxxxxxxxx.org',
    ),
    'datadirectory' => '/mnt/archive/nextcloud',
    'dbtype' => 'mysql',
    'version' => '18.0.7.1',
    'overwrite.cli.url' => 'https://xxxxxxxxxxxxxxxxxxxx.org',
    'dbname' => 'nextcloud',
    'dbhost' => 'localhost',
    'dbport' => '',
    'dbtableprefix' => 'oc_',
    'dbuser' => 'nextcloud',
    'dbpassword' => 'xxxxxxxxxxxxxxxx',
    'installed' => true,
    'mail_smtpmode' => 'sendmail',
    'mail_sendmailmode' => 'smtp',
    'mail_from_address' => 'xxxxxxxxx',
    'mail_domain' => 'xxxxxxxxxxxxxxx.org',
    'maintenance' => false,
    'theme' => '',
    'loglevel' => 2,
    'updater.release.channel' => 'stable',
    'mysql.utf8mb4' => true,
  );

Screenshot showing top snapshot htat shows the system 56% idle, 0 wait, and with 0.5GB RAM free.

image
image1000×633 349 KB

I have the same issue on another server, which is running NC 19, nginx, and Redis - uploads around 150-200kb/s, whereas SFTP uploads go arount 500-600kb/s.

I get upload speeds over 1Mb/s (on both servers) for a single large file.

(Nb. I disabled the brute force app before doing these tests)

Interested in any tips!

2

Read https://docs.nextcloud.com/server/19/admin_manual/installation/server_tuning.html

You can also find errors and configuration hints in the nextcloud-gui-admin-settings overview.

https://xxxxxxxxxxxxxxxxxxxx.org/settings/admin/overview

3

Last Time i tried The Webdav connection i could never make it nearly as fast as my network Connection was.

Trying to reduce latency/Roundtrips helped a bit for me (e.g. http2/tls1.3 and also how fast the server answers ), especially on small files. But the overall speed was quite underwhelming.

If you really insist on using Webdav i would recommend you to use wireshark to listen to the traffic (you need to add the SSL Key to wireshark and disable Forward secrecy), And determine if the actual upload is slow, or (at least what i determined in my case as the slowest part) the amount of requests inbetween the file uploads.

If you know what actually causes the slowness its easier to optimize that part

_,

4

@devnull thanks, I’ve done all that already yet still 200kbps upload while the server reports it’s mostly idle, not waiting on IO etc.

@rtznprmpftl

I guess you’re referring to just using the sync client instead?