Trouble with WebDAV performance :-(

ℹ️ Support
1

NextCloud 13.0.6, Debian 9 server, Apache with mod PHP, client connected to server via 1 GB network, low RTTs.

scp achieves about 22 MB/sec transfer rate.

Copying files via WebDAV is extremely slow. Uploading a tiny file (2 bytes) using CURL takes 5-6 seconds, so apparently there’s a really high latency for each operation.

I mounted the NextCloud DAV filesystem using davfs2 on a Linux system. Navigating the mounted filesystem is already sluggish on the shell (cd and dir). Copying files using “cp -a” or rsync (run locally with the mounted DAV filesystem as its target) is basically unusable.

The bruteforce_attempt table is empty, so bruteforce prevention does not seem to be in effect. To be sure, I also already disabled it in the config file.

Recursively copying a directory with 10 subdirectories in total and 4300 files, the total size being less the 80 MB, takes much more than 12 hours. That’s just ridiculous…

In comparison, a davfs2 mounted Apache DAV filesystem nearly feels like a local filesystem…

I fear I have some significant misconfiguration somewhere, but I wasn’t able to find any hint in the docs or using Google.

There are similar reports in the forum (e.g. WebDAV Performance Issues) but in those it either was caused by bruteforce prevention, or the issues were not solved or not answered. (Or I missed the appropriate answers…)

2

You will never reach the performance of SCP because Nextcloud fiddles everything through php which interacts with a database. However, it should be much faster than what you are experiencing.

Since it does a lot with the database, you should check your database configuration that the cache sizes are large enough, use tools such as mysqltuner or tuning-primer.sh. If you move a lot of files, you should also use redis as a file-locking cache because if not, everything is handled by the database which uses much more resources for this.

2 Likes
3

Yes, of course I’m aware that it won’t reach scp performance, I just mentioned that as a baseline to demonstrate the network is not the culprit.

I now configured Redis and apc/apcu and also added some indexes to my LDAP server against which I authenticate, but this merely improved response time from 5.x seconds to 4.x seconds, and at first glance overall performance is still as bad.

I’ve not yet optimized MariaDB, but the server is not really doing anything else at the moment, and I guess I should not expect improvements of several orders of a magnitude just be tweeking some MariaDB buffer sizes from their default values?

4

You mentioned ACPC caching, but have you enabled opcache/installed the opchache php package? My installation running on FreeBSD went from being unusable to like a spinning rust filesystem with that enabled. There should be a warning for the admin if that is missing

1 Like
5

Then check if your system has a lot of i/o-wait during upload (you can see this in top or even better in iotop). If yes, then an optimization will very likely help.

1 Like
6

Ok, there was significant IO and IO waits and I tweaked and tested a lot. I also used the mysqltuner script to get ideas about what I still could improve.

Apparently, MySQL updates to the oc_filecache table where the main cause for this.

The only setting which really made a difference was to set

innodb_flush_log_at_trx_commit=2

to 0 or 2. Even with this setting, it’s still quite slow, but some orders of a magnitude faster and the IO load caused by MySQL got reduced significantly.

(Addendum: It looks like davfs2 or maybe the specific combination of davfs2 and NextCloud WebDAV has to be blamed for the terrible performance. Using Windows Explorer or Konqueror “webdavs://” URLs, access to and navigation through WebDAV is actually quite fast, comparable to the NextCloud web interface. I didn’t perform any transfer / throughput tests, though.) Not sure what davfs2 is doing there. :-/ But maybe it shouldn’t be recommended as a client in the documentation then…)

Apparently, NextCloud executes many micro-transactions if WebDAV operations are performed, which really hit the DB server. The impact of this probably also has to do with the DB’s backend storage’s characteristics. In my case, the storage is backed by a synced DRBD (replicated/distributed block device) device, which probably is close to the worst case scenario if it comes to fsync latency/performance, as the block device has to wait for confirmation from the mirror before reporting “success”.

As I understand, the drawback of the DB config adjustment I did is the loss of guaranteed ACID integrity in case of a kernel / system crash, which also does not sound like a good solution. For the oc_filecache table alone, this probably would not matter too much, but the innodb_flush_log_at_trx_commit setting affects the whole MariaDB server with all databases running on it, so it really appears to be a big compromise to me…

Are there any other options to optimize oc_filecache performance? I wonder if I’m the only one who experiences those performance issues - or maybe noone actually uses WebDAV access to the files managed by NextCloud? Though this also sounds very unlikely to me, so I would expect configuration hints or howtos already in the official documentation (“What to do to actually get usable WebDAV performance from NextCloud”) or at least in many third-party blogs or the like, but this doesn’t seem to be the case…

7

i think everyone is accessing the files via webdav. but when you use the desktop sync client you don’t care about the speed since the files are synced in the background.

i didn’t use many small files to transfer only one large iso image. but I got 500Mb/s between two ec2 on aws.

8

I use this MariaDB instance only for Nextcloud and I keep always current database backups. For upload via NC client there is a potential improvement by uploading smaller files together and make a larger insert instead of many. Native webdav probably treats files individually, this would require more changes. If you have ideas, feel free to share them with the developers.

9

What advantage does Sabre hold over using the native Apache module? Being able to choose which one, via a config flag, might be useful.
Being able to use for example, rsync directly (a la OpenMediaVault) would be equally useful.

10

It’s written in php? How would you connect the apache module to the database? You have a file index in the filecache-table and all the sharing and permission stuff is in the database as well. A working alternative is to use a different sync solution which syncs files outside the Nextcloud data folder. Use this folder as a external storage within Nextcloud, then you can use the apache webdav-module or even different solutions, s-ftp, syncthing, …

11

Thanks for the clarity and ideas. :slight_smile:
As well as “standard” encrypted WebDAV backups, I also lftp files into a data subdirectory, then run an occ scan. I accept that the imported files won’t be encrypted(?) but in this case they’re just publicly available JPEGs.
The clarity/idea of using external storage to a local directory, seems like a better option.

12

Yes, they are then not encrypted (not sure if they are encrypted later automatically), not sure how it handles shared folder etc. The developers recommend not to do that, so I am not sure if there are perhaps more downsides.

13

My use, in this instance is purely for server backup, so sharing and other features are of no relevance.
Cheers.

14

:roll_eyes:
The Local folder (sic) doesn’t appear to get encrypted.

15

With encryption, you must put files through nextcloud and you can’t place them manually.

16

If it takes about 24 hours to sync less than 100 MB, you’ll also notice this if you’re using the client - your folders will be out-of-sync all the time, I guess…

Maybe it helps a bit if the native client merges operations on several small files, as it was stated in another post in this thread, but I still have the impression that I’m noticing much more massive performance issues than many other users.

I now assume it’s because if the system my nextcloud instance is running on, it’s in a VM on a dual-machine cluster which uses DRBD to keep both cluster machines in sync - probably an fsync is extremely expensive in this setup, but I’ve no influence on this configuration and have to live with it.

So I’ll try running with

innodb_flush_log_at_trx_commit=2

and hope for the best…

17

Maybe you should look into using ftp, sftp, scp or rsync for this purpose, which - without knowing your additional constraints, of course - all sound more sensible for the mentioned use case to me.

18

Somewhere I read that share information and permissions or something like that get lost in the process. (Which does not seem to be important for ejsolutions use case anyway, though - which however also leads to the question why she or he is using NextCloud for this use case at all.)

19

I’m a software developer, but I do not have the slightest idea of how NextCloud works internally - so I’m not sure what ideas I could share with the devs… I cannot assess whether it would be viable or even technically possibly to properly merge multiple WebDAV operations in a single DB transaction, which could have to alleviate the immediate performance problems somewhat, I guess…

20

Hi I did not read every post but it sounds like I have encountered a similar problem. I wrote a blog about it. Did not find a sollution though.
https://akseliratamo.fi/2018/09/21/nextcloudpi-samba-vs-webdav-speed/