Sync extremely slow with large number of files (on Desktop)

jvhoffbauer · December 28, 2020, 6:53pm

I have a user with 33GB 21135 directories and 167198 files of data. Sync on the laptop with the desktop client (3.0.3, macOS) is extremely slow and crashes multiple times without any error. It takes especially long for the ‘Checking for changes in remote’ step to finish.

Are there any recommendations on how to tackle this issue or reasons as to why it is slow?

From the logs I retrieve with nextcloud --logdir ... command, it seems like nextcloud is performing a single http request for every directory. Is this true?

Thanks in advance!

tflidd · December 28, 2020, 11:48pm

So the logs are not catching why it crashes? What is the last thing before the crash?

A long time ago, I tried how much I could speed up the upload of a large number of files:

github.com/owncloud/core

Upload performance OC 8.2.1 (50 mysql queries per file)

opened 12:22AM - 06 Dec 15 UTC

closed 04:54PM - 06 Apr 17 UTC

tflidd

enhancement discussion performance

Upload of many small files is very slow. I tested on a raspbery pi 2 which is kn…own to have a bad i/o-performance. During uploads, mysql-operation create an important io-wait on the system. It was worse with journaling of ext4, so I disabled it (it's a test system). Uploading a folder with 1180 files (18 MB) via the oc-client takes about 20-30 minutes (pictures, a lot of text files, take any php code of a project), but the number of sql-queries is extremely high (>50 queries per uploaded file!): | command | number | | --- | --- | | select | 49,681 | | update | 11,523 | | insert select | 2,487 | | set option | 1,526 | | delete | 511 | | insert | 423 | | begin | 17 | | commit | 17 | For upload, a smaller number of queries can be obtained when files are uploaded in an external folder via ftp (takes a few seconds) then this folder is included into owncloud, the scanning process takes a few minutes (it's hard to tell, 5 minutes?) and the number of queries (some queries are due to handling the web-interface): | command | number | | --- | --- | | select | 11,054 | | update | 3,040 | | insert select | 3,004 | | set option | 425 | | delete | 138 | I remember you worked on the deletion process for OC 8.0, this scales much better: 3000 files take 3500 update and 1400 select queries. Perhaps there is a similar improvement possible for upload operations ;-) System: RPi2, Debian Jessie Webserver: nginx/php-fpm Database: mariadb Owncloud 8.2.1 Apps enabled: - activity: 2.1.3 - calendar: 0.8.1 - contacts: 0.5.0.0 - files: 1.2.0 - files_external: 0.3.0 - files_pdfviewer: 0.7 - files_sharing: 0.7.0 - files_texteditor: 2.0 - files_trashbin: 0.7.0 - files_versions: 1.1.0 - files_videoviewer: 0.1.3 - firstrunwizard: 1.1 - galleryplus: 14.3.0 - provisioning_api: 0.3.0 - templateeditor: 0.1 I haven't tested with a pure webdav-client yet.

I managed to get up to 1000 files/minute. Not sure what has been improved since, impact of SSDs and faster systems. But for you perhaps a starting point.

Yes and every file, and each time the filecache-table is updated.

There is also a bug report: https://github.com/nextcloud/desktop/issues/691
(I think there is a part for the server config that can help to improve the situation, the crash and freezing of the client seems to be a client issue, and of course there could be a more intelligent upload).

jvhoffbauer · December 30, 2020, 9:10pm

I see. I ended up using rsync and then scanning the files, which was way faster.

There is an issue discussing the sync speed problem here: https://github.com/nextcloud/desktop/issues/691

mooleshacat · January 10, 2022, 6:26pm

@jvhoffbauer 's solution is correct workaround.

Proper fix is to add to bottom of php.ini for the FPM & CLI & CGI to add in:

memory_limit = 10G
post_max_size = 25G
upload_max_filesize= 25G

Then increase chunksize in /var/www/nextcloud/config/config.php:

‘chunkSize’ => ‘5120MB’,

Then you need to restart php-fpm:

systemctl restart php7.4-fpm

Simply replace the 7.4 with your version number, can be found with php -v or php -info

So far desktop sync sped right up. Still testing to see if Android app finally can finish its uploads.

I also added the preview image cacher plugin, ran the initial run, and added the cronjob. Next I will offload my MySQL, Redis, and Clamscan to a second raspi. That should offload lots of work from the pi running NextCloud

I wish there was a plugin/way to add a secondary pi to offload the primary pi’s resources for NextCloud. So you can add more processing power from another pi. It would be cool to have a way to set up master/slave next cloud servers, so you can have multiple servers serving nextcloud.

JosefNoll · February 25, 2022, 2:50pm

Thanks for the excellent answer - worked perfectly, through 10G memory limit is rather heavy (if you only have 8 GB memory).
But if the memory_limit = 6Gmade everything work smoothly. Thanks