Nextcloud Sync 2.0 brings 10x faster syncing

[system]

Originally published at: https://nextcloud.com/blog/nextcloud-sync-2-0-brings-10x-faster-syncing/

2021 is a year of massive improvements to the Nextcloud desktop client. The introduction of Instant Sync with Nextcloud Hub 21 brought file changes instantly to the desktop while decreasing server load from the clients by nearly 90%. There have also been numerous improvements to End-to-end Encryption and the Virtual Files, all hooking deep in the sync engine of the client.

One more big change is coming: a massive improvement to the syncing speed of small files, impacting the overall syncing speed significantly and further reducing server load. All this has added up to such a big change we’ve decided to up the version number of our sync engine.

Sync Engine 2.0

Small files are currently a bit of a bottleneck for the sync client, as each file is uploaded individually. Thus, 1000 1kb files will take 1000 connections, giving a lot of overhead compared to a single 1mb file. Version 2.0 of Nextcloud Sync will do away with this: the File Packing feature will upload multiple small files in a single request!

10x faster upload

This means faster upload of small files – hundreds, even thousands of times faster depending on the file size and network limitations, as internal testing on a tech preview shows! Of course, in a more typical scenario of mixed file sizes, the benefits are more modest, but it is not unlikely that most full sync runs will be between 2x to 10x faster.

10x lower server load

The sync client uploads multiple files in parallel to get the most speed out of the network. If many clients sync simultaneously, this can cause some serious load and by eliminating a large number of connections, the new sync engine has the potential to dramatically decrease the load on servers.

Syncing at a new level: 10x to 100x

With Instant Sync thanks to the High Performance Back-end for Files introduced earlier this year, and the File Packing coming to V2 of the Sync Engine, both user experience and server load will dramatically improve. This is augmented by dozens other improvements made to the syncing process over the last year.

Internal testing shows that in most situations, the sync speed will now simply be limited by the bandwidth of the network connection of the user. For the server, idle load will have decreased by 10x thanks to the HPB, while during syncing the number of connections is also reduced by about a factor 10!

Users will benefit from instant availability of their files, even if they need to have large numbers of small files made available locally.

Nextcloud Sync v2 will be made available as part of a desktop client release later this year, alongside the release of Nextcloud 23.

[Nemskiller]

I don’t get one thing :
In order to get this improvement we need to have a HPB ?

[GieltjE]

If you don’t have HPB yet (wether or not it is required for this new client (setting?)), give it a go.
The results are pretty good.

[mwalbeck]

Knowing what the HPB for files does and reading the announcement I’m pretty sure the HPB won’t be required. This will be an upgrade to the sync engine itself, where the HPB for files is still optional and needs to be installed and configured by an admin before it works. Of course for the best experience the HPB is recommended.

[just]

@system

Nextcloud Sync v2 will be made available as part of a desktop client release later this year, alongside the release of Nextcloud 23.

So, Sync 2.0 will not be supported from the webui itself?
If someone drops 1,000 1k files into a filedrop shared url they will still be on the old method.
Just checking.

[bb77]

@just

will still be on the old method.
Just checking

Most likley yes, because file packing has to be done by the client in order to upload the files with a single request.

[Ka_Pa]

Actually, from a purely technical point of view, there is no reason they couldn’t do it in the webui too. Similarly to the desktop client, they would collect the list of files in JavaScript, then pack and upload it to the server endpoint, all in JS. This would need to be a totally separate re-implementation of the packing logic & upload protocol however and couldn’t share code with the desktop client, so I doubt they will do this.

[danger89]

Thank you devs! This was one of my biggest problems with nextcloud for the past years.

Finally this issue is getting attention and solved! Thank you once again.

That is also why I created my own sync app: Melroy van den Berg / Nextcloud File Drop · GitLab

[Larry_Boyd]

What I’d really like to see, is a NON-SYNCHRONIZING virtual drive (maybe fuse for *nix, and whatever you do to make a drive-letter mapped mount on MS) using this new approach. This would be an alternative to using WebDAV, which is needed in order to completely avoid synchronizing.

[Gatak]

What is the HPB backend?

[Schmu]

HPB stands for High Performance Backend:

Nextcloud faster than ever! Introducing Files High Performance Back-end

Originally published at: Nextcloud faster than ever! Introducing Files High Performance Back-end – Nextcloud At a virtual presentation streamed worldwide, the Nextcloud team introduced the availability of Nextcloud Hub 21. Nextcloud Hub 21 brings a wide range of improvements in file handling performance with Files High Performance Back-end and important new collaboration features in Text, Talk, Groupware and Files. In this blog, we go into a bit more detail of the performance improvements…

[Gatak]

I see. Remember reading about it. It’s still not for webui or normal webdav connections, is it?

[jospoortvliet]

@Gatak

I see. Remember reading about it. It’s still not for webui or normal webdav connections, is it?

The HPB for Files sends notifications - both for the clients and the webUI. This makes sure the web UI or devices know that there has been a change and trigger a sync run to get the files or new notification (like an incoming call) in real-time instead of just checking every 30 seconds or using long polling (the PHP alternative for keeping a connection open, quite heavy on server resources).

This doesn’t speed up the sync per-se, but makes you wait less.

In a way, for the networking techies - the HPB decreases the latency, while Sync 2.0 increases the bandwidth And these two are, as usual, mostly independent.

@just

So, Sync 2.0 will not be supported from the webui itself?
If someone drops 1,000 1k files into a filedrop shared url they will still be on the old method.
Just checking.

Well, that’s not our initial plan, but in time we will get to that too, of course. In the end, it took a while to get chunking in the web uploader but we now have that, too (only for logged-in users btw) . So, then I think we can do this, too, it’s effectively the opposite of chunking but it might take a while, so don’t hold your breath. But you shouldn’t do that anyway

[jospoortvliet]

Given the speed limit is on the server side, I wonder how your app would be any faster? If it uses pure webdav and not our chunked upload, it could be less reliable than uploading with the client or web interface would be. I don’t want to diss you, it’s great you build an app for this - I’m just trying to figure out if we do something wrong or if it’s just a use case thing and not a performance thing - your app does seem useful if you just want to upload a file to a specific folder but not use the desktop client…

[maddox]

NC Server v.22 and Client v.3.4.0 have been released. Neither was there a hint regarding the new Sync, nor do I notice a difference. Can you tell me what is the state? When will the new sync be released?

[tflidd]

It should be working with the newly released version NC 23. There were a few issues with the v3.4.0 client (Desktop client 3.4.0 destroys local time stamp and keeps uploading data to server) and it isn’t pushed for updates any more. I’d wait for a fix for this issue and then test again.

[just]

Did you set it up? HPB requires some steps to get working so you’ll need to follow the documentation to get it running.

[maddox]

So I found the time for additional tests. Each client uses client v.3.4.0, Server is NC Hub2

Yes, i set up HPB. According to the stats command it is working.

Having a bulk of 9000 small files (each < 10kb) I noticed that upload is done in chunks of 100 files each.

The next computer, which has to download the files receives them NOT in bulk. Each file is downloaded seperately.

[szaimen]

@maddox

The next computer, which has to download the files receives them NOT in bulk. Each file is downloaded seperately.

This is expected. See Bulk upload · Issue #3657 · nextcloud/desktop · GitHub

[danger89]

(sorry for the late reply)

I did notice better upload performance during the past year. Lot of changes are most likely made in that area in the meanwhile.

I’m also unsure why the web interface upload could be any slower. Since the difference now seems to be minimal, I don’t think it’s worth running performance tests anymore. Or… maybe we still should?

My Nextcloud server (v25.0.3) runs on a beefy machine. Currently running PHP 8.1 (fpm) with tweaked PHP configs (applied all production settings you can think of) and Nginx reverse proxy (also running with production config).

I think the biggest performance improvement was maybe configuring MySQL (MariaDB) with production configs. Which is also explained in your docs as well as updating the 20-mysqli.ini file at the PHP side.