Nextcloud Sync 2.0 brings 10x faster syncing

Originally published at: https://nextcloud.com/blog/nextcloud-sync-2-0-brings-10x-faster-syncing/

2021 is a year of massive improvements to the Nextcloud desktop client. The introduction of Instant Sync with Nextcloud Hub 21 brought file changes instantly to the desktop while decreasing server load from the clients by nearly 90%. There have also been numerous improvements to End-to-end Encryption and the Virtual Files, all hooking deep in the sync engine of the client.

One more big change is coming: a massive improvement to the syncing speed of small files, impacting the overall syncing speed significantly and further reducing server load. All this has added up to such a big change we’ve decided to up the version number of our sync engine.

Sync Engine 2.0

Small files are currently a bit of a bottleneck for the sync client, as each file is uploaded individually. Thus, 1000 1kb files will take 1000 connections, giving a lot of overhead compared to a single 1mb file. Version 2.0 of Nextcloud Sync will do away with this: the File Packing feature will upload multiple small files in a single request!

10x faster upload

This means faster upload of small files – hundreds, even thousands of times faster depending on the file size and network limitations, as internal testing on a tech preview shows! Of course, in a more typical scenario of mixed file sizes, the benefits are more modest, but it is not unlikely that most full sync runs will be between 2x to 10x faster.

10x lower server load

The sync client uploads multiple files in parallel to get the most speed out of the network. If many clients sync simultaneously, this can cause some serious load and by eliminating a large number of connections, the new sync engine has the potential to dramatically decrease the load on servers.

Syncing at a new level: 10x to 100x

With Instant Sync thanks to the High Performance Back-end for Files introduced earlier this year, and the File Packing coming to V2 of the Sync Engine, both user experience and server load will dramatically improve. This is augmented by dozens other improvements made to the syncing process over the last year.

Internal testing shows that in most situations, the sync speed will now simply be limited by the bandwidth of the network connection of the user. For the server, idle load will have decreased by 10x thanks to the HPB, while during syncing the number of connections is also reduced by about a factor 10!

Users will benefit from instant availability of their files, even if they need to have large numbers of small files made available locally.

Nextcloud Sync v2 will be made available as part of a desktop client release later this year, alongside the release of Nextcloud 23.

10 Likes

I don’t get one thing :
In order to get this improvement we need to have a HPB ?

3 Likes

If you don’t have HPB yet (wether or not it is required for this new client (setting?)), give it a go.
The results are pretty good.

Knowing what the HPB for files does and reading the announcement I’m pretty sure the HPB won’t be required. This will be an upgrade to the sync engine itself, where the HPB for files is still optional and needs to be installed and configured by an admin before it works. Of course for the best experience the HPB is recommended.

3 Likes

So, Sync 2.0 will not be supported from the webui itself?
If someone drops 1,000 1k files into a filedrop shared url they will still be on the old method.
Just checking. :heart:

Most likley yes, because file packing has to be done by the client in order to upload the files with a single request.

Actually, from a purely technical point of view, there is no reason they couldn’t do it in the webui too. Similarly to the desktop client, they would collect the list of files in JavaScript, then pack and upload it to the server endpoint, all in JS. This would need to be a totally separate re-implementation of the packing logic & upload protocol however and couldn’t share code with the desktop client, so I doubt they will do this.

1 Like

Thank you devs! This was one of my biggest problems with nextcloud for the past years.

Finally this issue is getting attention and solved! Thank you once again.

That is also why I created my own sync app: Melroy van den Berg / Nextcloud File Drop · GitLab

2 Likes

What I’d really like to see, is a NON-SYNCHRONIZING virtual drive (maybe fuse for *nix, and whatever you do to make a drive-letter mapped mount on MS) using this new approach. This would be an alternative to using WebDAV, which is needed in order to completely avoid synchronizing.

2 Likes

What is the HPB backend?

HPB stands for High Performance Backend:

I see. Remember reading about it. It’s still not for webui or normal webdav connections, is it?

The HPB for Files sends notifications - both for the clients and the webUI. This makes sure the web UI or devices know that there has been a change and trigger a sync run to get the files or new notification (like an incoming call) in real-time instead of just checking every 30 seconds or using long polling (the PHP alternative for keeping a connection open, quite heavy on server resources).

This doesn’t speed up the sync per-se, but makes you wait less.

In a way, for the networking techies - the HPB decreases the latency, while Sync 2.0 increases the bandwidth :wink: And these two are, as usual, mostly independent.

Well, that’s not our initial plan, but in time we will get to that too, of course. In the end, it took a while to get chunking in the web uploader but we now have that, too (only for logged-in users btw) . So, then I think we can do this, too, it’s effectively the opposite of chunking :wink: but it might take a while, so don’t hold your breath. But you shouldn’t do that anyway :see_no_evil:

1 Like

Given the speed limit is on the server side, I wonder how your app would be any faster? If it uses pure webdav and not our chunked upload, it could be less reliable than uploading with the client or web interface would be. I don’t want to diss you, it’s great you build an app for this - I’m just trying to figure out if we do something wrong or if it’s just a use case thing and not a performance thing - your app does seem useful if you just want to upload a file to a specific folder but not use the desktop client…