Desktop client delta updates to files

[Hollerauer]

I reply on this post just to keep it alive and visible for the community, because I think the importance of this post/feature is underrated at the moment.

For me delta upload is a key feature of a sync client. No matter how your installation looks like, the overall goal when syncing is to keep traffic low and sync times short.

It’s clear to me, that it would be a big change in nextcloud/owncloud client’s methodology of syncing but from my point of view the invest is worth it.

From my perspective this is a key feature, which even big commercial-cloud-service-provider do not supply and therefor could be another stand-alone functionality of nextcloud. Therefore this feature should not be on a “nice-to-have” or “long-term-goal” list. You can have the most secure system in the world but in most cases customers will decide based on their user experience which system they choose.

Last I want to mention that if you combine delta sync with file-version-control (what would definitely make sense to me), this is exactly what rdiff does.

For those of you who are interested in publications on this topic this presentation of CERN might be interesting.

---

I have to modify my existing post, because I found some interesting article & discussion providing information I was not aware of.

For those who do not want to read everything I quote the part which is important to me:

@"Jos Poortvliet 2015-02-10 18:29

The question is: for what files is this delta sync relevant… Because it does not work very well on compressed or encrypted files (changing one bit changes everything) and is useless on small files (faster to just sync them). So, is it useful?

• it isn’t for most music (compressed)
• it isn’t for most pictures (compressed)
• it isn’t for most movies (compressed)
• it isn’t for most documents (too small and/or compressed)

Ask yourself. What files are 1. bigger than 20 mb and 2. not compressed or encrypted?

There are two use cases that I can think off:

• biology researchers. They have massive text files with ATCGACTTGC in them. Not sure if they change them, though…
• VM images. Most of those could benefit from incremental sync.

Now if you have multiple multi-GB VM images you’d like to be synced accross devices, you would benefit. And if you’re a biology researcher who changes his big text files with genetic data. But that is, what, 3% of our user base?

I am not saying it can’t work better than sending the full file. You can start to special-case stuff. You can decompress files and look at the content, see what changed. If you can predict what the compression algorithm does, you can know how to chunk up the file and what to send over. But you need to now how to deal with every file type under the sun – or at least those often used. You can’t use the same, generic algorithm on all – it won’t work.

For dropbox and friends, this isn’t a huge problem:

• They can special case all kinds of data because they just throw 100 people at the problem.
• They have massive amounts of storage and a business reason to to use it super efficiently. So they chunk everything up, every piece of a file that matches does not have to be uploaded. With 250 million users, you’ve got plenty of overlapping data and thus you can save a lot of syncing. Doesn’t work for a private ownCloud server, unfortunately…
• They have one client, no need for supporting generic webDAV or other clients.
• They control the storage protocol – ownCloud usally doesn’t! ownCloud supports external storage – and as you can’t expect your FTP server to deal with chunks, making this delta syncing work with external storage is both very hard and probably not half as beneficial as you’d hope).

Other open source file sync and share projects suffer from the same issues (except the last one, ownCloud’s external storage is, afaik, unique). Some have delta-sync, but I would think it’s more of a checkbox on a feature list than that it really helps with a lot of files…

So, we could work on making this work. It can probably save bandwidth on some files, paying a price in much increased server load and client load. We could work on this feature.

But we have many, MANY users who benefit from features like favoriting, selectively syncing files, sharing files with a right click in a file manager, showing the sharing status in the file manager and the many other features the client did introduce already or that the client team is working on.

So, help us decide: what feature to axe because selective sync, benefitting a very small number of users, is more important?

Personally, I would prefer to be able to roll back changes to files with a right click in the file manager over selective sync. And I would prefer to be able to drag a file on the ownCloud systray icon and get a share link in my clipboard ready do share over selective sync. And those features are probably not even on the roadmap yet, and I would push back selective sync for them.

Would you?

After reading this article and especially the quoted statement my personal opinion about the delta-sync topic is:

• The Idea to let customers pay for implementation of specific file-transfer algorithm sounds reasonable

• If not existing already the core (client & server) should have the functionality to implement file specific algorithm