Desktop client delta updates to files

[stratacast]

I checked the owncloud github on the sync client and only saw one thread that lead to a dead end. I was wondering if people are still interested in sending just the changes of a file and not the file in its entirety. I know a local business that wants to use Nextcloud for their internal use and clients but they work with big files all day long that take small changes throughout the day. It would be nice just to send maybe couple MB changes and not whole 100MB or whatever size files each time the client syncs with the server

[tflidd]

This features has been frequently asked for. Unfortunately, it wasn’t implemented yet. If you really need this feature for business application, try to contact the enterprise support. As enterprise customer you have certain influence on the future features and how they are prioritized. On bountysource there is already a considerable amount of money on this feature (>1000$), or if you have experience in programming you can directly help with the implementation.

There is a workaround that you use syncthing as a sync client and then use the sync-thing syncing folder as external storage in Nextcloud (to have the web-interface, sharing links, …)

[stratacast]

I was actually going to mention syncthing in my post haha. Good to know. Is there any influence on features even as a non-enterprise customer? I’m still not a seasoned programmer, but I’ve been learning languages like PHP so I could contribute to Nextcloud. I have a feeling delta syncing like what syncthing does might be a hard thing to implement but if I feel like I can accomplish it I might try to do the work in my spare time and hope smarter people than I can build on it

[Hollerauer]

I reply on this post just to keep it alive and visible for the community, because I think the importance of this post/feature is underrated at the moment.

For me delta upload is a key feature of a sync client. No matter how your installation looks like, the overall goal when syncing is to keep traffic low and sync times short.

It’s clear to me, that it would be a big change in nextcloud/owncloud client’s methodology of syncing but from my point of view the invest is worth it.

From my perspective this is a key feature, which even big commercial-cloud-service-provider do not supply and therefor could be another stand-alone functionality of nextcloud. Therefore this feature should not be on a “nice-to-have” or “long-term-goal” list. You can have the most secure system in the world but in most cases customers will decide based on their user experience which system they choose.

Last I want to mention that if you combine delta sync with file-version-control (what would definitely make sense to me), this is exactly what rdiff does.

For those of you who are interested in publications on this topic this presentation of CERN might be interesting.

---

I have to modify my existing post, because I found some interesting article & discussion providing information I was not aware of.

For those who do not want to read everything I quote the part which is important to me:

@"Jos Poortvliet 2015-02-10 18:29

The question is: for what files is this delta sync relevant… Because it does not work very well on compressed or encrypted files (changing one bit changes everything) and is useless on small files (faster to just sync them). So, is it useful?

• it isn’t for most music (compressed)
• it isn’t for most pictures (compressed)
• it isn’t for most movies (compressed)
• it isn’t for most documents (too small and/or compressed)

Ask yourself. What files are 1. bigger than 20 mb and 2. not compressed or encrypted?

There are two use cases that I can think off:

• biology researchers. They have massive text files with ATCGACTTGC in them. Not sure if they change them, though…
• VM images. Most of those could benefit from incremental sync.

Now if you have multiple multi-GB VM images you’d like to be synced accross devices, you would benefit. And if you’re a biology researcher who changes his big text files with genetic data. But that is, what, 3% of our user base?

I am not saying it can’t work better than sending the full file. You can start to special-case stuff. You can decompress files and look at the content, see what changed. If you can predict what the compression algorithm does, you can know how to chunk up the file and what to send over. But you need to now how to deal with every file type under the sun – or at least those often used. You can’t use the same, generic algorithm on all – it won’t work.

For dropbox and friends, this isn’t a huge problem:

• They can special case all kinds of data because they just throw 100 people at the problem.
• They have massive amounts of storage and a business reason to to use it super efficiently. So they chunk everything up, every piece of a file that matches does not have to be uploaded. With 250 million users, you’ve got plenty of overlapping data and thus you can save a lot of syncing. Doesn’t work for a private ownCloud server, unfortunately…
• They have one client, no need for supporting generic webDAV or other clients.
• They control the storage protocol – ownCloud usally doesn’t! ownCloud supports external storage – and as you can’t expect your FTP server to deal with chunks, making this delta syncing work with external storage is both very hard and probably not half as beneficial as you’d hope).

Other open source file sync and share projects suffer from the same issues (except the last one, ownCloud’s external storage is, afaik, unique). Some have delta-sync, but I would think it’s more of a checkbox on a feature list than that it really helps with a lot of files…

So, we could work on making this work. It can probably save bandwidth on some files, paying a price in much increased server load and client load. We could work on this feature.

But we have many, MANY users who benefit from features like favoriting, selectively syncing files, sharing files with a right click in a file manager, showing the sharing status in the file manager and the many other features the client did introduce already or that the client team is working on.

So, help us decide: what feature to axe because selective sync, benefitting a very small number of users, is more important?

Personally, I would prefer to be able to roll back changes to files with a right click in the file manager over selective sync. And I would prefer to be able to drag a file on the ownCloud systray icon and get a share link in my clipboard ready do share over selective sync. And those features are probably not even on the roadmap yet, and I would push back selective sync for them.

Would you?

After reading this article and especially the quoted statement my personal opinion about the delta-sync topic is:

• The Idea to let customers pay for implementation of specific file-transfer algorithm sounds reasonable

• If not existing already the core (client & server) should have the functionality to implement file specific algorithm

[Tim_Day]

Just wanted to add one potential user’s perspective here…

I work with a globally distributed group of collaborators who are basically professional data wranglers of content (text, images and video) for ebook-like software. We’re fairly intensive users of Dropbox but would be quite interested in some self-hosted replacement; Nextcloud got on my radar from a mention in a recent presentation at the Edinburgh techmeetup.

One use case which is absolutely crucial for us is:

• Someone has a ~1-to-10 GByte folder of .jpg and .mov files called, say, Dropbox/ebook_content_version_0.0.1/
• They copy that wholesale to Dropbox/ebook_content_version_0.0.2/ and maybe add or delete a few jpgs or simply rename a subdirectory containing a few files. Maybe only 1-10% of the bytes change.
• Dropbox is absolutely brilliant about syncing this to other collaborators so that Dropbox/ebook_content_version_0.0.2 appears for them with a minimal amount of bandwidth consumed; it’s obviously doing something clever with file checksums or hashes so that it knows most of ebook_content_version_0.0.2/ is “already there” in ebook_content_version_0.0.1/ (actually it seems to me Dropbox uses some sort of block-level hash as I’ve seen it sometimes transfer large files where just a few bytes have changed very quickly, but we don’t care so much about that).

From reading Nextcloud’s docs and issues and comments like this thread, it’s not entirely clear to me whether nextcloud supports our use-case as efficiently. I find quite a bit of discussion about the lack of efficient sync for the “a few bytes in a large file” case, but I can’t find anything about efficient copying/duplication of filesystem structures where most of the files don’t change… which suggests to me Nextcloud might already support it. Just can’t find a clear answer though… which suggests to me it does work efficiently, but it might just be that people are far more interested in the minor-changes-to-a-VHD case. Interested in a definitive answer before I dive in and just try it.

(The user group has proven fairly resistant to adopting version control tools like git/mercurial, BTW, before anyone points out those might be a better solution!).

[towerlexa]

Hi @All, i would like to tell you, that i’ve the same issue. The content from the linked dropbox account won’t never be updated

This is mostly for the subfolders! The content in the root directory of the dropboxwill be refreshed, but never the content of the subfolders.

[UEPSILON]

Same here …

It is also important for people with a bad or limited internet connection. Such a big difference if you have to update a file (for example a Layout-file with 120 MB) only once and afterwards the changed bits and bytes or the whole file 10 or 20 times with every change. The only reason not to fully move to nextcloud is the lack of this feature AND the consideration to jump over to ownCloud if this really works what they have planned with their DeltaUpdate project!