Desktop client delta updates to files

I checked the owncloud github on the sync client and only saw one thread that lead to a dead end. I was wondering if people are still interested in sending just the changes of a file and not the file in its entirety. I know a local business that wants to use Nextcloud for their internal use and clients but they work with big files all day long that take small changes throughout the day. It would be nice just to send maybe couple MB changes and not whole 100MB or whatever size files each time the client syncs with the server

1 Like

This features has been frequently asked for. Unfortunately, it wasn’t implemented yet. If you really need this feature for business application, try to contact the enterprise support. As enterprise customer you have certain influence on the future features and how they are prioritized. On bountysource there is already a considerable amount of money on this feature (>1000$), or if you have experience in programming you can directly help with the implementation.

There is a workaround that you use syncthing as a sync client and then use the sync-thing syncing folder as external storage in Nextcloud (to have the web-interface, sharing links, …)

I was actually going to mention syncthing in my post haha. Good to know. Is there any influence on features even as a non-enterprise customer? I’m still not a seasoned programmer, but I’ve been learning languages like PHP so I could contribute to Nextcloud. I have a feeling delta syncing like what syncthing does might be a hard thing to implement but if I feel like I can accomplish it I might try to do the work in my spare time and hope smarter people than I can build on it

I reply on this post just to keep it alive and visible for the community, because I think the importance of this post/feature is underrated at the moment.

For me delta upload is a key feature of a sync client. No matter how your installation looks like, the overall goal when syncing is to keep traffic low and sync times short.

It’s clear to me, that it would be a big change in nextcloud/owncloud client’s methodology of syncing but from my point of view the invest is worth it.

From my perspective this is a key feature, which even big commercial-cloud-service-provider do not supply and therefor could be another stand-alone functionality of nextcloud. Therefore this feature should not be on a “nice-to-have” or “long-term-goal” list. You can have the most secure system in the world but in most cases customers will decide based on their user experience which system they choose.

Last I want to mention that if you combine delta sync with file-version-control (what would definitely make sense to me), this is exactly what rdiff does.

For those of you who are interested in publications on this topic this presentation of CERN might be interesting.

I have to modify my existing post, because I found some interesting article & discussion providing information I was not aware of.

For those who do not want to read everything I quote the part which is important to me:

After reading this article and especially the quoted statement my personal opinion about the delta-sync topic is:

  • The Idea to let customers pay for implementation of specific file-transfer algorithm sounds reasonable

  • If not existing already the core (client & server) should have the functionality to implement file specific algorithm

Just wanted to add one potential user’s perspective here…

I work with a globally distributed group of collaborators who are basically professional data wranglers of content (text, images and video) for ebook-like software. We’re fairly intensive users of Dropbox but would be quite interested in some self-hosted replacement; Nextcloud got on my radar from a mention in a recent presentation at the Edinburgh techmeetup.

One use case which is absolutely crucial for us is:

  • Someone has a ~1-to-10 GByte folder of .jpg and .mov files called, say, Dropbox/ebook_content_version_0.0.1/
  • They copy that wholesale to Dropbox/ebook_content_version_0.0.2/ and maybe add or delete a few jpgs or simply rename a subdirectory containing a few files. Maybe only 1-10% of the bytes change.
  • Dropbox is absolutely brilliant about syncing this to other collaborators so that Dropbox/ebook_content_version_0.0.2 appears for them with a minimal amount of bandwidth consumed; it’s obviously doing something clever with file checksums or hashes so that it knows most of ebook_content_version_0.0.2/ is “already there” in ebook_content_version_0.0.1/ (actually it seems to me Dropbox uses some sort of block-level hash as I’ve seen it sometimes transfer large files where just a few bytes have changed very quickly, but we don’t care so much about that).

From reading Nextcloud’s docs and issues and comments like this thread, it’s not entirely clear to me whether nextcloud supports our use-case as efficiently. I find quite a bit of discussion about the lack of efficient sync for the “a few bytes in a large file” case, but I can’t find anything about efficient copying/duplication of filesystem structures where most of the files don’t change… which suggests to me Nextcloud might already support it. Just can’t find a clear answer though… which suggests to me it does work efficiently, but it might just be that people are far more interested in the minor-changes-to-a-VHD case. Interested in a definitive answer before I dive in and just try it.

(The user group has proven fairly resistant to adopting version control tools like git/mercurial, BTW, before anyone points out those might be a better solution!).

Hi @All, i would like to tell you, that i’ve the same issue. The content from the linked dropbox account won’t never be updated

This is mostly for the subfolders! The content in the root directory of the dropboxwill be refreshed, but never the content of the subfolders.

Same here …

It is also important for people with a bad or limited internet connection. Such a big difference if you have to update a file (for example a Layout-file with 120 MB) only once and afterwards the changed bits and bytes or the whole file 10 or 20 times with every change. The only reason not to fully move to nextcloud is the lack of this feature AND the consideration to jump over to ownCloud if this really works what they have planned with their DeltaUpdate project!