Getting mad copying data with rclone to an external FTP storage with encryption

Nextcloud version: 22.3.0
Operating system and version: Ubuntu 20.04
Apache or nginx version: nginx/1.18.0
PHP version: 7.4.3

This is more a rant (or a beg for help) rather than a support request. :sweat_smile:

I’m trying to copy a local directory to a Nextcloud one using rclone. The Nextcloud destination directory is an FTP location mounted as external storage, with encryption enabled.

It’s not a real error so I don’t have NC logs to show. I’ve been trying to copy this stuff (around 2TB) for months now (due to bandwidth and other issues, outlined below). I hoped I reached the end, unfortunately there’s a new surprise after every run.

I already wrote on rclone forum, but as I’m growing doubts and failures I’m trying to gather help from both communities.

Back to the root, my goal is to back up a constantly growing video archive. I want to add new videos, and possibly delete some on the source keeping them on Nextcloud. I’ve chosen NC as we already have it for common file sharing. Unfortunately I had to do a new installation, as enabling encryption would make regular files inaccessible (I have to dig into our ticketing to find the related issues, if someone is interested).

Anyway a dedicated NC with just the external FTP storage and encryption enabled worked. It’s dead slow, but seems working.

I start the rclone copy processes, the Nextcloud(FTP) archive grows. As it takes time, I had to run rclone several times to have the upload completed
I didn’t check what was going on as it is a big archive, I relied on the stability of both rclone and Nextcloud. My bad for not testing all the stuff with a small portion of the files first.

Now that I’m (almost) done, I realize rclone is not fully capable of detecting changes on files.

At first I used the default check mechanism, based on modified timestamp and file size. Now I found out Nextcloud is unable to store mtime for external storage #15192.
So I fell back to --size-only check. No way, as Nextcloud seems to report back the encrypted file size, which will never be the same as the original file! This doesn’t seem to be entirely true, as for some dirs the size reported by rclone lsl is the same of the local file, even tough the file on the external FTP storage is of course bigger.

Unfortunately I’ve not been able to identify what introduces this difference of behavior.
Hints on how to identify this pattern could be a step ahead, at this stage.

My last resort is using rclone --checksum. A first run on a small directory seems to correctly identify existing file and skip them, I’m now testing on a larger one.

If anyone had similar experiences, they’re welcome.