Freezing during download

Hi all,

I’m trying to make a backup of my nextcloud profile. Using webdav, windows won’t download the files due to size limitations. Using rclone the files start out fine and then slowly freeze.

With rclone nextcloud will stop responding and require a hard php-fpm/nginx restart to fix it.

Downloading the entire folder from nextcloud (automatic zip) will get to around 59GB and then stall and fail.

Any ideas? Webserver is nginx with php7.3-fpm.

What exactly are you trying to do? A zip download through the webinterface will most likely not work, as the server needs a lot of space to generate the zip file for you, which is likely to fail. Zip download is not meant for huge amounts of data.

I am not aware of any file size limitations in WebDAV. In General, you should be able to download all your data through WebDAV. What is the exact error message?

Furthermore, both ZIP and WebDAV aren’t meant for backups. You should use a proper tool like rsync over ssh access if possible, as that will give you much more reliable and reproducible results (As well as incremental backups without having to download all files again, only changed ones etc.)

The server has over 20TB free space so that’s not the issue.

The solution to my problem ended up being limiting rclone to a single transfer at a time on a linux kvm. Apparently the built in windows webdav also has a built in file size limitation which has to be removed by editing the registry.

The web interface still failed to download the zip folder so webdav was the only solution.

Furthermore, both ZIP and WebDAV aren’t meant for backups. You should use a proper tool like rsync over ssh access if possible, as that will give you much more reliable and reproducible results (As well as incremental backups without having to download all files again, only changed ones etc.)

Yes they are. How else do you backup your files? If I was a regular nextcloud user in a production environment none of those options would be possible.

Using h5ai you can download very large 200GB+ folders with an automatic tar/zip function, same concept should apply to nextcloud.

Why not use the Nextcloud Client, or any other proper WebDAV client, for backing up our data? If you were a regular user, and had no ssh access, that would be the best option to get a 1:1 copy of your data.

Of course, you can use the ZIP Download as backup function. But that’s not what it’s meant for. A change in the architecture would be needed, where ZIP Download isn’t happening in real time, but where the zip file is generated first and a download link is sent once it is ready. Generating ZIP files on the fly is not stable enough for transfers longer than a couple of minutes. Too much can go wrong, from connection instability over buffer underruns to memory/space problems, that would need the whole download to be restarted from the beginning (including the ZIP Compression). That just doesn’t scale for bigger transfers.

What are you basing this assumption on?

Why not use the Nextcloud Client, or any other proper WebDAV client, for backing up our data? If you were a regular user, and had no ssh access, that would be the best option to get a 1:1 copy of your data.

Okay so the built in windows webdav is not a proper client? And rclone isn’t either?

What is a “proper webdav client” in your eyes other then the nextcloud app?

Will wait for a response.

Hey, what’s wrong? I was just posting an answer to your question. I don’t see the point in reacting like that.

Of course, a client that introduces arbitrary file size limits, that are not part of the WebDAV specification, is not properly implemented. If it was, you would have been able to use it in order to download your data.

I guess, as your problem is solved, there is no point in discussing further. Have fun with Nextcloud!

Yesh more assumptions here. Problem has not been solved!

Reinstalling fixed the zip issue but webdav is still an issue.

I notice you haven’t answered my question on what is a proper webdav client! Is there something else I should be using other then the nextcloud app? I mean the nextcloud app cant be the only proper webdav client!

Thanks!