Large 7zip file downloads stop at 500MB

ℹ️ Support
1

I uploaded a 698MB compressed 7zip file to my NextCloud server. The upload went fine, and the file appears as 698MB in NextCloud.

When someone attempts to download the 7z file to a Windows machine, the file download appears to complete without error, but the file size is only 500MB (524,423,168 bytes). I’m able to reproduce this issue, and am unable to download the entire 7zip file.

I initially thought this was a max file size issue, so I increased the max file size setting to 2GB in the .htaccess file on my server, per these instructions:

https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/big_file_upload_configuration.html

Despite this change, downloaded 7z files are still limited to 500MB.

image

As a separate test, I uploaded a 787MB uncompressed TXT file. The large uncompressed text file downloads fine, and the full 787MB file is complete and readable after download.

Is there something about downloading a large 7zip file that is different than downloading a large text file? Is this a known limitation or issue with NextCloud?

If this is not a known issue, is there a github repo where I can submit an issue?

2

Are you sure the whole file was uploaded? You can use checksums (app for web-interface) to compare the checksum to your local file. Or check the real filesize in the file system.

3

Great idea!

Thanks for the suggestion of the Checksum app! That is brilliant.

The checksum on my NextCloud server matches the original file that I have locally. So it appears that the NextCloud server does have a complete copy of the file.

MD5Hash

And cPanel shows the same file size.

image
image938×110 5.04 KB

Based on this, it would seem that the file is intact on the server, but the issue is with downloading 7zip files from NextCloud?

I’ll try with a large .zip file to see if it has the same issue.

Does NextCloud maintain detailed logs that might have any clues? Perhaps a streaming error when the file is downloaded?

I’m open to any other suggestions.

4

I was able to download the full 7zip file from cPanel. So it does appear that something in the NextCloud server is the cause of the 500MB download limit with 7zip files.

image

I’ll try a .zip file next to see if it behaves the same.

5

In case it is relevant, I found this old GitHub Issue that mentions problems with 7zip files:

Could there be some configuration option with this streaming library that is limiting file downloads to 500MB?

6

If you suspect, it is just the 7z extension, I’d change the extension only and try to download.

The linked topic says that some of the issues are fixed in newer versions of ownCloud and Nextcloud applied several fixes as well. Normally if you download a single file, it is downloaded directly without zipping (in contrast to several files or whole folders).

The logfiles would be interesting as well. Do you use a reverse proxy? I found this: Problems downloading large (>1GB) files in Nextcloud - #5 by loicmla - Support - Enough

7

I tried naming the file extension and found something interesting.

If I use an extension of “.TEST”, the 7z file is downloaded in a similar manner as the 7z, and the file is limited to 500MB. It appears to use a different ‘protocol’ (?) or process, as my browser displays the expected total file size.

But if I rename it with an extension of .txt, NextCloud uses a different download mechanism. My browser downloads the file without showing the expected file size.

image

So my guess is that NextCloud uses a different file streaming mechanism for compressed files. That topic is far outside of my realm, but it reminds me of binary vs text FTP transfers and resumable FTP transfers. And whatever component is handling the binary file streaming has a 500MB limit.

The workaround appears to be to rename large 7zip files (> 500MB) with a “.txt” extension.

But if anyone knows how to increase that 7z file size limit, I’d be interested.