Can't get past 2 gb upload limit

catalyst1987 · September 8, 2019, 12:02pm

Support intro

Sorry to hear you’re facing problems

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can

Nextcloud version: 16.0.4
Operating system and version: Ubuntu Mate 18.04.3 on Raspberry Pi 3
Apache version : 2.4.29
PHP version: 7.2

The issue you are facing:
For whatever reason, I am still unable to upload files bigger than 2 GB despite going through everything on https://docs.nextcloud.com/server/stable/admin_manual/configuration_files/big_file_upload_configuration.html?highlight=file%20size

I checked the phpinfo.php and it definitely shows an upload limit of 16 GB. What’s interesting is that if I try to upload via either the sync client or via the web interface, it looks like it gets all the way through to uploading the size of the file and then errors. I for the life of me can’t figure out what is going wrong exactly. Any help would be greatly appreciated.

Many Thanks.

The output of your Nextcloud log in Admin > Logging:

Error	remote	TypeError: Return value of OC\Files\Storage\Wrapper\Encryption::writeStream() must be of the type integer, float returned
/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php - line 630:

OC\Files\Storage\Wrapper\Encryption->writeStream("downloads a ... t", null, null)

/var/www/nextcloud/apps/dav/lib/Connector/Sabre/File.php - line 191:

OC\Files\Storage\Wrapper\Wrapper->writeStream("downloads a ... t", null)

/var/www/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php - line 156:

OCA\DAV\Connector\Sabre\File->put(null)

/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Tree.php - line 316:

OCA\DAV\Connector\Sabre\Directory->createFile("THE_GODFATHER.m4v", null)

/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Tree.php - line 130:

Sabre\DAV\Tree->copyNode(OCA\DAV\Upload\FutureFile {}, OCA\DAV\Conn ... {}, "THE_GODFATHER.m4v")

/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Tree.php - line 161:

Sabre\DAV\Tree->copy("uploads/Clo ... e", "files/Cloud ... v")

/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php - line 642:

Sabre\DAV\Tree->move("uploads/Clo ... e", "files/Cloud ... v")

<<closure>>

Sabre\DAV\CorePlugin->httpMove(Sabre\HTTP\R ... "}, Sabre\HTTP\Response {})

/var/www/nextcloud/3rdparty/sabre/event/lib/EventEmitterTrait.php - line 105:

call_user_func_array([ Sabre\DAV\ ... "], [ Sabre\HTTP ... }])

/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 479:

Sabre\Event\EventEmitter->emit("method:MOVE", [ Sabre\HTTP ... }])

/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 254:

Sabre\DAV\Server->invokeMethod(Sabre\HTTP\R ... "}, Sabre\HTTP\Response {})

/var/www/nextcloud/apps/dav/lib/Server.php - line 316:

Sabre\DAV\Server->exec()

/var/www/nextcloud/apps/dav/appinfo/v2/remote.php - line 35:

OCA\DAV\Server->exec()

/var/www/nextcloud/remote.php - line 163:

require_once("/var/www/ne ... p")

Schmu · September 8, 2019, 2:10pm

Hi,

Could be a really tough one.

You are running the 64bit version of Ubuntu?
What filesystem are you using?

Are you uploading the file to the main data directory or is it some kind of other external storage like SMB share?
If so, is encryption only enabled for the that storage you are uploading to or for all connected and especially internal storages?

And the tmp directory which PHP and the webserver are using is big enough, right?

catalyst1987 · September 8, 2019, 2:26pm

Hello,
Thanks for the reply on this.

Since this is on a RPi3, it is using the ARM version which I guess works the same as 32bit.

The root file system is using Ext4 but the upload is supposed to save on a Local storage (an external HDD connected to the Pi) which is using NTFS.

I haven’t enabled encryption anywhere. Files smaller than 2GB upload just fine, so I wouldn’t have thought it would be an encryption problem, but I could be wrong.

tmp directory is within the root filesystem on the SD card. It still have about 9 GB of spare disk space left and the file I am trying to upload is about 2.1 GB in size.

Many Thanks.

KarlF12 · September 8, 2019, 4:39pm

This is asking for trouble, and I would encourage you to reconsider. The only reason to have the data folder on NTFS is if you intend to access the files directly outside of Nextcloud which is inadvisable (even though a lot of people do it anyway) and will probably cause strange issues.

It’s fine to use the external drive, but it should be formatted ext4 or another supported filesystem and not accessed except through Nextcloud.

yawicz · September 8, 2019, 7:51pm

Hi catalyst1987,

Since this is on a RPi3, it is using the ARM version which I guess works the same as 32bit.

I’m not a pro, but I think that’s exactly where your (and also my) problem lies: The 32bit architecture of the Raspberry Pi (OS).
As you can read here or here, the processes in 32bit systems can only handle up to 2GB at once - thus in this case the php process can’t handle more than a 2GB file at once.

Please correct me if I’m wrong, but I think that may be the reason.

My workaround is to zip files bigger than 2GB into 2GB splits - not super nice, but does the job.

Best,
yawicz

catalyst1987 · September 8, 2019, 8:03pm

It should be able to upload the file in chunks to get around this limitation. You just need to make sure your php.ini files are set to upload bigger than 2 GB. Interestingly enough, I get a partial file with “.part” extension that is 2.2 GB in size (which is larger than my original file) but then when finishing processing the file, it gets a 500 error.

catalyst1987 · September 8, 2019, 8:06pm

I tried changing my fstab to mount the drives with ntfs-3g instead, but no further luck. Since I only have about 70 GB of space left on that particular drive, I need to buy a new one anyways, so I may try that and format it to ext4 and move everything from the old drive to the new one.

yawicz · September 8, 2019, 8:35pm

Ah, okay, that is interesting, to be honest I never tried so far because I thought it wouldn’t work, so I will do that right away and see if it works on my system.
Since we’re both on Raspberry Pi, may I also refer you to the topic I just opened (Nextcloud limited to exactly 15 Mbit - What is the bottleneck / max performance of Raspberry Pi? - #2 by yawicz) to post your max up-/download speeds? I would be really interested if it’s only me or if my max speeds are “normal”.
I’ll report the upload of more than 2GB works on my system later on.

Edit:
Ok, it actually works on my system, the upload finishes and the file is there, but it still throws an error in the log:

Error PHP file_put_contents(): content truncated from 4194304306 to 2147483647 bytes at /var/www/nextcloud/lib/private/Files/Storage/Local.php#488

{“reqId”:“Bx4l0NG0KsxdzLlQthY0”,“level”:3,“time”:“2019-09-08T21:22:44+00:00”,“remoteAddr”:“x.x.x.x”,“user”:“user”,“app”:“PHP”,“method”:“MOVE”,“url”:“/cloud/remote.php/dav/uploads/user/web-file-upload-1092d7830d97b8456cdb0810109fcc2f-1567975014271/.file”,“message”:“file_put_contents(): content truncated from 4194304306 to 2147483647 bytes at /var/www/nextcloud/lib/private/Files/Storage/Local.php#488”,“userAgent”:“Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0”,“version”:“16.0.4.1”}

Sorry if I can’t be of more help.

catalyst1987 · September 21, 2019, 10:28am

Okay, so I got a new hard drive and have it run with ext4. And unfortunately I still get the same problem so I don’t think the file system is the issue.

Blockquote

|Error|remote|TypeError: Return value of OC\Files\Storage\Wrapper\Encryption::writeStream() must be of the type integer, float returned

/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php - line 630:OC\Files\Storage\Wrapper\Encryption->writeStream(“downloads a … t”, null, null)
/var/www/nextcloud/apps/dav/lib/Connector/Sabre/File.php - line 191:OC\Files\Storage\Wrapper\Wrapper->writeStream(“downloads a … t”, null)
/var/www/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php - line 156:OCA\DAV\Connector\Sabre\File->put(null)
/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Tree.php - line 316:OCA\DAV\Connector\Sabre\Directory->createFile(“THE_GODFATHER.m4v”, null)
/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Tree.php - line 130:Sabre\DAV\Tree->copyNode(OCA\DAV\Upload\FutureFile {}, OCA\DAV\Conn … {}, “THE_GODFATHER.m4v”)
/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Tree.php - line 161:Sabre\DAV\Tree->copy(“uploads/Clo … e”, “files/Cloud … v”)
/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php - line 642:Sabre\DAV\Tree->move(“uploads/Clo … e”, “files/Cloud … v”)
<>Sabre\DAV\CorePlugin->httpMove(Sabre\HTTP\R … "}, Sabre\HTTP\Response {})
/var/www/nextcloud/3rdparty/sabre/event/lib/EventEmitterTrait.php - line 105:call_user_func_array([ Sabre\DAV\ … "], [ Sabre\HTTP … }])
/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 479:Sabre\Event\EventEmitter->emit(“method:MOVE”, [ Sabre\HTTP … }])
/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 254:Sabre\DAV\Server->invokeMethod(Sabre\HTTP\R … "}, Sabre\HTTP\Response {})
/var/www/nextcloud/apps/dav/lib/Server.php - line 316:Sabre\DAV\Server->exec()
/var/www/nextcloud/apps/dav/appinfo/v2/remote.php - line 35:OCA\DAV\Server->exec()
/var/www/nextcloud/remote.php - line 163:require_once(“/var/www/ne … p”)|
| — | — |

Blockquote

catalyst1987 · October 5, 2019, 1:12pm

Okay, so just an FYI in case it helps out anyone else who encounters the same problem, I managed to figure this out.

Even though in the settings I had encryption unticked, I had the encryption module enabled in the list of Nextcloud Apps. When I disabled the default encryption app, I was then able to upload my 2 GB+ file across.