Uploading Files bigger than 2GB Fails with "Error when assembling chunks, status code 500"

buddens93 · May 8, 2020, 3:14pm

Setup
Raspberry Pi 4b (4GB) running Raspian 10 Buster
NCP 1.25.0 (installed via curl one-liner)
External Storage Support 1.9.0
NextCloud 18.0.4.2 (Apache2.4) with NCP LetsEncrypt behind Router/Port-Forward NAT (no proxy)

Expected behavior
Being able to upload files bigger than 2GB via Nextcloud WebUI

Actual behavior
Drag’ n’ drop file in Nextcloud WebUI fully uploads until ‘processing’, then spits “Error when assembling chunks, status code 500” error message.
Uploaded file can be seen in the destination folder but hidden with .part file appended at the end.
Uploaded file does not get renamed as original and chunked file segments in /tmp do not get deleted.

Observations/Notes
The uploaded .part file is exactly the same size as original and has an identical checksum, indicating the file is fully uploaded, successfully chunked in /tmp and successfully re-assembled at destination - but then failing on rename.

Destination folder is a SMB share on a ReadyNAS mounted in Nextcloud using External Storage support app. All files below 2GB successfully upload to this location without issue and I can rename files and create folders as needed.

Troubleshooting to-date

Triple-checked all PHP parameters (using loaded phpinfo.php) are set to support large file uploads as per the many forum support posts and https://docs.nextcloud.com/server/stable/admin_manual/configuration_files/big_file_upload_configuration.html
Disabled mod_reqtimeout module (not using mod_fcgid)
Disabled HTTP2 (no change, so re-enabled)
Created guest/anonymous SMB destination folder with 777 permissions (same result)
Mounted local NFS storage to use as /tmp
Downgrading to Nextcloud 18.0.3 and upgrading to Beta 19

I’m at a loss as to what’s causing the final rename to fail, I can’t see what else might be timing out (SSL?) and the fact the file chunks and re-assembles bit-for-bit at the destination makes me think this isn’t the usual 32-bit headaches or PHP restrictions.

System information

NextCloudPi diagnostics


NextCloudPi version  v1.25.0
distribution         Raspbian GNU/Linux 10 \n \l
automount            no
USB devices          none
datadir              /var/www/nextcloud/data
data in SD           yes
data filesystem      ext2/ext3
data disk usage      8.0G/30G
rootfs usage         8.0G/30G
swapfile             /var/swap
dbdir                /var/lib/mysql
Nextcloud check      ok
Nextcloud version    18.0.4.2
HTTPD service        up
PHP service          up
MariaDB service      up
Redis service        up
Postfix service      up
internet check       ok
port check 80        open
port check 443       open
IP                   ***REMOVED SENSITIVE VALUE***
gateway              ***REMOVED SENSITIVE VALUE***
interface            eth0
certificates         ***REMOVED SENSITIVE VALUE***
NAT loopback         no
uptime               1:48

Nextcloud configuration


{
    "system": {
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": {
            "0": "localhost",
            "11": "***REMOVED SENSITIVE VALUE***",
            "1": "192.168.22.3",
            "4": "***REMOVED SENSITIVE VALUE***",
            "5": "nextcloudpi.local",
            "7": "nextcloudpi",
            "8": "nextcloudpi.lan",
            "12": "***REMOVED SENSITIVE VALUE***"
        },
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "18.0.4.2",
        "overwrite.cli.url": "https:\/\/***REMOVED SENSITIVE VALUE***\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "memcache.local": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "timeout": 0,
            "password": "***REMOVED SENSITIVE VALUE***"
        },
        "tempdirectory": "\/var\/www\/nextcloud\/data\/tmp",
        "mail_smtpmode": "sendmail",
        "mail_smtpauthtype": "LOGIN",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "preview_max_x": "2048",
        "preview_max_y": "2048",
        "jpeg_quality": "60",
        "overwriteprotocol": "https",
        "loglevel": "2",
        "log_type": "file",
        "maintenance": false,
        "htaccess.RewriteBase": "\/",
        "mail_sendmailmode": "pipe",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "data-fingerprint": "57fdcd603890ec673c4cbfec6e2b2c8a"
    }
}

Nextcloud logs




{"reqId":"XrVQ44RE0yutN3Ifaf3dUgAAVAA","level":3,"time":"2020-05-08T12:31:09+00:00","remoteAddr":"***REMOVED SENSITIVE VALUE***","user":"joe","app":"no app in context","method":"PUT","url":"/remote.php/dav/uploads/joe/web-file-upload-3fd3b59b597d0beeada0ab231132b4dd-1588941033030/0","message":{"Exception":"Sabre\\DAV\\Exception\\BadRequest","Message":"Expected filesize of 10485760 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 1400832 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.","Code":0,"Trace":[{"file":"/var/www/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php","line":156,"function":"put","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[null]},{"file":"/var/www/nextcloud/apps/dav/lib/Upload/UploadFolder.php","line":47,"function":"createFile","class":"OCA\\DAV\\Connector\\Sabre\\Directory","type":"->","args":["0",null]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":1096,"function":"createFile","class":"OCA\\DAV\\Upload\\UploadFolder","type":"->","args":["0",null]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":525,"function":"createFile","class":"Sabre\\DAV\\Server","type":"->","args":["uploads/joe/web-file-upload-3fd3b59b597d0beeada0ab231132b4dd-1588941033030/0",null,null]},{"function":"httpPut","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[{"absoluteUrl":"https://***REMOVED SENSITIVE VALUE***/remote.php/dav/uploads/joe/web-file-upload-3fd3b59b597d0beeada0ab231132b4dd-1588941033030/0","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/var/www/nextcloud/3rdparty/sabre/event/lib/EventEmitterTrait.php","line":105,"function":"call_user_func_array","args":[[{"__class__":"Sabre\\DAV\\CorePlugin"},"httpPut"],[{"absoluteUrl":"https://***REMOVED SENSITIVE VALUE***/remote.php/dav/uploads/joe/web-file-upload-3fd3b59b597d0beeada0ab231132b4dd-1588941033030/0","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":479,"function":"emit","class":"Sabre\\Event\\EventEmitter","type":"->","args":["method:PUT",[{"absoluteUrl":"https://***REMOVED SENSITIVE VALUE***/remote.php/dav/uploads/joe/web-file-upload-3fd3b59b597d0beeada0ab231132b4dd-1588941033030/0","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":254,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[{"absoluteUrl":"https://***REMOVED SENSITIVE VALUE***/remote.php/dav/uploads/joe/web-file-upload-3fd3b59b597d0beeada0ab231132b4dd-1588941033030/0","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/var/www/nextcloud/apps/dav/lib/Server.php","line":319,"function":"exec","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/nextcloud/apps/dav/appinfo/v2/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/nextcloud/remote.php","line":165,"args":["/var/www/nextcloud/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/nextcloud/apps/dav/lib/Connector/Sabre/File.php","Line":229,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36","version":"18.0.4.2"}
{"reqId":"XrVQ44RE0yutN3Ifaf3dUgAAVAA","level":4,"time":"2020-05-08T12:31:09+00:00","remoteAddr":"***REMOVED SENSITIVE VALUE***","user":"joe","app":"webdav","method":"PUT","url":"/remote.php/dav/uploads/joe/web-file-upload-3fd3b59b597d0beeada0ab231132b4dd-1588941033030/0","message":{"Exception":"Sabre\\DAV\\Exception\\BadRequest","Message":"Expected filesize of 10485760 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 1400832 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.","Code":0,"Trace":[{"file":"/var/www/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php","line":156,"function":"put","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[null]},{"file":"/var/www/nextcloud/apps/dav/lib/Upload/UploadFolder.php","line":47,"function":"createFile","class":"OCA\\DAV\\Connector\\Sabre\\Directory","type":"->","args":["0",null]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":1096,"function":"createFile","class":"OCA\\DAV\\Upload\\UploadFolder","type":"->","args":["0",null]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":525,"function":"createFile","class":"Sabre\\DAV\\Server","type":"->","args":["uploads/joe/web-file-upload-3fd3b59b597d0beeada0ab231132b4dd-1588941033030/0",null,null]},{"function":"httpPut","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[{"absoluteUrl":"https://***REMOVED SENSITIVE VALUE***/remote.php/dav/uploads/joe/web-file-upload-3fd3b59b597d0beeada0ab231132b4dd-1588941033030/0","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/var/www/nextcloud/3rdparty/sabre/event/lib/EventEmitterTrait.php","line":105,"function":"call_user_func_array","args":[[{"__class__":"Sabre\\DAV\\CorePlugin"},"httpPut"],[{"absoluteUrl":"https://***REMOVED SENSITIVE VALUE***/remote.php/dav/uploads/joe/web-file-upload-3fd3b59b597d0beeada0ab231132b4dd-1588941033030/0","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":479,"function":"emit","class":"Sabre\\Event\\EventEmitter","type":"->","args":["method:PUT",[{"absoluteUrl":"https://***REMOVED SENSITIVE VALUE***/remote.php/dav/uploads/joe/web-file-upload-3fd3b59b597d0beeada0ab231132b4dd-1588941033030/0","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":254,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[{"absoluteUrl":"https://***REMOVED SENSITIVE VALUE***/remote.php/dav/uploads/joe/web-file-upload-3fd3b59b597d0beeada0ab231132b4dd-1588941033030/0","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/var/www/nextcloud/apps/dav/lib/Server.php","line":319,"function":"exec","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/nextcloud/apps/dav/appinfo/v2/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/nextcloud/remote.php","line":165,"args":["/var/www/nextcloud/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/nextcloud/apps/dav/lib/Connector/Sabre/File.php","Line":229,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36","version":"18.0.4.2"}
{"reqId":"XrVfSdO9ReOVx5FZkIcF3wAABgo","level":3,"time":"2020-05-08T13:32:39+00:00","remoteAddr":"***REMOVED SENSITIVE VALUE***","user":"joe","app":"remote","method":"MOVE","url":"/remote.php/dav/uploads/joe/web-file-upload-fb81ff0be52545afac487680bc3991b3-1588943313178/.file","message":{"Exception":"TypeError","Message":"Return value of OC\\Files\\Storage\\Common::writeStream() must be of the type int, float returned","Code":0,"Trace":[{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":630,"function":"writeStream","class":"OC\\Files\\Storage\\Common","type":"->","args":["Kifisia/Kifisia-Zero/EZdrummer2_Demo.zip.ocTransferId985752881.part",null,null]},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":630,"function":"writeStream","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["Kifisia/Kifisia-Zero/EZdrummer2_Demo.zip.ocTransferId985752881.part",null,null]},{"file":"/var/www/nextcloud/apps/dav/lib/Connector/Sabre/File.php","line":192,"function":"writeStream","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["Kifisia/Kifisia-Zero/EZdrummer2_Demo.zip.ocTransferId985752881.part",null]},{"file":"/var/www/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php","line":156,"function":"put","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[null]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Tree.php","line":316,"function":"createFile","class":"OCA\\DAV\\Connector\\Sabre\\Directory","type":"->","args":["EZdrummer2_Demo.zip",null]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Tree.php","line":130,"function":"copyNode","class":"Sabre\\DAV\\Tree","type":"->","args":[{"__class__":"OCA\\DAV\\Upload\\FutureFile"},{"__class__":"OCA\\DAV\\Connector\\Sabre\\Directory"},"EZdrummer2_Demo.zip"]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Tree.php","line":161,"function":"copy","class":"Sabre\\DAV\\Tree","type":"->","args":["uploads/joe/web-file-upload-fb81ff0be52545afac487680bc3991b3-1588943313178/.file","files/joe/Joe/Kifisia/Kifisia-Zero/EZdrummer2_Demo.zip"]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":642,"function":"move","class":"Sabre\\DAV\\Tree","type":"->","args":["uploads/joe/web-file-upload-fb81ff0be52545afac487680bc3991b3-1588943313178/.file","files/joe/Joe/Kifisia/Kifisia-Zero/EZdrummer2_Demo.zip"]},{"function":"httpMove","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[{"absoluteUrl":"https://***REMOVED SENSITIVE VALUE***/remote.php/dav/uploads/joe/web-file-upload-fb81ff0be52545afac487680bc3991b3-1588943313178/.file","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/var/www/nextcloud/3rdparty/sabre/event/lib/EventEmitterTrait.php","line":105,"function":"call_user_func_array","args":[[{"__class__":"Sabre\\DAV\\CorePlugin"},"httpMove"],[{"absoluteUrl":"https://***REMOVED SENSITIVE VALUE***/remote.php/dav/uploads/joe/web-file-upload-fb81ff0be52545afac487680bc3991b3-1588943313178/.file","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":479,"function":"emit","class":"Sabre\\Event\\EventEmitter","type":"->","args":["method:MOVE",[{"absoluteUrl":"https://***REMOVED SENSITIVE VALUE***/remote.php/dav/uploads/joe/web-file-upload-fb81ff0be52545afac487680bc3991b3-1588943313178/.file","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":254,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[{"absoluteUrl":"https://***REMOVED SENSITIVE VALUE***/remote.php/dav/uploads/joe/web-file-upload-fb81ff0be52545afac487680bc3991b3-1588943313178/.file","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/var/www/nextcloud/apps/dav/lib/Server.php","line":319,"function":"exec","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/nextcloud/apps/dav/appinfo/v2/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/nextcloud/remote.php","line":165,"args":["/var/www/nextcloud/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/nextcloud/lib/private/Files/Storage/Common.php","Line":866,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36","version":"18.0.4.2"}

Loaded PHP Version 7.3.14-1 Settings (as per phpinfo.ini)

upload_max_filesize = 10G

Apache nextcloud.conf
LimitRequestBody 0
SSLRenegBufferSize 10486000

Any ideas or guidance gratefully received!

Cheers

forzajappie · May 9, 2020, 8:55pm

It seems that more people ran into this or similar problems, you can google on your error message. You can always try some other scenario’s just to check where the error is originating from. Can you upload the file to that location directly? Can you upload the same file to another location within your nextcloud instance?

Vincent_Stans · May 10, 2020, 8:45am

You can check if it’s a samba issue

https://www.ibm.com/support/pages/samba-share-will-not-allow-copying-files-larger-2-gb

buddens93 · May 12, 2020, 5:39pm

It seems that more people ran into this or similar problems, you can google on your error message. You can always try some other scenario’s just to check where the error is originating from. Can you upload the file to that location directly? Can you upload the same file to another location within your nextcloud instance?

@forzajappie Your right, there’s lots of posts about people having issues uploading large files but they all refer to 504 or other errors, not 500. I’ve still tried all of the timeout fixes and max limits advised for Apache/PHP but none fix the issue. I can write and read files/folders to the samba storage from within the Nextcloud WebUI, but uploading anything over 2GB fails on rename (even though the full .part file is in the destination smb share).

You can check if it’s a samba issue
[https://www.ibm.com/support/pages/samba-share-will-not-allow-copying-files-larger-2-gb ]

@Vincent_Stans Thanks for the suggestion - I think this references an issue with earlier versions of Samba which has since been addressed. It’s also worth pointing out I have got the same setup working succesfully between a FreeNAS SMB share and a Nextcloud Ubuntu VM, so it’s not a limit of SMB/CIFS.

I’ve created a test setup with Rockstor NAS (VM) and been able to replicate the issue using:

RPi3 running latest Raspian with NextCloudPi one-liner CURL setup
RPi3 running latest NextCloudPi RPI Image
Ubuntu 20.04 VM running latest NextCloud Pi Docker
Mounted External Storage as FTP (ruling out SMB issue)

All the above failed with the same 500 error, along with the following log entry:
[remote] Error: TypeError: Return value of OC\Files\Storage\Common::writeStream() must be of the type int, float returned at <<closure>>

The only workaround I’ve come across so far is mounting SMB share as local storage (via fstab) and using the local paths option in External storage app, which isn’t ideal.

Update:
As a final solution I resorted to installing NCP on a Debian 64-bit build for the RPi using this guide here (I actually used a 3B+ with success):
Installing 64-bit Debian on a Raspberry Pi 4
I was then finally able to upload files bigger than 2GB!

Thanks