"no space left on device" when uploading 1.5GB or larger file

Nextcloud version (eg, 20.0.5): 21.0.1
Operating system and version (eg, Ubuntu 20.04): Linux 3.10.0-962 el7.x86_64
Apache or nginx version (eg, Apache 2.4.25): Apache litespeed
PHP version (eg, 7.4): 7.4.15

The issue you are facing:
upload of files over 1.5GB to external s3 storage appear to fail after upload. The progress bar fills and then switches to “Processing files …”. That never changes.

Confirmed large files support on external storage by using rclone.

I see “no space left on device”, but what device? All have plenty of space.

[w ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 126G 0 126G 0% /dev
/dev/sda4 58T 38T 18T 69% /opt
/dev/loop0 3.9G 222M 3.4G 7% /tmp
[w ~]$

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:

  1. connect external s3 storage
  2. upload 1.7GB file

The log entry
{"reqId":"YJF-eRWzRIv9VFkPWMl7JAACFRY","level":3,"time":"2021-05-04T17:09:10+00:00","remoteAddr":"A.B.C.D","user":"X","app":"PHP","method":"MOVE","url":"/remote.php/dav/uploads/X/web-file-upload-1d0247c2d6e3ce05a111915967eab45d-1620147067945/.file","message":{"Exception":"Error","Message":"fwrite(): write of 8192 bytes failed with errno=28 No space left on device at /public_html/3rdparty/guzzlehttp/psr7/src/Stream.php#247","Code":0,"Trace":[{"function":"onError","class":"OC\\Log\\ErrorHandler","type":"::","args":[8,"fwrite(): write of 8192 bytes failed with errno=28 No space left on device","/public_html/3rdparty/guzzlehttp/psr7/src/Stream.php",247,{"string":null}]},{"file":"/public_html/3rdparty/guzzlehttp/psr7/src/Stream.php","line":247,"function":"fwrite","args":[null,null]},{"file":"/public_html/3rdparty/guzzlehttp/psr7/src/Utils.php","line":53,"function":"write","class":"GuzzleHttp\\Psr7\\Stream","type":"->","args":[null]},{"file":"/public_html/3rdparty/guzzlehttp/psr7/src/functions.php","line":216,"function":"copyToStream","class":"GuzzleHttp\\Psr7\\Utils","type":"::","args":[{"stream":{"stream":{"__class__":"GuzzleHttp\\Psr7\\Stream"},"__class__":"GuzzleHttp\\Psr7\\LimitStream"},"__class__":"Aws\\HashingStream"},{"__class__":"GuzzleHttp\\Psr7\\Stream"},-1]},{"file":"/public_html/3rdparty/aws/aws-sdk-php/src/S3/MultipartUploader.php","line":117,"function":"GuzzleHttp\\Psr7\\copy_to_stream","args":[{"stream":{"stream":{"__class__":"GuzzleHttp\\Psr7\\Stream"},"__class__":"GuzzleHttp\\Psr7\\LimitStream"},"__class__":"Aws\\HashingStream"},{"__class__":"GuzzleHttp\\Psr7\\Stream"}]},{"file":"/public_html/3rdparty/aws/aws-sdk-php/src/Multipart/AbstractUploader.php","line":54,"function":"createPart","class":"Aws\\S3\\MultipartUploader","type":"->","args":[false,4]},{"file":"/public_html/3rdparty/aws/aws-sdk-php/src/CommandPool.php","line":55,"function":"getUploadCommands","class":"Aws\\Multipart\\AbstractUploader","type":"->","args":[{"__class__":"Closure"}]},{"function":"Aws\\{closure}","class":"Aws\\CommandPool","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/EachPromise.php","line":211,"function":"next","class":"Generator","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/EachPromise.php","line":153,"function":"advanceIterator","class":"GuzzleHttp\\Promise\\EachPromise","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/EachPromise.php","line":81,"function":"refillPending","class":"GuzzleHttp\\Promise\\EachPromise","type":"->","args":[]},{"file":"/public_html/3rdparty/aws/aws-sdk-php/src/CommandPool.php","line":79,"function":"promise","class":"GuzzleHttp\\Promise\\EachPromise","type":"->","args":[]},{"file":"/public_html/3rdparty/aws/aws-sdk-php/src/Multipart/AbstractUploadManager.php","line":132,"function":"promise","class":"Aws\\CommandPool","type":"->","args":[]},{"function":"Aws\\Multipart\\{closure}","class":"Aws\\Multipart\\AbstractUploadManager","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Coroutine.php","line":140,"function":"send","class":"Generator","type":"->","args":[{"__class__":"Aws\\Result"}]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":204,"function":"_handleSuccess","class":"GuzzleHttp\\Promise\\Coroutine","type":"->","args":[{"__class__":"Aws\\Result"}]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":153,"function":"callHandler","class":"GuzzleHttp\\Promise\\Promise","type":"::","args":[1,{"__class__":"Aws\\Result"},null]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/TaskQueue.php","line":48,"function":"GuzzleHttp\\Promise\\{closure}","class":"GuzzleHttp\\Promise\\Promise","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/public_html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php","line":158,"function":"run","class":"GuzzleHttp\\Promise\\TaskQueue","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php","line":183,"function":"tick","class":"GuzzleHttp\\Handler\\CurlMultiHandler","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":248,"function":"execute","class":"GuzzleHttp\\Handler\\CurlMultiHandler","type":"->","args":["*** sensitive parameter replaced ***"]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":224,"function":"invokeWaitFn","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":269,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":226,"function":"invokeWaitList","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":269,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":226,"function":"invokeWaitList","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":62,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Coroutine.php","line":67,"function":"wait","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":248,"function":"GuzzleHttp\\Promise\\{closure}","class":"GuzzleHttp\\Promise\\Coroutine","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":224,"function":"invokeWaitFn","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":269,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":226,"function":"invokeWaitList","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":62,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/public_html/3rdparty/aws/aws-sdk-php/src/Multipart/AbstractUploadManager.php","line":83,"function":"wait","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/public_html/lib/private/Files/ObjectStore/S3ObjectTrait.php","line":97,"function":"upload","class":"Aws\\Multipart\\AbstractUploadManager","type":"->","args":[]},{"file":"/public_html/apps/files_external/lib/Lib/Storage/AmazonS3.php","line":687,"function":"writeObject","class":"OCA\\Files_External\\Lib\\Storage\\AmazonS3","type":"->","args":["X9.MP4",null]},{"file":"/public_html/apps/files_external/lib/Lib/Storage/AmazonS3.php","line":535,"function":"writeBack","class":"OCA\\Files_External\\Lib\\Storage\\AmazonS3","type":"->","args":["/tmp/oc_tmp_QcNujw","X9.MP4"]},{"function":"OCA\\Files_External\\Lib\\Storage\\{closure}","class":"OCA\\Files_External\\Lib\\Storage\\AmazonS3","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/public_html/apps/files_external/3rdparty/icewind/streams/src/CallbackWrapper.php","line":119,"function":"call_user_func","args":[{"__class__":"Closure"}]},{"function":"stream_close","class":"Icewind\\Streams\\CallbackWrapper","type":"->","args":[]},{"file":"/public_html/lib/private/Files/Storage/Common.php","line":871,"function":"fclose","args":[null]},{"file":"/public_html/lib/private/Files/Storage/Wrapper/Wrapper.php","line":629,"function":"writeStream","class":"OC\\Files\\Storage\\Common","type":"->","args":["X9.MP4",null,null]},{"file":"/public_html/lib/private/Files/Storage/Wrapper/Wrapper.php","line":629,"function":"writeStream","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["X9.MP4",null,null]},{"file":"/public_html/lib/private/Files/Storage/Wrapper/Wrapper.php","line":629,"function":"writeStream","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["X9.MP4",null,null]},{"file":"/public_html/apps/dav/lib/Connector/Sabre/File.php","line":207,"function":"writeStream","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["X9.MP4",null]},{"file":"/public_html/apps/dav/lib/Connector/Sabre/Directory.php","line":156,"function":"put","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[null]},{"file":"/public_html/3rdparty/sabre/dav/lib/DAV/Tree.php","line":305,"function":"createFile","class":"OCA\\DAV\\Connector\\Sabre\\Directory","type":"->","args":["X9.MP4",null]},{"file":"/public_html/3rdparty/sabre/dav/lib/DAV/Tree.php","line":133,"function":"copyNode","class":"Sabre\\DAV\\Tree","type":"->","args":[{"__class__":"OCA\\DAV\\Upload\\FutureFile"},{"__class__":"OCA\\DAV\\Connector\\Sabre\\Directory"},"X9.MP4"]},{"file":"/public_html/3rdparty/sabre/dav/lib/DAV/Tree.php","line":163,"function":"copy","class":"Sabre\\DAV\\Tree","type":"->","args":["uploads/X/web-file-upload-1d0247c2d6e3ce05a111915967eab45d-1620147067945/.file","files/X/X/X9.MP4"]},{"file":"/public_html/apps/dav/lib/Upload/ChunkingPlugin.php","line":95,"function":"move","class":"Sabre\\DAV\\Tree","type":"->","args":["uploads/X/web-file-upload-1d0247c2d6e3ce05a111915967eab45d-1620147067945/.file","files/X/X/X9.MP4"]},{"file":"/public_html/apps/dav/lib/Upload/ChunkingPlugin.php","line":77,"function":"performMove","class":"OCA\\DAV\\Upload\\ChunkingPlugin","type":"->","args":["uploads/X/web-file-upload-1d0247c2d6e3ce05a111915967eab45d-1620147067945/.file","files/X/X/X9.MP4"]},{"file":"/public_html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"beforeMove","class":"OCA\\DAV\\Upload\\ChunkingPlugin","type":"->","args":["uploads/X/web-file-upload-1d0247c2d6e3ce05a111915967eab45d-1620147067945/.file","files/X/X/X9.MP4"]},{"file":"/public_html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":603,"function":"emit","class":"Sabre\\DAV\\Server","type":"->","args":["beforeMove",["uploads/X/web-file-upload-1d0247c2d6e3ce05a111915967eab45d-1620147067945/.file","files/X/X/X9.MP4"]]},{"file":"/public_html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpMove","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[{"__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/public_html/3rdparty/sabre/dav/lib/DAV/Server.php","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->","args":["method:MOVE",[{"__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"/public_html/3rdparty/sabre/dav/lib/DAV/Server.php","line":253,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[{"__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/public_html/3rdparty/sabre/dav/lib/DAV/Server.php","line":321,"function":"start","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/public_html/apps/dav/lib/Server.php","line":332,"function":"exec","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/public_html/apps/dav/appinfo/v2/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server","type":"->","args":[]},{"file":"/public_html/remote.php","line":167,"args":["/public_html/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/public_html/lib/private/Log/ErrorHandler.php","Line":92,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36","version":"21.0.1.1","id":"609182269c140"}

your system complains that no space is left in this directory. If you say you have enough space everywhere I’d check the permissions/owner on that folder… (together with all other folders)

Thanks very much for the help.

At each level from root, I find the permissions to be the same 755, and all the files in the target directory are 644. But this seems an odd place to upload a file, since it is all php code files.

Also, this does work for smaller files, like 1GB or even 1.4GB.

I can not find this path, which is what I thought was the location;
/remote.php/dav/uploads/X/web-file-upload-1d0247c2d6e3ce05a111915967eab45d-1620147067945/.file

When uploading big files, they are split into chunks, uploaded one by one and then re-assembled back into the original file. If you reach the “Processing files” stage, that normally means the chunks are uploaded and the re-assembly started…
The most common error at this stage is “error when assembling chunks” and that most of the time means slow processing: the CPU is too weak to finish the job in the allotted time…

I haven’t seen cases where 1.4GB file can be uploaded and 1.7GB cannot.
Must have something to do with S3 as storage…

This is most likely one of the chunks…

EDIT:
Discussion here might help (changing chunk sizes)… But S3 is some sort of exception…

Nice pointer!

With that I was able to find the uploads, home of many many 10MB files. It is not out of space.

The most common error at this stage is “error when assembling chunks” and that most of the time means slow processing: the CPU is too weak to finish the job in the allotted time…

This may apply since this is a basic offering from a provider. The upload is complete and
“Processing files” stage has begun.

The output of du shows larger files have been received, but not completed processing;

]$ du
1851628 ./web-file-upload-1d0247c2d6e3ce05a111915967eab45d-1620126265453
2730264 ./web-file-upload-3de22495a1b36b39cd73149bd3ab045a-1620069877505
1851628 ./web-file-upload-1d0247c2d6e3ce05a111915967eab45d-1620076785139
2611040 ./web-file-upload-a74840ac57c0c79f8bb5ef84a3541790-1620066632332
1851628 ./web-file-upload-1d0247c2d6e3ce05a111915967eab45d-1620075149967
1851628 ./web-file-upload-1d0247c2d6e3ce05a111915967eab45d-1620147067945
12747828 .

So not a space problem, and changing the chunk size may improve performance, but that is not a current concern.

Will these “leftovers” ever be cleaned up? Seems costly to leave behind.

Like changing the chunk size, is there a way to change the time alotted for “Processing”.

And lastly, could this “Processing” be a memory operation, and that is what runs out?
I really really hope not, since that would be reserving large amount of memory to be rarely used.

If/when the re-assembly succeeds, those chunks are dumped.
If not - they probably stay… Not sure about automatic cleanup…

Have a look here

No idea really… I suspect this is something S3 specific…

Try increasing the chunk size, double for example… Maybe S3 has a limit on the number of files