Unable to upload large files to external storage, "no space left on device"

Support intro

Using a hosted solution the upload proceeds until the progress bar is filled, then switches to “processing files …” which never ends.

file of 1.4GB works, a file of 1.7GB never uploads.

Nextcloud version (eg, 20.0.5): 21.0.1
Operating system and version (eg, Ubuntu 20.04): Linux 3.10.0-962 el7.x86_64
Apache or nginx version (eg, Apache 2.4.25): Apache litespeed
PHP version (eg, 7.4): 7.4.15

The issue you are facing:
upload of files over 1.5GB to external s3 storage appear to fail after upload. The progress bar fills and then switches to “Processing files …”. That never changes.

Confirmed large files support on external storage by using rclone.

I see “no space left on device”, but what device? All have plenty of space.

[w ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 126G 0 126G 0% /dev
/dev/sda4 58T 38T 18T 69% /opt
/dev/loop0 3.9G 222M 3.4G 7% /tmp
[w ~]$

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:

  1. connect external s3 storage
  2. upload 1.7GB file

The output of your Nextcloud log in Admin > Logging:

{"reqId":"YJF-eRWzRIv9VFkPWMl7JAACFRY","level":3,"time":"2021-05-04T17:09:10+00:00","remoteAddr":"A.B.C.D","user":"X","app":"PHP","method":"MOVE","url":"/remote.php/dav/uploads/X/web-file-upload-1d0247c2d6e3ce05a111915967eab45d-1620147067945/.file","message":{"Exception":"Error","Message":"fwrite(): write of 8192 bytes failed with errno=28 No space left on device at /public_html/3rdparty/guzzlehttp/psr7/src/Stream.php#247","Code":0,"Trace":[{"function":"onError","class":"OC\\Log\\ErrorHandler","type":"::","args":[8,"fwrite(): write of 8192 bytes failed with errno=28 No space left on device","/public_html/3rdparty/guzzlehttp/psr7/src/Stream.php",247,{"string":null}]},{"file":"/public_html/3rdparty/guzzlehttp/psr7/src/Stream.php","line":247,"function":"fwrite","args":[null,null]},{"file":"/public_html/3rdparty/guzzlehttp/psr7/src/Utils.php","line":53,"function":"write","class":"GuzzleHttp\\Psr7\\Stream","type":"->","args":[null]},{"file":"/public_html/3rdparty/guzzlehttp/psr7/src/functions.php","line":216,"function":"copyToStream","class":"GuzzleHttp\\Psr7\\Utils","type":"::","args":[{"stream":{"stream":{"__class__":"GuzzleHttp\\Psr7\\Stream"},"__class__":"GuzzleHttp\\Psr7\\LimitStream"},"__class__":"Aws\\HashingStream"},{"__class__":"GuzzleHttp\\Psr7\\Stream"},-1]},{"file":"/public_html/3rdparty/aws/aws-sdk-php/src/S3/MultipartUploader.php","line":117,"function":"GuzzleHttp\\Psr7\\copy_to_stream","args":[{"stream":{"stream":{"__class__":"GuzzleHttp\\Psr7\\Stream"},"__class__":"GuzzleHttp\\Psr7\\LimitStream"},"__class__":"Aws\\HashingStream"},{"__class__":"GuzzleHttp\\Psr7\\Stream"}]},{"file":"/public_html/3rdparty/aws/aws-sdk-php/src/Multipart/AbstractUploader.php","line":54,"function":"createPart","class":"Aws\\S3\\MultipartUploader","type":"->","args":[false,4]},{"file":"/public_html/3rdparty/aws/aws-sdk-php/src/CommandPool.php","line":55,"function":"getUploadCommands","class":"Aws\\Multipart\\AbstractUploader","type":"->","args":[{"__class__":"Closure"}]},{"function":"Aws\\{closure}","class":"Aws\\CommandPool","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/EachPromise.php","line":211,"function":"next","class":"Generator","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/EachPromise.php","line":153,"function":"advanceIterator","class":"GuzzleHttp\\Promise\\EachPromise","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/EachPromise.php","line":81,"function":"refillPending","class":"GuzzleHttp\\Promise\\EachPromise","type":"->","args":[]},{"file":"/public_html/3rdparty/aws/aws-sdk-php/src/CommandPool.php","line":79,"function":"promise","class":"GuzzleHttp\\Promise\\EachPromise","type":"->","args":[]},{"file":"/public_html/3rdparty/aws/aws-sdk-php/src/Multipart/AbstractUploadManager.php","line":132,"function":"promise","class":"Aws\\CommandPool","type":"->","args":[]},{"function":"Aws\\Multipart\\{closure}","class":"Aws\\Multipart\\AbstractUploadManager","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Coroutine.php","line":140,"function":"send","class":"Generator","type":"->","args":[{"__class__":"Aws\\Result"}]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":204,"function":"_handleSuccess","class":"GuzzleHttp\\Promise\\Coroutine","type":"->","args":[{"__class__":"Aws\\Result"}]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":153,"function":"callHandler","class":"GuzzleHttp\\Promise\\Promise","type":"::","args":[1,{"__class__":"Aws\\Result"},null]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/TaskQueue.php","line":48,"function":"GuzzleHttp\\Promise\\{closure}","class":"GuzzleHttp\\Promise\\Promise","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/public_html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php","line":158,"function":"run","class":"GuzzleHttp\\Promise\\TaskQueue","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php","line":183,"function":"tick","class":"GuzzleHttp\\Handler\\CurlMultiHandler","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":248,"function":"execute","class":"GuzzleHttp\\Handler\\CurlMultiHandler","type":"->","args":["*** sensitive parameter replaced ***"]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":224,"function":"invokeWaitFn","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":269,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":226,"function":"invokeWaitList","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":269,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":226,"function":"invokeWaitList","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":62,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Coroutine.php","line":67,"function":"wait","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":248,"function":"GuzzleHttp\\Promise\\{closure}","class":"GuzzleHttp\\Promise\\Coroutine","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":224,"function":"invokeWaitFn","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":269,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":226,"function":"invokeWaitList","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/public_html/3rdparty/guzzlehttp/promises/src/Promise.php","line":62,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/public_html/3rdparty/aws/aws-sdk-php/src/Multipart/AbstractUploadManager.php","line":83,"function":"wait","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/public_html/lib/private/Files/ObjectStore/S3ObjectTrait.php","line":97,"function":"upload","class":"Aws\\Multipart\\AbstractUploadManager","type":"->","args":[]},{"file":"/public_html/apps/files_external/lib/Lib/Storage/AmazonS3.php","line":687,"function":"writeObject","class":"OCA\\Files_External\\Lib\\Storage\\AmazonS3","type":"->","args":["X9.MP4",null]},{"file":"/public_html/apps/files_external/lib/Lib/Storage/AmazonS3.php","line":535,"function":"writeBack","class":"OCA\\Files_External\\Lib\\Storage\\AmazonS3","type":"->","args":["/tmp/oc_tmp_QcNujw","X9.MP4"]},{"function":"OCA\\Files_External\\Lib\\Storage\\{closure}","class":"OCA\\Files_External\\Lib\\Storage\\AmazonS3","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/public_html/apps/files_external/3rdparty/icewind/streams/src/CallbackWrapper.php","line":119,"function":"call_user_func","args":[{"__class__":"Closure"}]},{"function":"stream_close","class":"Icewind\\Streams\\CallbackWrapper","type":"->","args":[]},{"file":"/public_html/lib/private/Files/Storage/Common.php","line":871,"function":"fclose","args":[null]},{"file":"/public_html/lib/private/Files/Storage/Wrapper/Wrapper.php","line":629,"function":"writeStream","class":"OC\\Files\\Storage\\Common","type":"->","args":["X9.MP4",null,null]},{"file":"/public_html/lib/private/Files/Storage/Wrapper/Wrapper.php","line":629,"function":"writeStream","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["X9.MP4",null,null]},{"file":"/public_html/lib/private/Files/Storage/Wrapper/Wrapper.php","line":629,"function":"writeStream","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["X9.MP4",null,null]},{"file":"/public_html/apps/dav/lib/Connector/Sabre/File.php","line":207,"function":"writeStream","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["X9.MP4",null]},{"file":"/public_html/apps/dav/lib/Connector/Sabre/Directory.php","line":156,"function":"put","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[null]},{"file":"/public_html/3rdparty/sabre/dav/lib/DAV/Tree.php","line":305,"function":"createFile","class":"OCA\\DAV\\Connector\\Sabre\\Directory","type":"->","args":["X9.MP4",null]},{"file":"/public_html/3rdparty/sabre/dav/lib/DAV/Tree.php","line":133,"function":"copyNode","class":"Sabre\\DAV\\Tree","type":"->","args":[{"__class__":"OCA\\DAV\\Upload\\FutureFile"},{"__class__":"OCA\\DAV\\Connector\\Sabre\\Directory"},"X9.MP4"]},{"file":"/public_html/3rdparty/sabre/dav/lib/DAV/Tree.php","line":163,"function":"copy","class":"Sabre\\DAV\\Tree","type":"->","args":["uploads/X/web-file-upload-1d0247c2d6e3ce05a111915967eab45d-1620147067945/.file","files/X/X/X9.MP4"]},{"file":"/public_html/apps/dav/lib/Upload/ChunkingPlugin.php","line":95,"function":"move","class":"Sabre\\DAV\\Tree","type":"->","args":["uploads/X/web-file-upload-1d0247c2d6e3ce05a111915967eab45d-1620147067945/.file","files/X/X/X9.MP4"]},{"file":"/public_html/apps/dav/lib/Upload/ChunkingPlugin.php","line":77,"function":"performMove","class":"OCA\\DAV\\Upload\\ChunkingPlugin","type":"->","args":["uploads/X/web-file-upload-1d0247c2d6e3ce05a111915967eab45d-1620147067945/.file","files/X/X/X9.MP4"]},{"file":"/public_html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"beforeMove","class":"OCA\\DAV\\Upload\\ChunkingPlugin","type":"->","args":["uploads/X/web-file-upload-1d0247c2d6e3ce05a111915967eab45d-1620147067945/.file","files/X/X/X9.MP4"]},{"file":"/public_html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":603,"function":"emit","class":"Sabre\\DAV\\Server","type":"->","args":["beforeMove",["uploads/X/web-file-upload-1d0247c2d6e3ce05a111915967eab45d-1620147067945/.file","files/X/X/X9.MP4"]]},{"file":"/public_html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpMove","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[{"__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/public_html/3rdparty/sabre/dav/lib/DAV/Server.php","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->","args":["method:MOVE",[{"__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"/public_html/3rdparty/sabre/dav/lib/DAV/Server.php","line":253,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[{"__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/public_html/3rdparty/sabre/dav/lib/DAV/Server.php","line":321,"function":"start","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/public_html/apps/dav/lib/Server.php","line":332,"function":"exec","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/public_html/apps/dav/appinfo/v2/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server","type":"->","args":[]},{"file":"/public_html/remote.php","line":167,"args":["/public_html/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/public_html/lib/private/Log/ErrorHandler.php","Line":92,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36","version":"21.0.1.1","id":"609182269c140"}

I see no space left on device, which device?

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

{
    "instanceid": "***REMOVED SENSITIVE VALUE***",
    "passwordsalt": "***REMOVED SENSITIVE VALUE***",
    "secret": "***REMOVED SENSITIVE VALUE***",
    "trusted_domains": [
        "***REMOVED SENSITIVE VALUE***"
    ],
    "datadirectory": "***REMOVED SENSITIVE VALUE***",
    "dbtype": "mysql",
    "version": "21.0.1.1",
    "overwrite.cli.url": "https:\/\/***REMOVED SENSITIVE VALUE***",
    "dbname": "***REMOVED SENSITIVE VALUE***",
    "dbhost": "***REMOVED SENSITIVE VALUE***",
    "dbport": "",
    "dbtableprefix": "oc_",
    "dbuser": "***REMOVED SENSITIVE VALUE***",
    "dbpassword": "***REMOVED SENSITIVE VALUE***",
    "installed": true
}

The output of your Apache/nginx/system log in /var/log/____:

A.B.C.D - - [04/May/2021:13:09:01 -0400] "POST /index.php/apps/text/session/sync HTTP/2.0" 200 218
A.B.C.D - - [04/May/2021:13:09:05 -0400] "POST /index.php/apps/text/public/session/sync HTTP/2.0" 200 229
A.B.C.D - - [04/May/2021:13:09:06 -0400] "POST /index.php/apps/text/session/sync HTTP/2.0" 200 218
A.B.C.D - - [04/May/2021:13:09:11 -0400] "POST /index.php/apps/text/session/sync HTTP/2.0" 200 218
A.B.C.D - - [04/May/2021:13:09:11 -0400] "POST /index.php/apps/text/public/session/sync HTTP/2.0" 200 229
A.B.C.D - - [04/May/2021:13:09:16 -0400] "POST /index.php/apps/text/session/sync HTTP/2.0" 200 218
A.B.C.D - - [04/May/2021:13:09:17 -0400] "POST /index.php/apps/text/public/session/sync HTTP/2.0" 200 229
A.B.C.D - - [04/May/2021:13:09:22 -0400] "POST /index.php/apps/text/session/sync HTTP/2.0" 200 218
A.B.C.D - - [04/May/2021:13:09:24 -0400] "POST /index.php/apps/text/public/session/sync HTTP/2.0" 200 229
A.B.C.D - - [04/May/2021:13:09:25 -0400] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/2.0" 304 -
A.B.C.D - - [04/May/2021:13:09:27 -0400] "POST /index.php/apps/text/session/sync HTTP/2.0" 200 218
A.B.C.D - - [04/May/2021:13:09:30 -0400] "POST /index.php/apps/text/public/session/sync HTTP/2.0" 200 229
W.X.Y.Z - - [04/May/2021:13:09:30 -0400] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/2.0" 304 -
A.B.C.D - - [04/May/2021:13:09:32 -0400] "POST /index.php/apps/text/session/sync HTTP/2.0" 200 218
A.B.C.D - - [04/May/2021:13:09:36 -0400] "POST /index.php/apps/text/public/session/sync HTTP/2.0" 200 229
A.B.C.D - - [04/May/2021:13:09:36 -0400] "PUT /index.php/apps/user_status/heartbeat HTTP/2.0" 204 -
A.B.C.D - - [04/May/2021:13:09:37 -0400] "GET /ocs/v2.php/apps/user_status/api/v1/user_status HTTP/2.0" 200 149
A.B.C.D - - [04/May/2021:13:09:37 -0400] "POST /index.php/apps/text/session/sync HTTP/2.0" 200 218
A.B.C.D - - [04/May/2021:13:09:42 -0400] "POST /index.php/apps/text/public/session/sync HTTP/2.0" 200 229
A.B.C.D - - [04/May/2021:13:09:42 -0400] "POST /index.php/apps/text/session/sync HTTP/2.0" 200 218
A.B.C.D - - [04/May/2021:13:09:48 -0400] "POST /index.php/apps/text/public/session/sync HTTP/2.0" 200 229
A.B.C.D - - [04/May/2021:13:09:48 -0400] "POST /index.php/apps/text/session/sync HTTP/2.0" 200 218
A.B.C.D - - [04/May/2021:13:09:53 -0400] "POST /index.php/apps/text/session/sync HTTP/2.0" 200 218
A.B.C.D - - [04/May/2021:13:09:54 -0400] "POST /index.php/apps/text/public/session/sync HTTP/2.0" 200 229
A.B.C.D - - [04/May/2021:13:09:55 -0400] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/2.0" 304 -
W.X.Y.Z - - [04/May/2021:13:09:57 -0400] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/2.0" 304 -
A.B.C.D - - [04/May/2021:13:09:58 -0400] "POST /index.php/apps/text/session/sync HTTP/2.0" 200 219

It seems silly, but since the devices I can query show plenty of space, is this an in memory operation? And more RAM is required? That seems problematic since occasional large files means large amounts of memory need to be provisioned, an expensive solution. These 1.5GB files should be multi part in the s3 world, but I am unsure how NextCloud behaves, since it is not s3 for local storage.