4GB+ Uploads to Ceph RGW S3 Failing with 23.0.2

Nextcloud version (eg, 20.0.5): 23.0.2
Operating system and version (eg, Ubuntu 20.04): Docker on Centos 8.1
Apache or nginx version (eg, Apache 2.4.25): nextcloud:fpm-alpine
PHP version (eg, 7.4): nextcloud:fpm-alpine

The issue you are facing:

Uploading files larger than 4GB while using a S3 primary storage fails with error:

Stream from assembly node shorter than expected, got 84309120 bytes, expected 209715200

Storage is a new private CEPH cluster running the latest build of Pacific, with RGW setup across three nodes with Ceph ingress filtering load balancing between them. The issue appears to occur on file recontruction while the multiparts are uploaded into a complete bucket.

RGW S3 connectivity is good with smaller files, and completes just fine with files up to 4GB but any over that and this issue occurs.

I traced down all the timeout errors that would be associated with this between the php-fpm and nginx proxy servers using local storage first. So the following options have been configured in the proxy, web and app instances:

.user.ini

output_buffering=0
upload_max_filesize=16G
post_max_size=16G
max_input_time=3600
max_execution_time=3600
memory_limit=16G

nginx web

fastcgi_read_timeout 600;

nginx proxy

proxy_connect_timeout  600;
proxy_send_timeout     600;
proxy_read_timeout     600;
send_timeout           600;
client_body_timeout    600;

Reading various forum posts and Git Issues, I have tried to adjust the S3MultiPart upload parameters which I can see work, but do not fix this issue. There appear to be posts alluding to adjustments made on the CEPH RGW instances, but they do not actually say what was adjusted to fix this issue.

The output of your Nextcloud log in Admin > Logging:

{"reqId":"ySkgoi6TPzb5oeyt6HUl","level":4,"time":"2022-02-24T20:33:37+00:00","remoteAddr":"192.168.240.1","user":"localadmin","app":"webdav","method":"MOVE","url":"/remote.php/dav/uploads/localadmin/web-file-upload-15ac3cdab922a35e6d48463d90df4c6c-1645734467789/.file","message":"Stream from assembly node shorter than expected, got 84309120 bytes, expected 209715200","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36","version":"23.0.2.1","exception":{"Exception":"Sabre\\DAV\\Exception","Message":"Stream from assembly node shorter than expected, got 84309120 bytes, expected 209715200","Code":0,"Trace":[{"file":"/var/www/html/apps/dav/lib/Connector/Sabre/File.php","line":259,"function":"convertToSabreException","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[{"__class__":"Exception"}]},{"file":"/var/www/html/apps/dav/lib/Connector/Sabre/Directory.php","line":155,"function":"put","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[null]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php","line":307,"function":"createFile","class":"OCA\\DAV\\Connector\\Sabre\\Directory","type":"->","args":["dolby_test_g050_tiff_dv5.mp4",null]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php","line":133,"function":"copyNode","class":"Sabre\\DAV\\Tree","type":"->","args":[{"__class__":"OCA\\DAV\\Upload\\FutureFile"},{"__class__":"OCA\\DAV\\Files\\FilesHome"},"dolby_test_g050_tiff_dv5.mp4"]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php","line":163,"function":"copy","class":"Sabre\\DAV\\Tree","type":"->","args":["uploads/localadmin/web-file-upload-15ac3cdab922a35e6d48463d90df4c6c-1645734467789/.file","files/localadmin/dolby_test_g050_tiff_dv5.mp4"]},{"file":"/var/www/html/apps/dav/lib/Upload/ChunkingPlugin.php","line":94,"function":"move","class":"Sabre\\DAV\\Tree","type":"->","args":["uploads/localadmin/web-file-upload-15ac3cdab922a35e6d48463d90df4c6c-1645734467789/.file","files/localadmin/dolby_test_g050_tiff_dv5.mp4"]},{"file":"/var/www/html/apps/dav/lib/Upload/ChunkingPlugin.php","line":76,"function":"performMove","class":"OCA\\DAV\\Upload\\ChunkingPlugin","type":"->","args":["uploads/localadmin/web-file-upload-15ac3cdab922a35e6d48463d90df4c6c-1645734467789/.file","files/localadmin/dolby_test_g050_tiff_dv5.mp4"]},{"file":"/var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"beforeMove","class":"OCA\\DAV\\Upload\\ChunkingPlugin","type":"->","args":["uploads/localadmin/web-file-upload-15ac3cdab922a35e6d48463d90df4c6c-1645734467789/.file","files/localadmin/dolby_test_g050_tiff_dv5.mp4"]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":603,"function":"emit","class":"Sabre\\DAV\\Server","type":"->","args":["beforeMove",["uploads/localadmin/web-file-upload-15ac3cdab922a35e6d48463d90df4c6c-1645734467789/.file","files/localadmin/dolby_test_g050_tiff_dv5.mp4"]]},{"file":"/var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpMove","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[{"__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->","args":["method:MOVE",[{"__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php","line":253,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[{"__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php","line":321,"function":"start","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/apps/dav/lib/Server.php","line":339,"function":"exec","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/apps/dav/appinfo/v2/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/remote.php","line":166,"args":["/var/www/html/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/html/apps/dav/lib/Connector/Sabre/File.php","Line":676,"Previous":{"Exception":"Exception","Message":"Stream from assembly node shorter than expected, got 84309120 bytes, expected 209715200","Code":0,"Trace":[{"function":"stream_read","class":"OCA\\DAV\\Upload\\AssemblyStream","type":"->","args":[8192]},{"file":"/var/www/html/3rdparty/icewind/streams/src/Wrapper.php","line":55,"function":"fread","args":[null,8192]},{"file":"/var/www/html/lib/private/Files/Stream/HashWrapper.php","line":63,"function":"stream_read","class":"Icewind\\Streams\\Wrapper","type":"->","args":[8192]},{"function":"stream_read","class":"OC\\Files\\Stream\\HashWrapper","type":"->","args":[8192]},{"file":"/var/www/html/3rdparty/icewind/streams/src/Wrapper.php","line":55,"function":"fread","args":[null,8192]},{"file":"/var/www/html/lib/private/Files/Stream/HashWrapper.php","line":63,"function":"stream_read","class":"Icewind\\Streams\\Wrapper","type":"->","args":[8192]},{"function":"stream_read","class":"OC\\Files\\Stream\\HashWrapper","type":"->","args":[8192]},{"file":"/var/www/html/3rdparty/icewind/streams/src/Wrapper.php","line":55,"function":"fread","args":[null,8192]},{"file":"/var/www/html/lib/private/Files/Stream/HashWrapper.php","line":63,"function":"stream_read","class":"Icewind\\Streams\\Wrapper","type":"->","args":[8192]},{"function":"stream_read","class":"OC\\Files\\Stream\\HashWrapper","type":"->","args":[8192]},{"file":"/var/www/html/3rdparty/icewind/streams/src/Wrapper.php","line":55,"function":"fread","args":[null,8192]},{"file":"/var/www/html/3rdparty/icewind/streams/src/CallbackWrapper.php","line":96,"function":"stream_read","class":"Icewind\\Streams\\Wrapper","type":"->","args":[8192]},{"function":"stream_read","class":"Icewind\\Streams\\CallbackWrapper","type":"->","args":[8192]},{"file":"/var/www/html/3rdparty/icewind/streams/src/Wrapper.php","line":55,"function":"fread","args":[null,8192]},{"file":"/var/www/html/3rdparty/icewind/streams/src/CountWrapper.php","line":87,"function":"stream_read","class":"Icewind\\Streams\\Wrapper","type":"->","args":[8192]},{"function":"stream_read","class":"Icewind\\Streams\\CountWrapper","type":"->","args":[8192]},{"file":"/var/www/html/3rdparty/guzzlehttp/psr7/src/Stream.php","line":228,"function":"fread","args":[null,8192]},{"file":"/var/www/html/3rdparty/guzzlehttp/psr7/src/AppendStream.php","line":205,"function":"read","class":"GuzzleHttp\\Psr7\\Stream","type":"->","args":[8192]},{"file":"/var/www/html/3rdparty/guzzlehttp/psr7/src/LimitStream.php","line":152,"function":"read","class":"GuzzleHttp\\Psr7\\AppendStream","type":"->","args":[8192]},{"file":"/var/www/html/3rdparty/aws/aws-sdk-php/src/HashingStream.php","line":38,"function":"read","class":"GuzzleHttp\\Psr7\\LimitStream","type":"->","args":[8192]},{"file":"/var/www/html/3rdparty/guzzlehttp/psr7/src/Utils.php","line":53,"function":"read","class":"Aws\\HashingStream","type":"->","args":[8192]},{"file":"/var/www/html/3rdparty/aws/aws-sdk-php/src/S3/MultipartUploader.php","line":117,"function":"copyToStream","class":"GuzzleHttp\\Psr7\\Utils","type":"::","args":[{"__class__":"Aws\\HashingStream","stream":{"__class__":"GuzzleHttp\\Psr7\\LimitStream","stream":{"__class__":"GuzzleHttp\\Psr7\\AppendStream"}}},{"__class__":"GuzzleHttp\\Psr7\\Stream"}]},{"file":"/var/www/html/3rdparty/aws/aws-sdk-php/src/Multipart/AbstractUploader.php","line":54,"function":"createPart","class":"Aws\\S3\\MultipartUploader","type":"->","args":[false,6]},{"file":"/var/www/html/3rdparty/aws/aws-sdk-php/src/CommandPool.php","line":55,"function":"getUploadCommands","class":"Aws\\Multipart\\AbstractUploader","type":"->","args":[{"__class__":"Closure"}]},{"function":"Aws\\{closure}","class":"Aws\\CommandPool","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/guzzlehttp/promises/src/EachPromise.php","line":212,"function":"next","class":"Generator","type":"->","args":[]},{"file":"/var/www/html/3rdparty/guzzlehttp/promises/src/EachPromise.php","line":238,"function":"advanceIterator","class":"GuzzleHttp\\Promise\\EachPromise","type":"->","args":[]},{"file":"/var/www/html/3rdparty/guzzlehttp/promises/src/EachPromise.php","line":183,"function":"step","class":"GuzzleHttp\\Promise\\EachPromise","type":"->","args":[2]},{"file":"/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php","line":204,"function":"GuzzleHttp\\Promise\\{closure}","class":"GuzzleHttp\\Promise\\EachPromise","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php","line":153,"function":"callHandler","class":"GuzzleHttp\\Promise\\Promise","type":"::","args":[1,"*** sensitive parameter replaced ***",null]},{"file":"/var/www/html/3rdparty/guzzlehttp/promises/src/TaskQueue.php","line":48,"function":"GuzzleHttp\\Promise\\{closure}","class":"GuzzleHttp\\Promise\\Promise","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php","line":158,"function":"run","class":"GuzzleHttp\\Promise\\TaskQueue","type":"->","args":[]},{"file":"/var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php","line":183,"function":"tick","class":"GuzzleHttp\\Handler\\CurlMultiHandler","type":"->","args":[]},{"file":"/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php","line":248,"function":"execute","class":"GuzzleHttp\\Handler\\CurlMultiHandler","type":"->","args":["*** sensitive parameter replaced ***"]},{"file":"/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php","line":224,"function":"invokeWaitFn","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php","line":269,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php","line":226,"function":"invokeWaitList","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php","line":269,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php","line":226,"function":"invokeWaitList","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php","line":62,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/var/www/html/3rdparty/guzzlehttp/promises/src/Coroutine.php","line":67,"function":"wait","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php","line":248,"function":"GuzzleHttp\\Promise\\{closure}","class":"GuzzleHttp\\Promise\\Coroutine","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php","line":224,"function":"invokeWaitFn","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php","line":269,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php","line":226,"function":"invokeWaitList","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php","line":62,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/var/www/html/3rdparty/aws/aws-sdk-php/src/Multipart/AbstractUploadManager.php","line":83,"function":"wait","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"/var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php","line":121,"function":"upload","class":"Aws\\Multipart\\AbstractUploadManager","type":"->","args":[]},{"file":"/var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php","line":154,"function":"writeMultiPart","class":"OC\\Files\\ObjectStore\\S3","type":"->","args":["urn:oid:1395",{"__class__":"GuzzleHttp\\Psr7\\AppendStream"},"video/mp4"]},{"file":"/var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php","line":489,"function":"writeObject","class":"OC\\Files\\ObjectStore\\S3","type":"->","args":["urn:oid:1395",null,"video/mp4"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php","line":647,"function":"writeStream","class":"OC\\Files\\ObjectStore\\ObjectStoreStorage","type":"->","args":["files/dolby_test_g050_tiff_dv5.mp4",null,null]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php","line":647,"function":"writeStream","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["files/dolby_test_g050_tiff_dv5.mp4",null,null]},{"file":"/var/www/html/apps/dav/lib/Connector/Sabre/File.php","line":206,"function":"writeStream","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["files/dolby_test_g050_tiff_dv5.mp4",null]},{"file":"/var/www/html/apps/dav/lib/Connector/Sabre/Directory.php","line":155,"function":"put","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[null]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php","line":307,"function":"createFile","class":"OCA\\DAV\\Connector\\Sabre\\Directory","type":"->","args":["dolby_test_g050_tiff_dv5.mp4",null]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php","line":133,"function":"copyNode","class":"Sabre\\DAV\\Tree","type":"->","args":[{"__class__":"OCA\\DAV\\Upload\\FutureFile"},{"__class__":"OCA\\DAV\\Files\\FilesHome"},"dolby_test_g050_tiff_dv5.mp4"]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php","line":163,"function":"copy","class":"Sabre\\DAV\\Tree","type":"->","args":["uploads/localadmin/web-file-upload-15ac3cdab922a35e6d48463d90df4c6c-1645734467789/.file","files/localadmin/dolby_test_g050_tiff_dv5.mp4"]},{"file":"/var/www/html/apps/dav/lib/Upload/ChunkingPlugin.php","line":94,"function":"move","class":"Sabre\\DAV\\Tree","type":"->","args":["uploads/localadmin/web-file-upload-15ac3cdab922a35e6d48463d90df4c6c-1645734467789/.file","files/localadmin/dolby_test_g050_tiff_dv5.mp4"]},{"file":"/var/www/html/apps/dav/lib/Upload/ChunkingPlugin.php","line":76,"function":"performMove","class":"OCA\\DAV\\Upload\\ChunkingPlugin","type":"->","args":["uploads/localadmin/web-file-upload-15ac3cdab922a35e6d48463d90df4c6c-1645734467789/.file","files/localadmin/dolby_test_g050_tiff_dv5.mp4"]},{"file":"/var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"beforeMove","class":"OCA\\DAV\\Upload\\ChunkingPlugin","type":"->","args":["uploads/localadmin/web-file-upload-15ac3cdab922a35e6d48463d90df4c6c-1645734467789/.file","files/localadmin/dolby_test_g050_tiff_dv5.mp4"]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":603,"function":"emit","class":"Sabre\\DAV\\Server","type":"->","args":["beforeMove",["uploads/localadmin/web-file-upload-15ac3cdab922a35e6d48463d90df4c6c-1645734467789/.file","files/localadmin/dolby_test_g050_tiff_dv5.mp4"]]},{"file":"/var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpMove","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[{"__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->","args":["method:MOVE",[{"__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php","line":253,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[{"__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php","line":321,"function":"start","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/apps/dav/lib/Server.php","line":339,"function":"exec","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/apps/dav/appinfo/v2/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/remote.php","line":166,"args":["/var/www/html/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/html/apps/dav/lib/Upload/AssemblyStream.php","Line":159},"CustomMessage":"--"}}

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'apps_paths' => 
  array (
    0 => 
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 => 
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' => 
  array (
    'host' => 'redis',
    'password' => '',
    'port' => 6379,
  ),
  'overwriteprotocol' => 'https',
  'objectstore' => 
  array (
    'class' => '\\OC\\Files\\ObjectStore\\S3',
    'arguments' => 
    array (
      'bucket' => 'postcloud-data',
      'key' => ''*********************************',
      'secret' => '*********************************',
      'region' => 'default',
      'hostname' => 'neptune.pwny.com',
      'port' => '20443',
      'objectPrefix' => 'urn:oid:',
      'autocreate' => false,
      'use_ssl' => true,
      'use_path_style' => true,
      'legacy_auth' => false,
    ),
  ),
  'instanceid' => 'occy5uq57kw5',
  'passwordsalt' => ''*********************************',
  'secret' => ''*********************************',
  'trusted_domains' => 
  array (
    0 => 'postcloud.pwny.com',
  ),
  'datadirectory' => '/var/www/html/data',
  'dbtype' => 'pgsql',
  'version' => '23.0.2.1',
  'overwrite.cli.url' => 'https://postcloud.pwny.com',
  'dbname' => 'nextcloud',
  'dbhost' => 'db',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'oc_localadmin',
  'dbpassword' => 'ItphfJ5w56NkDAaoTzYEL5qoEzQw1e',
  'installed' => true,
);

The output of your Apache/nginx/system log in /var/log/____:

192.168.240.1 - - [24/Feb/2022:20:31:15 +0000] "PUT /remote.php/dav/uploads/localadmin/web-file-upload-15ac3cdab922a35e6d48463d90df4c6c-1645734467789/3774873600 HTTP/1.1" 201 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36" "10.4.2.92"
192.168.240.1 - - [24/Feb/2022:20:31:26 +0000] "PUT /remote.php/dav/uploads/localadmin/web-file-upload-15ac3cdab922a35e6d48463d90df4c6c-1645734467789/3984588800 HTTP/1.1" 201 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36" "10.4.2.92"
192.168.240.1 - - [24/Feb/2022:20:31:37 +0000] "PUT /remote.php/dav/uploads/localadmin/web-file-upload-15ac3cdab922a35e6d48463d90df4c6c-1645734467789/4194304000 HTTP/1.1" 201 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36" "10.4.2.92"
192.168.240.1 - - [24/Feb/2022:20:31:43 +0000] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36" "10.4.2.92"
192.168.240.1 - - [24/Feb/2022:20:31:48 +0000] "PUT /remote.php/dav/uploads/localadmin/web-file-upload-15ac3cdab922a35e6d48463d90df4c6c-1645734467789/4404019200 HTTP/1.1" 201 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36" "10.4.2.92"
192.168.240.1 - - [24/Feb/2022:20:31:59 +0000] "PUT /remote.php/dav/uploads/localadmin/web-file-upload-15ac3cdab922a35e6d48463d90df4c6c-1645734467789/4613734400 HTTP/1.1" 201 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36" "10.4.2.92"
192.168.240.1 - - [24/Feb/2022:20:32:10 +0000] "PUT /remote.php/dav/uploads/localadmin/web-file-upload-15ac3cdab922a35e6d48463d90df4c6c-1645734467789/4823449600 HTTP/1.1" 201 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36" "10.4.2.92"
192.168.240.1 - - [24/Feb/2022:20:32:13 +0000] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36" "10.4.2.92"
192.168.240.1 - - [24/Feb/2022:20:32:21 +0000] "PUT /remote.php/dav/uploads/localadmin/web-file-upload-15ac3cdab922a35e6d48463d90df4c6c-1645734467789/5033164800 HTTP/1.1" 201 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36" "10.4.2.92"
192.168.240.1 - - [24/Feb/2022:20:32:32 +0000] "PUT /remote.php/dav/uploads/localadmin/web-file-upload-15ac3cdab922a35e6d48463d90df4c6c-1645734467789/5242880000 HTTP/1.1" 201 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36" "10.4.2.92"
192.168.240.1 - - [24/Feb/2022:20:32:41 +0000] "PUT /remote.php/dav/uploads/localadmin/web-file-upload-15ac3cdab922a35e6d48463d90df4c6c-1645734467789/5452595200 HTTP/1.1" 201 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36" "10.4.2.92"
192.168.240.1 - - [24/Feb/2022:20:32:41 +0000] "GET /index.php/apps/files/ajax/getstoragestats.php?dir=%2F HTTP/1.1" 200 199 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36" "10.4.2.92"
192.168.240.1 - - [24/Feb/2022:20:32:43 +0000] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36" "10.4.2.92"
192.168.240.1 - - [24/Feb/2022:20:32:43 +0000] "PUT /apps/user_status/heartbeat HTTP/1.1" 200 118 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36" "10.4.2.92"
192.168.240.1 - - [24/Feb/2022:20:32:43 +0000] "GET /index.php/apps/files/ajax/getstoragestats.php?dir=%2F HTTP/1.1" 200 199 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36" "10.4.2.92"
192.168.240.1 - - [24/Feb/2022:20:33:13 +0000] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36" "10.4.2.92"
2022/02/24 20:33:37 [error] 34#34: *106 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Uncaught TypeError: hash_final(): Argument #1 ($context) must be a valid Hash Context resource in /var/www/html/lib/private/Files/Stream/HashWrapper.php:70
Stack trace:
#0 /var/www/html/lib/private/Files/Stream/HashWrapper.php(70): hash_final(Object(HashContext))
#1 [internal function]: OC\Files\Stream\HashWrapper->stream_close()
#2 /var/www/html/3rdparty/icewind/streams/src/Wrapper.php(96): fclose(Resource id #30)
#3 /var/www/html/3rdparty/icewind/streams/src/CallbackWrapper.php(117): Icewind\Streams\Wrapper->stream_close()
#4 [internal function]: Icewind\Streams\CallbackWrapper->stream_close()
#5 /var/www/html/3rdparty/icewind/streams/src/Wrapper.php(96): fclose(Resource id #33)
#6 /var/www/html/3rdparty/icewind/streams/src/CountWrapper.php(99): Icewind\Streams\Wrapper->stream_close()
#7 [internal function]: Icewind\Streams\CountWrapper->stream_close()
#8 /var/www/html/3rdparty/guzzlehttp/psr7/src/Stream.php(108): fclose(Resource id #37)
#9 /var/www/html/3rdparty/guzzlehttp/psr7/src/Stream.php(74): GuzzleHttp\Psr7\Stream->close()
#10 [internal function]: GuzzleHttp\Psr7\Stream->__destruct()
#11 {main}
  thrown in /var/www/html/lib/private/Files/Stream/HashWrapper.php on line 70PHP message: PHP Fatal error:  Uncaught TypeError: hash_final(): Argument #1 ($context) must be a valid Hash Context resource in /var/www/html/lib/private/Files/Stream/HashWrapper.php:70
Stack trace:
#0 /var/www/html/lib/private/Files/Stream/HashWrapper.php(70): hash_final(Object(HashContext))
#1 [internal function]: OC\Files\Stream\HashWrapper->stream_close()
192.168.240.1 - - [24/Feb/2022:20:33:37 +0000] "MOVE /remote.php/dav/uploads/localadmin/web-file-upload-15ac3cdab922a35e6d48463d90df4c6c-1645734467789/.file HTTP/1.1" 500 282 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36" "10.4.2.92"
#2 {main}
  thrown in /var/www/html/lib/private/Files/Stream/HashWrapper.php on line 70" while reading upstream, client: 192.168.240.1, server: , request: "MOVE /remote.php/dav/uploads/localadmin/web-file-upload-15ac3cdab922a35e6d48463d90df4c6c-1645734467789/.file HTTP/1.1", upstream: "fastcgi://172.18.0.5:9000", host: "postcloud.pwny.com"
192.168.240.1 - - [24/Feb/2022:20:33:38 +0000] "GET /index.php/apps/files/ajax/getstoragestats.php?dir=%2F HTTP/1.1" 200 199 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36" "10.4.2.92"
192.168.240.1 - - [24/Feb/2022:20:33:43 +0000] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36" "10.4.2.92"

Exact same Stack Trace for me for files > 4 GB using:

  • Docker on Synology DMS
  • Nextcloud v23.0.4
  • Wasabi S3
  • Same .usi.ini settings

I’m not sure if it’s the same issue, but there is some documentation for large file uploads:
https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/big_file_upload_configuration.html?highlight=large%20files