Large files cannot be uploaded when object storage is used as the primary storage

[Hydrostic]

I’m use object storage as the primary storage, and during using I met a problem. When the uploaded file is large(the test file size is 5GB), it can’t transfer from my server to object storage server.
Here are the logs:

[tflidd]

Some user found a workaround for an older version and a different storage:

Primary Storage S3, Files large more than 4GB

I could resolve this issue as well by patching /lib/private/Files/ObjectStore/S3ObjectTrait.php and lowering S3_UPLOAD_PART_SIZE from 512MB to 256MB. It might make sense to make this a configuration option? I guess the optimal size here is different depending on the S3 backend used? (e.g., I’m not using AWS S3 but DigitalOcean Spaces)

There was a bug report last year as well:

Large files fail to upload & written to bucket with no metadata in the database to link them to anything

[carpediemmlf]

couple of experiences from dealing with this issue:

1. check if the s3 bucket is using ipv4/6 dualstack, in which case connecting via ipv6 might be slower with a much bigger latency

2. in Nextcloud 21.0.2, there are three parameters
   PART_MIN_SIZE
   PART_MAX_SIZE
   PART_MAX_NUM

that you can play with inside
nextcloud/3rdparty/aws/aws-sdk-php/src/S3/MultipartUploader.php

to adjust the nextcloud upload chunking to resolve the latency issue