I’m use object storage as the primary storage, and during using I met a problem. When the uploaded file is large(the test file size is 5GB), it can’t transfer from my server to object storage server.
tflidd
October 25, 2020, 4:07pm
#2
Some user found a workaround for an older version and a different storage:
I could resolve this issue as well by patching /lib/private/Files/ObjectStore/S3ObjectTrait.php and lowering S3_UPLOAD_PART_SIZE from 512MB to 256MB.
It might make sense to make this a configuration option? I guess the optimal size here is different depending on the S3 backend used? (e.g., I’m not using AWS S3 but DigitalOcean Spaces)
There was a bug report last year as well:
opened 12:48PM - 30 Oct 19 UTC
closed 01:37AM - 01 Nov 19 UTC
Preface
I understand S3 as a primary storage is still technically considered an enterprise feature. I have a specific use case for...
0. Needs triage
bug
couple of experiences from dealing with this issue:
check if the s3 bucket is using ipv4/6 dualstack, in which case connecting via ipv6 might be slower with a much bigger latency
in Nextcloud 21.0.2, there are three parameters
that you can play with inside
to adjust the nextcloud upload chunking to resolve the latency issue
As I had this issue as well just now, I thought to share the solution that worked for me:
I had the same problem but with my Nextcloud version (23) I had to find a different solution. I found the eventual solution thanks to this comment: S3 default upload part_size set to 500MB · Issue #24390 · nextcloud/server · GitHub
I needed to edit two different files to fix this:
In file:
/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/S3/MultipartUploader.php
Have the following settings:
const PART_MIN_SIZE = 4194304;
const PART_MAX_SIZE = 1073741824;
const PART_MAX_NUM = 1000…
