Can't upload Large File size (>20GB) to S3 in AWS

,
Support intro

Sorry to hear you’re facing problems :slightly_frowning_face:

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can :heart:

When we tried to put a file larger than 20GB into our NextCloud (EC2), it couldn’t be uploaded.
This symptom occurs once every three times.

Before our test, we had preliminary fixed a constants in “S3ObjectTrait.php” like it below.

S3_UPLOAD_PART_SIZE from 500MB to 250MB

(ref: S3 random storage problem on large files)

Could you tell us how to improve our NextCloud??

Nextcloud version : 17.0.2
Operating system and version : CentOS7.9
Apache or nginx version : nginx/1.18.0
PHP version : 7.3.25

The output of your Nextcloud log in Admin > Logging:

An exception occurred while uploading parts to a multipart upload. The following parts had errors: - Part 2: Error executing "UploadPart" on "http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=2&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--"; AWS HTTP error: Client error: `PUT http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=2&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--` resulted in a `400 Bad Request` response: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server w (truncated...) RequestTimeout (client): Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed. - <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>AD206973CF62494A</RequestId><HostId>ruro8rnjqtL5M8wVBd9TZGiMKDpbnp70HA7inoVAduBfTaAfOmZjxI+hPSz4OT6q7bzvkzt2chM=</HostId></Error> - Part 6: Error executing "UploadPart" on "http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=6&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--"; AWS HTTP error: Client error: `PUT http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=6&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--` resulted in a `400 Bad Request` response: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server w (truncated...) RequestTimeout (client): Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed. - <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>3B0F82BABDB02A3D</RequestId><HostId>/df791IeiCFxRu5KeZZ3fhkJN+sOWt9VYSTRlE2WZ69kLTzwptsmLGEHxBazAYo0XSYG3fPbsls=</HostId></Error> - Part 7: Error executing "UploadPart" on "http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=7&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--"; AWS HTTP error: Client error: `PUT http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=7&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--` resulted in a `400 Bad Request` response: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server w (truncated...) RequestTimeout (client): Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed. - <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>337A854105540C86</RequestId><HostId>t4uQZ/dYVzBzIUItWfk8WzBiYdbs4FyqNJaBP0PVIPBE32ePVS+BhdavppAGia0TslMwh/i8KMU=</HostId></Error> - Part 8: Error executing "UploadPart" on "http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=8&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--"; AWS HTTP error: Client error: `PUT http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=8&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--` resulted in a `400 Bad Request` response: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server w (truncated...) RequestTimeout (client): Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed. - <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>7FF1B038ECBF669D</RequestId><HostId>YgF/dk/25eUd4HRdOS+dioDKnmSH9UdkRJnJtVcSe2GXPcXU5Uaxf/7Hii1OK4L0lUEQuql93A0=</HostId></Error> - Part 9: Error executing "UploadPart" on "http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=9&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--"; AWS HTTP error: Client error: `PUT http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=9&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--` resulted in a `400 Bad Request` response: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server w (truncated...) RequestTimeout (client): Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed. - <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>C2B90603BA06BCB5</RequestId><HostId>tCGyxbh7LOWxN7Rc49fVXKPXlVzMp2ZO9jTZBnjbzE3q3fP2ssVwwbikL+RK0wZt0c+iXuUSP7M=</HostId></Error> - Part 10: Error executing "UploadPart" on "http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=10&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--"; AWS HTTP error: Client error: `PUT http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=10&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--` resulted in a `400 Bad Request` response: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server w (truncated...) RequestTimeout (client): Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed. - <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>16B98DD14AB94B46</RequestId><HostId>5qldJat3Pw86CZSMU6cNzo420XaxKHqK8bIDSQmusRH2wJ5vY8Aue65RbPnahrMT8/NkBh5BT9s=</HostId></Error> - Part 11: Error executing "UploadPart" on "http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=11&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--"; AWS HTTP error: Client error: `PUT http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=11&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--` resulted in a `400 Bad Request` response: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server w (truncated...) RequestTimeout (client): Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed. - <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>602337832FB0BE0F</RequestId><HostId>l2GA+sIh866adsnoZ3CyjX/7D2VUuOeRiyGW7mKrmxQsN1+g0DpUqZNlskp00m/J5mG64qce29U=</HostId></Error> - Part 12: Error executing "UploadPart" on "http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=12&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--"; AWS HTTP error: Client error: `PUT http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=12&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--` resulted in a `400 Bad Request` response: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server w (truncated...) RequestTimeout (client): Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed. - <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>9BEFD72A1E423C77</RequestId><HostId>YOq4u2i0d9WFniabjFLF38IRqyTTnk/vqwQ0U16256FIxejVo49FnB2mfGPmrspsBPWVAELicKA=</HostId></Error> - Part 13: Error executing "UploadPart" on "http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=13&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--"; AWS HTTP error: cURL error 55: Send failure: Connection reset by peer (see http://curl.haxx.se/libcurl/c/libcurl-errors.html) (server): 100 Continue - - Part 14: Error executing "UploadPart" on "http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=14&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--"; AWS HTTP error: Client error: `PUT http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=14&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--` resulted in a `400 Bad Request` response: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server w (truncated...) RequestTimeout (client): Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed. - <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>63B1DA6111919E9C</RequestId><HostId>ta4RmQ5bl7eTeoL+tA6TXMrKzDAyMDS2+xoweQwGx+00yuObGfX6UvomjzNShetIHSt5YYREynQ=</HostId></Error> - Part 15: Error executing "UploadPart" on "http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=15&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--"; AWS HTTP error: Client error: `PUT http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=15&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--` resulted in a `400 Bad Request` response: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server w (truncated...) RequestTimeout (client): Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed. - <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>443903EF20C0F971</RequestId><HostId>SidWcT576vMywFQ2VK4m6q8TBZb1jbAsKEiRKYQXoJbVoljEYf6q87o/fBbiN3ba4tY0ePfidP4=</HostId></Error> - Part 16: Error executing "UploadPart" on "http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=16&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--"; AWS HTTP error: Client error: `PUT http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=16&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--` resulted in a `400 Bad Request` response: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server w (truncated...) RequestTimeout (client): Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed. - <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>D574680F4A397989</RequestId><HostId>aPrv7A0c64YxOoiKlDNT0DL4Y8TrMyIovv3shQTWlrt0IuPqiyN5ldU9UhwNfkBE3LkCk2DdCwk=</HostId></Error> - Part 22: Error executing "UploadPart" on "http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=22&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--"; AWS HTTP error: Client error: `PUT http://{{OUR NEXTCLOUD}}.s3.ap-northeast-1.amazonaws.com/urn%3Aoid%3A18049?partNumber=22&uploadId=S9O73M4_0HGmWEgltBEs5tlZ1KjotpCD9Z6onHZ8ivxBurGHQ0X0G2SAtkhelwhrpnZUH7Gfq3bxoBDvUlxSWO5gRQvvXhnR_YBIzmhf49XbQeFlWQgwWB53pEy50BWd5P30d6hiwGWlruD9E33k3Q--` resulted in a `400 Bad Request` response: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server w (truncated...) RequestTimeout (client): Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed. - <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>7J1X7H8M6Z5G8R7R</RequestId><HostId>BEm5nWGjknnFV1ceTfo/4ECHzOhQjunpH/jDQZEJsYr/w3T8UNn/dSTwcmOEOfX+6zjn0f0gLfs=</HostId></Error>

Thank you for your attention at advance.

referring to your posted errormessage it could hasppen due to a time-out problem of the connection.

We appreciate to you, Jimmy for giving your comment.
We tried to get a good confirmation against time-out setting, but our problem has not been cleared so far.
We keep to check all conf files and those settings once more.

Thank you, again.