Enable chunking v2 for Object Storage S3

Labouledefortboyard · April 5, 2024, 9:22am

Nextcloud version (eg, 20.0.5): 28.0.3
Operating system and version (eg, Ubuntu 20.04): Ubuntu server 22.0.4
Apache or nginx version (eg, Apache 2.4.25): Apache 2.4.52
PHP version (eg, 7.4): 8.1

The issue you are facing:

Hello everyone,

I’m still a neophyte on Nextcloud, so excuse me if I say aberrations or mistakes.

I have a next cloud with Primary stroage: Object storage S3.

For storage that can be expanded as required.

When I upload large files (>20 GB) I get the message “unknow error”.

I’ve noticed that my local system storage (which is 20 GB max) is 100% full when uploading, and that the message appears at this point.

From the research I’ve been able to do, I’ve deduced that this is one of the problems with chunking, which assembles the file locally into a temporary file before sending it to the S3 object storage.

My question is whether my resonance is good? And if so, is there any way of solving this problem without having to increase my server’s local storage (for me, this solution is not scalable)?

I’ve seen that version 2 of chunking on nextcloud allows this without having to first assemble the file on the local server. With the “MultipartUpload for uploading chunks to s3” feature.

github.com/nextcloud/server

Use MultipartUpload for uploading chunks to s3

nextcloud:master ← nextcloud:enh/s3-multipart-upload-api

opened 04:54PM - 19 May 21 UTC

juliushaertl

+1091 -28

Implements https://github.com/nextcloud/server/issues/19414 This allows chunk…ed upload streaming the chunks directly to S3 so that the final MOVE request no longer needs to assemble the final file on the Nextcloud server. Ref: https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html ## Current API changes to trigger the new upload mechanism (required changes on the client side) The DAV capabilities expose a feature flag `s3-multipart` if available. Additional requirements on top of the current chunking API in order to trigger it: - New additional header is required on all chunked upload requests (MKCOL/PUT/MOVE) - `Destination: http://nextcloud.dev.local/remote.php/dav/files/admin/myfiletarget` - Limitations - Chunk filename must be numberic between 1 and 10000 - Chunks must be at least 5MB of size (except for the last one) - Chunks cannot be downloaded from the upload directory ## Summary of how S3 handles MultipartUpload - CreateMultipartUpload is called with the destination bucket, object key and will return an uploadId - UploadPart gets called for each part with the bucket, objectKey, uploadId, contentLength and the content, it returns a set of an ETag and the UploadId which need to be stored for the CompleteMultipartUpload - CompleteMultipartUpload Receives the list of parts and merges them directly on S3 to the final object ## How it currently works - MKCOL Create the UploadFolder, initiate the MultipartUpload through the IChunkedFileWrite storage and persist the returned chunkToken (uploadId) in the dav properties of the folder - If the destination file does not exist we create a .target file in the upload folder to have a filecache entry - The destination is required upfront to allow writing directly to the target file - PUT Writes the stream data directly to the S3 object through the IChunkedFileWrite storage - Validates the DAV chunk filename name to be numeric between 1 and 10000 - MOVE the MultipartUpload will be completed - if the file exists we are done - if the file didn't exist, call rename to move the .target file from the upload folder to the destination - If a storage doesn't exist the new API calls will just fallback to the previous chunking mechanism ## Test script <details> <summary>Bash script to test the upload</summary> ``` #!/bin/bash # Usage: ./upload.sh targetname1 # Uploads a file consisting of 3 chunks and downloads it a again to verify the upload if [ -z "$1" ] then echo "Usage: ./s3-chunk filepath" exit 1 fi CHUNKSIZE=10 CHUNKS=2 > /tmp/one.testbin for i in $(seq 1 $CHUNKS) do CHUNKSIZE=$(( $RANDOM % 10 + 5 )) dd if=/dev/random of=/tmp/$i.testbin bs=${CHUNKSIZE}M count=1 2>/dev/null cat /tmp/$i.testbin >> /tmp/one.testbin done ls -l /tmp/*.testbin SERVER=http://nextcloud.dev.local USER=admin PASS=admin FILENAME=$1 timestamp=$(date +%s) #echo "=> Cleanup existing file" #curl -X DELETE -u $USER:$PASS "$SERVER/remote.php/dav/files/$USER/$FILENAME" DESTINATION="$SERVER/remote.php/dav/files/admin/$FILENAME" echo "Uploading to $USER:$PASS $SERVER/remote.php/dav/uploads/$USER/$timestamp" echo "=> MKCOL" curl -X MKCOL -u $USER:$PASS $SERVER/remote.php/dav/uploads/$USER/$timestamp \ -H "Destination: $DESTINATION" echo "=> PUT" for i in $(seq 1 $CHUNKS | shuf) do echo "- PART $i" (curl -X PUT -u $USER:$PASS $SERVER/remote.php/dav/uploads/$USER/$timestamp/$i -T /tmp/$i.testbin -H "X-S3-Multipart-Destination: files/admin/$FILENAME" \ -H "Destination: $DESTINATION"; echo " - DONE $i") & done wait # exit 1; echo "=> MOVE" curl -X MOVE -u $USER:$PASS --header "Destination:$SERVER/remote.php/dav/files/$USER/$FILENAME" $SERVER/remote.php/dav/uploads/$USER/$timestamp/.file \ -H "Destination: $DESTINATION" curl -u $USER:$PASS $SERVER/remote.php/dav/files/$USER/$FILENAME > /tmp/compare.testbin md5sum /tmp/one.testbin md5sum /tmp/compare.testbin ls -l /tmp/*.testbin rm /tmp/*.testbin ``` </details>

https://docs.nextcloud.com/server/latest/developer_manual/client_apis/WebDAV/chunking.html

Could this solve my problem and if so how do I activate it?

Thank you very much for your feedback

jtr · April 5, 2024, 12:23pm

What are your memcache.* settings? v2 chunking depends on memcache.distributed being configured and being either Redis or Memcached. Without it, the fallback is v1[1].

EDIT: I should also mention that how you’re uploading matters to. I responded assuming you’re using either the Web UI (web Files client) or one of the official mobile/desktop Files clients.

[1] fix: Validate that we have a proper distributed cache configured by juliushaertl · Pull Request #41354 · nextcloud/server · GitHub

Labouledefortboyard · April 8, 2024, 3:15pm

Hello,

Thank you for your prompt reply.
(Yes sorry for the upload I use the the Web UI)

I don’t have a memory caching system on my server because I didn’t see the point and it was optional.

If I activate it, it will activate version 2 of chunking and solve my problem?

Thank you in advance and have a nice day!

jtr · April 8, 2024, 9:31pm

Yes, it should.

Also your situation reminded me there is still probably a gap in the docs for v2 chunking (which is fairly new) on the Admin side… I’ll see what I can do about that.

Labouledefortboyard · April 9, 2024, 3:44pm

Hello,

I’ve just implemented memcache on my server with APCU at REDIS.
So chuncking v2 is automatically activated?

It seems to work when I use an account on the web page, the assembly no longer seems to be done on the server locally. So that’s good, but it still doesn’t work on the “File drop” share page.
I’ve just noticed this point in the documentation, so that might explain it.

Previously I had only tested uploading with “file drop”, I wonder if uploading would have worked from the user page. I think not, since chuncking v2 wasn’t enabled, but it’s an interesting point.

Do you think this feature could be added to the “File Drop” web page in the future?

Thanks

jtr · April 9, 2024, 4:21pm

Chunking for File drop (public/anonymous uploads) is planned, but not yet implemented. Give it an upvote if so inclined: Allow chunked public upload · Issue #4109 · nextcloud/server · GitHub

Labouledefortboyard · April 10, 2024, 7:47am

Hello,

Thanks for the information. Great I will do it !
Thanks again for your help !

system · April 18, 2024, 7:48am

This topic was automatically closed 8 days after the last reply. New replies are no longer allowed.