How to assemble chunks while uploading large file using sabre webdav php client?

[kotelnikus]

hi

I would like to implement chunked upload for large files. I was following this documentation Chunked file upload — Nextcloud latest Developer Manual latest documentation but I struggle with the final the most important step - Assembling the chunks

The code I used moves the whole temp directory to new directory that is named as the final file.

$sourcePath = $tempFolder . '.file';
$destinationPath = $baseUri . $uploadedFolderName . $uploadedFileName;

// Send the MOVE request
$response = $client->request('MOVE', $tempFolder, null, [
    'Destination' => $destinationPath
]);

• I am using sabre webdav ftp client
• I am using Nextcloud server implemented by wedos.cz on cd.wedos.com. So I do not have any control of the server

My thinking is that

1. syntax of the assembling code is not right or
2. there must be something set up or turned on on the server or
3. the doc says " The API is only available for registered users of your instance. And uses the path: /remote.php/dav/uploads/." but I use files in the url. If I use uploads then I cannot see if the temp folder was created. There is no error though. It looks like the chunks were uploaded but I cannot confirm and also there is no error returned when uploading. But the MOVE finishes with an error message “The destination node is not found”

Could someone please help me to fix this?

The code for uploading chunks

// Read the file in chunks and upload
$fp = fopen($filePath, 'rb');
$chunkNumber = 0;
$startByte = 0;
while (!feof($fp)) {
    $chunk = fread($fp, $chunkSize);
    $endByte = $startByte + strlen($chunk) - 1;

    // Generate a unique filename based on start and end bytes
    $chunkFileName = sprintf('%020d-%020d', $startByte, $endByte);
    $chunkPath = $tempFolder . '/' . $chunkFileName;

    // Upload the chunk
    // $client->put($chunkPath, $chunk);

//     // Upload the chunk
    try {
        $client->request('PUT', $chunkPath, $chunk);
        echo "Uploaded chunk $chunkNumber successfully.\n";
    } catch (Exception $e) {
        echo "Error uploading chunk $chunkNumber: " . $e->getMessage() . "\n";
        break; // Handle errors appropriately
    }



    $startByte += strlen($chunk);
    $chunkNumber++;
}
fclose($fp);

[jtr]

The docs you linked are for Nextcloud Server v20. We’re on v30 these days.

See here for the Version 2 chunking API: Chunked file upload — Nextcloud latest Developer Manual latest documentation

[kotelnikus]

any chance you can help me to make it work? I believe I have everything right

<?php

// installed via - composer require sabre/dav

include 'vendor/autoload.php';
use Sabre\DAV\Client;

$baseUri = '/remote.php/dav/files/wedos-auth-7518fe8d-0f5f-4c87-99a2-6exxxxxxx/';
// $baseUri = '/remote.php/dav/uploads/wedos-auth-7518fe8d-0f5f-4c87-99a2-6exxxxxxxx/';

$domain = 'https://cd.wedos.com';

$settings = array(
    'baseUri' => $domain . $baseUri,
    'userName' => 'xxxxxxxx@gmail.com',
    'password' => 'xxxxxx'
);

$client = new Client($settings);


$filePath = 'deprese.pdf';

$uploadedFileName = 'deprese.pdf';
$uploadedFolderName = 'uploaded/';

$chunkSize = 1048576 * 5; // 1 MB * 5 = 5MB

// Create a temporary folder for chunks
$tempFolder = $baseUri . uniqid('chunks_');

// var_dump($tempFolder);

$response = $client->request('MKCOL', $tempFolder);
// var_dump($response);




// // Read the file in chunks and upload
 $fp = fopen($filePath, 'rb');
 $chunkNumber = 0;
 $fileSize = filesize($filePath);

 while (!feof($fp)) {
     $chunk = fread($fp, $chunkSize);
     $chunkNumber++;

//     // Generate a unique filename for the chunk
        // $chunkFileName = sprintf('%010d', $chunkNumber);
        $chunkFileName =  $chunkNumber;
        $chunkPath = $tempFolder . '/' . $chunkFileName;

    // Upload the chunk
    try {
        $client->request('PUT', $chunkPath, $chunk, [
            'OC-Total-Length' => $fileSize
        ]);
        echo "Uploaded chunk $chunkNumber successfully.\n";
    } catch (Exception $e) {
        echo "Error uploading chunk $chunkNumber: " . $e->getMessage() . "\n";
        break; // Handle errors appropriately
    }

 }
 fclose($fp);

// // Assemble the final file using MOVE
// Define the source and destination paths


$sourcePath = $tempFolder . '/.file';
$destinationPath = $baseUri . $uploadedFolderName . $uploadedFileName;

// var_dump($destinationPath);


// Send the MOVE request
$response = $client->request('MOVE', $tempFolder, null, [
    'Destination' => $destinationPath,
    'OC-Total-Length' => $fileSize
]);

var_dump($response);

[jtr]

@kotelnikus

$baseUri = '/remote.php/dav/files/wedos-auth-7518fe8d-0f5f-4c87-99a2-6exxxxxxx/';

The above should be /uploads/ and appears to already include a uniquely named folder for the upload embedded in it statically. You appear to be attempting to create another uniquely named folder within it below:

@kotelnikus

// Create a temporary folder for chunks
$tempFolder = $baseUri . uniqid('chunks_');

// var_dump($tempFolder);

$response = $client->request('MKCOL', $tempFolder);

I don’t see anywhere you’re including the Destination header with the MKCOL request.

@kotelnikus

    // Upload the chunk
    try {
        $client->request('PUT', $chunkPath, $chunk, [
            'OC-Total-Length' => $fileSize
        ]);

It also appears your PUT needs the Destination header.

P.S. You may find the original PR that implemented v2 chunking of interest. It has a sample bash script that implements uploading.

[kotelnikus]

you are right … I updated the code and now

1. I am able to create temp directory
2. I am able to upload chunks
   • if I upload to …/files/… I can see them.
   • if I upload to …/upload/… I cant see them but I would say they were uploaded. There was no error
3. So the only step for uploading a large file is to merge all the chunks.

@jtr

$baseUri = '/remote.php/dav/files/wedos-auth-7518fe8d-0f5f-4c87-99a2-6exxxxxxx/';

The above should be /uploads/ and appears to already include a uniquely named folder for the upload embedded in it statically. You appear to be attempting to create another uniquely named folder within it below:

Do you mean wedos-auth-7518fe8d-0f5f-4c87-99a2-6exxxxxxx ? It is an userId.

@jtr

// Create a temporary folder for chunks
$tempFolder = $baseUri . uniqid('chunks_');

// var_dump($tempFolder);

$response = $client->request('MKCOL', $tempFolder);

I don’t see anywhere you’re including the Destination header with the MKCOL request.

I am using Sabre webdav client. So the syntax is different. The directory is created - I can see it via web UI. Of course if I use …/files/ not …/uploads/

The same applies for PUT. It works. I am able to upload. But the merge.

Below is the latest version of my code. It produces another error that I described in another post . ’ Incompatible node types error’

<?php

// installed via - composer require sabre/dav

include 'vendor/autoload.php';
use Sabre\DAV\Client;

$baseUri = '/remote.php/dav/files/wedos-auth-7518fe8d-0f5f-4c87-99a2-xxx/';
$baseUriUpload = '/remote.php/dav/uploads/wedos-auth-7518fe8d-0f5f-4c87-99a2-xxx/';

$domain = 'https://cd.wedos.com';

$settings = array(
    'baseUri' => $domain . $baseUriUpload,
    'userName' => 'xxx@gmail.com',
    'password' => 'xx'
);

$client = new Client($settings);
$filePath = 'deprese.pdf';

$uploadedFileName = 'deprese.pdf';
$uploadedFolderName = 'uploaded/';

$chunkSize = 1048576 * 5; // 1 MB * 5 = 5MB

// Create a temporary folder for chunks
$tempFolder = $baseUriUpload . uniqid('chunks_');

var_dump($tempFolder);

$response = $client->request('MKCOL', $tempFolder);

// // Read the file in chunks and upload
 $fp = fopen($filePath, 'rb');
 $chunkNumber = 0;
 $fileSize = filesize($filePath);

 while (!feof($fp)) {
     $chunk = fread($fp, $chunkSize);
     $chunkNumber++;

//     // Generate a unique filename for the chunk
        // $chunkFileName = sprintf('%010d', $chunkNumber);
        $chunkFileName =  $chunkNumber;
        $chunkPath = $tempFolder . '/' . $chunkFileName;

    // Upload the chunk
    try {
        $client->request('PUT', $chunkPath, $chunk, [
            'OC-Total-Length' => $fileSize
        ]);
        echo "Uploaded chunk $chunkNumber successfully.\n";
    } catch (Exception $e) {
        echo "Error uploading chunk $chunkNumber: " . $e->getMessage() . "\n";
        break; // Handle errors appropriately
    }

 }
 fclose($fp);

// // Assemble the final file using MOVE
// Define the source and destination paths

$sourcePath = $tempFolder . '/.file';
$destinationPath = $baseUri . $uploadedFolderName . $uploadedFileName;

// Send the MOVE request
$response = $client->request('MOVE', $tempFolder, null, [
    'Destination' => $destinationPath,
    'OC-Total-Length' => $fileSize
]);

var_dump($response);

[kotelnikus]

For the final step of a chunked upload, one needs to send a MOVE request from the pseudo-file:

<server>/remote.php/dav/uploads/<user>/<folder>/.file

to the destination file:

<server>/remote.php/dav/files/<user>/<folder>/<file>

So, in my case, the correct way to do the request is:

$sourcePath = $tempFolder . '/.file';
$destinationPath = $baseUri . $uploadedFolderName . $uploadedFileName;

$response = $client->request('MOVE', $sourcePath, null, [
    'Destination' => $destinationPath,
    'OC-Total-Length' => $fileSize
]);