S3 external storage : every uploaded files are 0 byte plus can't create/delete dir

ℹ️ Support
,
1

Hi there. As the title says i have issues with external storage with S3 compliant mount. I successfully configured the S3 connection: i can see and browse the folder through nextcloud but when i try to upload a file in it, the uploaded file will always be 0 bytes. In the log i got this error:

[files_external] Error: Aws\S3\Exception\S3Exception: Error parsing response for CopyObject: AWS parsing error: Error parsing XML: String could not be parsed as XML at <<closure>>

 0. /var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/S3/AmbiguousSuccessParser.php line 58
    Aws\S3\RetryableMalformedResponseParser->__invoke(Aws\Command {}, "*** sensitive parameter replaced ***")
 1. /var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/S3/GetBucketLocationParser.php line 29
    Aws\S3\AmbiguousSuccessParser->__invoke(Aws\Command {}, "*** sensitive parameter replaced ***")
 2. /var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/WrappedHttpHandler.php line 125
    Aws\S3\GetBucketLocationParser->__invoke(Aws\Command {}, "*** sensitive parameter replaced ***")
 3. /var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/WrappedHttpHandler.php line 92
    Aws\WrappedHttpHandler->parseResponse(Aws\Command {}, GuzzleHttp\Psr7\Request {}, "*** sensitive parameter replaced ***", [])
 4. /var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php line 203
    Aws\WrappedHttpHandler->Aws\{closure}("*** sensitive parameters replaced ***")
 5. /var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php line 156
    GuzzleHttp\Promise\Promise::callHandler(1, "*** sensitive parameter replaced ***", [GuzzleHttp\Prom ... }])
 6. /var/www/nextcloud/3rdparty/guzzlehttp/promises/src/TaskQueue.php line 47
    GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}("*** sensitive parameters replaced ***")
 7. /var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php line 98
    GuzzleHttp\Promise\TaskQueue->run()
 8. /var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php line 125
    GuzzleHttp\Handler\CurlMultiHandler->tick()
 9. /var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php line 246
    GuzzleHttp\Handler\CurlMultiHandler->execute(true)
10. /var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php line 223
    GuzzleHttp\Promise\Promise->invokeWaitFn()
11. /var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php line 267
    GuzzleHttp\Promise\Promise->waitIfPending()
12. /var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php line 225
    GuzzleHttp\Promise\Promise->invokeWaitList()
13. /var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php line 267
    GuzzleHttp\Promise\Promise->waitIfPending()
14. /var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php line 225
    GuzzleHttp\Promise\Promise->invokeWaitList()
15. /var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php line 62
    GuzzleHttp\Promise\Promise->waitIfPending()
16. /var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/AwsClientTrait.php line 58
    GuzzleHttp\Promise\Promise->wait()
17. /var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/AwsClientTrait.php line 77
    Aws\AwsClient->execute(Aws\Command {})
18. /var/www/nextcloud/apps/files_external/lib/Lib/Storage/AmazonS3.php line 572
    Aws\AwsClient->__call("copyObject", [{Bucket: "gabem ... }])
19. /var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php line 354
    OCA\Files_External\Lib\Storage\AmazonS3->touch("upload-test.txt", 1582987619)
20. /var/www/nextcloud/lib/private/Files/Storage/Wrapper/PermissionsMask.php line 100
    OC\Files\Storage\Wrapper\Wrapper->touch("upload-test.txt", 1582987619)
21. /var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php line 354
    OC\Files\Storage\Wrapper\PermissionsMask->touch("upload-test.txt", 1582987619)
22. /var/www/nextcloud/lib/private/Files/Storage/Wrapper/Availability.php line 364
    OC\Files\Storage\Wrapper\Wrapper->touch("upload-test.txt", 1582987619)
23. /var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php line 354
    OC\Files\Storage\Wrapper\Availability->touch("upload-test.txt", 1582987619)
24. /var/www/nextcloud/lib/private/Files/View.php line 1160
    OC\Files\Storage\Wrapper\Wrapper->touch("upload-test.txt", 1582987619)
25. /var/www/nextcloud/lib/private/Files/View.php line 569
    OC\Files\View->basicOperation("touch", "/S3data/upload-test.txt", ["touch"], 1582987619)
26. /var/www/nextcloud/apps/dav/lib/Connector/Sabre/File.php line 302
    OC\Files\View->touch("/S3data/upload-test.txt", 1582987619)
27. /var/www/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php line 156
    OCA\DAV\Connector\Sabre\File->put(null)
28. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 1096
    OCA\DAV\Connector\Sabre\Directory->createFile("upload-test.txt", null)
29. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php line 525
    Sabre\DAV\Server->createFile("S3data/upload-test.txt", null, null)
30. <<closure>>
    Sabre\DAV\CorePlugin->httpPut(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
31. /var/www/nextcloud/3rdparty/sabre/event/lib/EventEmitterTrait.php line 105
    call_user_func_array([Sabre\DAV\CorePlugin {},"httpPut"], [Sabre\HTTP\Requ ... }])
32. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 479
    Sabre\Event\EventEmitter->emit("method:PUT", [Sabre\HTTP\Requ ... }])
33. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 254
    Sabre\DAV\Server->invokeMethod(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
34. /var/www/nextcloud/apps/dav/appinfo/v1/webdav.php line 82
    Sabre\DAV\Server->exec()
35. /var/www/nextcloud/remote.php line 165
    require_once("/var/www/nextcl ... p")

PUT /remote.php/webdav/S3data/upload-test.txt
from xxx.xxx.xxx.xxx by admin at 2020-02-29T14:47:29+00:00

Same error when i try to create a folder: the uploaded result is not a folder but another file of 0 bytes with the name of the folder. I’m using Nextcloud up to date to the last stable version 18.0.1

I would like to describe another scenario which adds some other infos: if i try to upload a SINGLE FILE into nextcloud home directory first (local storage) and after that copy/move it into the S3 folder mounted by external storage, the uploaded file is with correct size, no errors.

Another case is when i try to copy/move a folder (with files) from local nextcloud to s3 mounted external storage. The results will be a 0 bytes file with the same name of the copied/moved folder. If i check the content of the S3 storage with it’s own web admin panel, i see this:

  1. testfolder (file, 0 byte)
  2. testfolder (dir, correctly uploaded with files inside)

NOTE: The folder “testfolder” is hidden to nextcloud until i delete the "testfolder (file, 0 byte) from the S3 bucket . Only after this operation the folder will be visible under nextcloud.

2

I would start investigating there. Cannot help you further without more information regarding your setup including the S3 provider, how you’ve configured the bucket and how it’s setup on external storage.

3

@Paradox551

First of all thank you for answering. The S3 provider is Filebase. The bucket is properly configured, i usually access it with rclone without any problems. The config under external storage seems good, i have green tick next to the folder name and i can browse and download from it. The problem seems to occur only when i directly upload to the S3 mount (tried both with android app and web browser): as i previously said, if i upload to local first, then copy/move to S3 dir, everything works good for single file. Talking about directories instead when i try to copy/move from local to S3 dir, the tricky thing already described occurs. If some other informations are needed let me know…

4

Do you have a large enough hard drive? When you upload via nextcloud it’s stored locally and then synced to the back end S3.

5

Yep, the test i did was with text files and other small files: the biggest one was about 80MBytes

6

Open a bug report on GitHub so the developers can investigate.

1 Like
7

I have a little update about this. I was making some tests with a local minio instance and discovered that enabling the “enable path style” option, everything started working good. Before it was disabled. So now the direct upload working good with Filebase too. Thank you to everyone for the support.