Data migration with nextcloud using s3 object storage as primary storage possible?

Hi ,

It seams like I have successfully connected to drop box and google drive in the admin section of nextcloud. when I try data migration for either I get reported data size in drop box and contacts caleders and drive shows up.

It says

1000 files in Google Drive (922 MiB + 396 MiB shared with you)

0 files imported (0%)
Google Drive background import process will begin soon. You can close this page. You will be notified when it finishes.

the same for drop box , but no files ever sync.

The files never download.

is it an issue that i am using S3 as primary storage ?

nextcloud on 20.0.8
ubuntu 20.04
Php 7
apache latest stable
mysql latest stable

I have logs of errors like this in the logging…

|Error|PHP|Error: fread(): read of 8192 bytes failed with errno=9 Bad file descriptor at /var/www/html/nextcloud/3rdparty/icewind/streams/src/Wrapper.php#91

  1. <>OC\Log\ErrorHandler::onError()
  2. /var/www/html/nextcloud/3rdparty/icewind/streams/src/Wrapper.php - line 91:fread()
  3. /var/www/html/nextcloud/3rdparty/icewind/streams/src/CallbackWrapper.php - line 98:Icewind\Streams\Wrapper->stream_read()
  4. <>Icewind\Streams\CallbackWrapper->stream_read()
  5. /var/www/html/nextcloud/3rdparty/icewind/streams/src/Wrapper.php - line 91:fread()
  6. /var/www/html/nextcloud/3rdparty/icewind/streams/src/CallbackWrapper.php - line 98:Icewind\Streams\Wrapper->stream_read()
  7. <>Icewind\Streams\CallbackWrapper->stream_read()
  8. /var/www/html/nextcloud/3rdparty/guzzlehttp/psr7/src/Stream.php - line 225:fread()
  9. /var/www/html/nextcloud/3rdparty/guzzlehttp/psr7/src/functions.php - line 867:GuzzleHttp\Psr7\Stream->read()
  10. /var/www/html/nextcloud/3rdparty/guzzlehttp/guzzle/src/Exception/RequestException.php - line 127:GuzzleHttp\Psr7\get_message_body_summary()
  11. /var/www/html/nextcloud/3rdparty/guzzlehttp/guzzle/src/Exception/RequestException.php - line 107:GuzzleHttp\Exception\RequestException::getResponseBodySummary()
  12. /var/www/html/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php - line 65:GuzzleHttp\Exception\RequestException::create()
  13. /var/www/html/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 203:GuzzleHttp\Middleware::GuzzleHttp{closure}("*** sensiti … *")
  14. /var/www/html/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 156:GuzzleHttp\Promise\Promise::callHandler()
  15. /var/www/html/nextcloud/3rdparty/guzzlehttp/promises/src/TaskQueue.php - line 47:GuzzleHttp\Promise\Promise::GuzzleHttp\Promise{closure}("*** sensiti … *")
  16. /var/www/html/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 246:GuzzleHttp\Promise\TaskQueue->run()
  17. /var/www/html/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 223:GuzzleHttp\Promise\Promise->invokeWaitFn()
  18. /var/www/html/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 267:GuzzleHttp\Promise\Promise->waitIfPending()
  19. /var/www/html/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 225:GuzzleHttp\Promise\Promise->invokeWaitList()
  20. /var/www/html/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php - line 62:GuzzleHttp\Promise\Promise->waitIfPending()
  21. /var/www/html/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php - line 183:GuzzleHttp\Promise\Promise->wait()
  22. /var/www/html/nextcloud/lib/private/Http/Client/Client.php - line 233:GuzzleHttp\Client->request()
  23. /var/www/html/nextcloud/apps/integration_google/lib/Service/GoogleAPIService.php - line 331:OC\Http\Client\Client->get()
  24. /var/www/html/nextcloud/apps/integration_google/lib/Service/GoogleDriveAPIService.php - line 373:OCA\Google\Service\GoogleAPIService->simpleDownload()
  25. /var/www/html/nextcloud/apps/integration_google/lib/Service/GoogleDriveAPIService.php - line 278:OCA\Google\Service\GoogleDriveAPIService->getFile()
  26. /var/www/html/nextcloud/apps/integration_google/lib/Service/GoogleDriveAPIService.php - line 160:OCA\Google\Service\GoogleDriveAPIService->importFiles()
  27. /var/www/html/nextcloud/apps/integration_google/lib/BackgroundJob/ImportDriveJob.php - line 36:OCA\Google\Service\GoogleDriveAPIService->importDriveJob()
  28. /var/www/html/nextcloud/lib/public/BackgroundJob/Job.php - line 80:OCA\Google\BackgroundJob\ImportDriveJob->run()
  29. /var/www/html/nextcloud/lib/public/BackgroundJob/QueuedJob.php - line 48:OCP\BackgroundJob\Job->execute()
  30. /var/www/html/nextcloud/cron.php - line 149:OCP\BackgroundJob\QueuedJob->execute()|
    | — | — |
Support intro

Sorry to hear you’re facing problems :slightly_frowning_face:

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can :heart:

Nextcloud version (eg, 20.0.5): replace me
Operating system and version (eg, Ubuntu 20.04): replace me
Apache or nginx version (eg, Apache 2.4.25): replace me
PHP version (eg, 7.4): replace me

The issue you are facing:

Is this the first time you’ve seen this error? (Y/N):

Steps to replicate it:

The output of your Nextcloud log in Admin > Logging:

PASTE HERE

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

PASTE HERE

The output of your Apache/nginx/system log in /var/log/____:

PASTE HERE

OK, I updated nextcloud to 21 and some files start coming in…

BUT i’m still getting tons of errors and many files fail…

{“reqId”:“BEpY91hT2wXIhhVGwaQz”,“level”:3,“time”:“2021-04-09T13:44:23+00:00”,“remoteAddr”:“123.198.238.235”,“user”:"–",“app”:“PHP”,“method”:“GET”,“url”:"/cron.php",“message”:{“Exception”:“Error”,“Message”:“fread(): read of 8192 bytes failed with errno=9 Bad file descriptor at /var/www/html/nextcloud/3rdparty/icewind/streams/src/Wrapper.php#55”,“Code”:0,“Trace”:[{“function”:“onError”,“class”:“OC\Log\ErrorHandler”,“type”:"::"},{“file”:"/var/www/html/nextcloud/3rdparty/icewind/streams/src/Wrapper.php",“line”:55,“function”:“fread”},{“file”:"/var/www/html/nextcloud/3rdparty/icewind/streams/src/CallbackWrapper.php",“line”:96,“function”:“stream_read”,“class”:“Icewind\Streams\Wrapper”,“type”:"->"},{“function”:“stream_read”,“class”:“Icewind\Streams\CallbackWrapper”,“type”:"->"},{“file”:"/var/www/html/nextcloud/3rdparty/icewind/streams/src/Wrapper.php",“line”:55,“function”:“fread”},{“file”:"/var/www/html/nextcloud/3rdparty/icewind/streams/src/CallbackWrapper.php",“line”:96,“function”:“stream_read”,“class”:“Icewind\Streams\Wrapper”,“type”:"->"},{“function”:“stream_read”,“class”:“Icewind\Streams\CallbackWrapper”,“type”:"->"},{“file”:"/var/www/html/nextcloud/3rdparty/guzzlehttp/psr7/src/Stream.php",“line”:228,“function”:“fread”},{“file”:"/var/www/html/nextcloud/3rdparty/guzzlehttp/psr7/src/Message.php",“line”:72,“function”:“read”,“class”:“GuzzleHttp\Psr7\Stream”,“type”:"->"},{“file”:"/var/www/html/nextcloud/3rdparty/guzzlehttp/guzzle/src/BodySummarizer.php",“line”:25,“function”:“bodySummary”,“class”:“GuzzleHttp\Psr7\Message”,“type”:"::"},{“file”:"/var/www/html/nextcloud/3rdparty/guzzlehttp/guzzle/src/Exception/RequestException.php",“line”:107,“function”:“summarize”,“class”:“GuzzleHttp\BodySummarizer”,“type”:"->"},{“file”:"/var/www/html/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php",“line”:69,“function”:“create”,“class”:“GuzzleHttp\Exception\RequestException”,“type”:"::"},{“file”:"/var/www/html/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php",“line”:204,“function”:“GuzzleHttp\{closure}”,“class”:“GuzzleHttp\Middleware”,“type”:"::",“args”:["*** sensitive parameters replaced "]},{“file”:"/var/www/html/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php",“line”:153,“function”:“callHandler”,“class”:“GuzzleHttp\Promise\Promise”,“type”:"::"},{“file”:"/var/www/html/nextcloud/3rdparty/guzzlehttp/promises/src/TaskQueue.php",“line”:48,“function”:“GuzzleHttp\Promise\{closure}”,“class”:“GuzzleHttp\Promise\Promise”,“type”:"::",“args”:[" sensitive parameters replaced ***"]},{“file”:"/var/www/html/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php",“line”:248,“function”:“run”,“class”:“GuzzleHttp\Promise\TaskQueue”,“type”:"->"},{“file”:"/var/www/html/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php",“line”:224,“function”:“invokeWaitFn”,“class”:“GuzzleHttp\Promise\Promise”,“type”:"->"},{“file”:"/var/www/html/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php",“line”:269,“function”:“waitIfPending”,“class”:“GuzzleHttp\Promise\Promise”,“type”:"->"},{“file”:"/var/www/html/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php",“line”:226,“function”:“invokeWaitList”,“class”:“GuzzleHttp\Promise\Promise”,“type”:"->"},{“file”:"/var/www/html/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php",“line”:62,“function”:“waitIfPending”,“class”:“GuzzleHttp\Promise\Promise”,“type”:"->"},{“file”:"/var/www/html/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php",“line”:187,“function”:“wait”,“class”:“GuzzleHttp\Promise\Promise”,“type”:"->"},{“file”:"/var/www/html/nextcloud/lib/private/Http/Client/Client.php",“line”:236,“function”:“request”,“class”:“GuzzleHttp\Client”,“type”:"->"},{“file”:"/var/www/html/nextcloud/apps/integration_google/lib/Service/GoogleAPIService.php",“line”:331,“function”:“get”,“class”:“OC\Http\Client\Client”,“type”:"->"},{“file”:"/var/www/html/nextcloud/apps/integration_google/lib/Service/GoogleDriveAPIService.php",“line”:444,“function”:“simpleDownload”,“class”:“OCA\Google\Service\GoogleAPIService”,“type”:"->"},{“file”:"/var/www/html/nextcloud/apps/integration_google/lib/Service/GoogleDriveAPIService.php",“line”:278,“function”:“getFile”,“class”:“OCA\Google\Service\GoogleDriveAPIService”,“type”:"->"},{“file”:"/var/www/html/nextcloud/apps/integration_google/lib/Service/GoogleDriveAPIService.php",“line”:160,“function”:“importFiles”,“class”:“OCA\Google\Service\GoogleDriveAPIService”,“type”:"->"},{“file”:"/var/www/html/nextcloud/apps/integration_google/lib/BackgroundJob/ImportDriveJob.php",“line”:36,“function”:“importDriveJob”,“class”:“OCA\Google\Service\GoogleDriveAPIService”,“type”:"->"},{“file”:"/var/www/html/nextcloud/lib/public/BackgroundJob/Job.php",“line”:80,“function”:“run”,“class”:“OCA\Google\BackgroundJob\ImportDriveJob”,“type”:"->"},{“file”:"/var/www/html/nextcloud/lib/public/BackgroundJob/QueuedJob.php",“line”:48,“function”:“execute”,“class”:“OCP\BackgroundJob\Job”,“type”:"->"},{“file”:"/var/www/html/nextcloud/cron.php",“line”:150,“function”:“execute”,“class”:“OCP\BackgroundJob\QueuedJob”,“type”:"->"}],“File”:"/var/www/html/nextcloud/lib/private/Log/ErrorHandler.php",“Line”:92,“CustomMessage”:"–"},“userAgent”:“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36”,“version”:“21.0.1.1”,“id”:“60731e6367bb6”}