Nextcloud server version: 18.0.5
Nextcloud Desktop Client: 2.6.5
Operating system and version: Debian 9
Apache or nginx version: Nginx 1.10.3
PHP version : 7.3
The issue you are facing:
I have tried several times (for 3+ days each) to run desktop sync for a large number of files and directories (150GB of thousands of files and directories) on both Linode and Digital Ocean s3 object storage. I have tried both uploading the files to an empty bucket and downloading from a pre-populated bucket. In both cases, the sync process eventually stalls with errors like:
Expected filesize of 1024000 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 0 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.
There is this related thread for that error, but it is unresolved and does not seem related to s3/object storage, so I first wanted to see if other people have had any success with large s3 file syncs like this.
I have successfully sync’d smaller amounts of storage (~10GB) to test, so I’m not sure where the cut-off is, but sheer volume of files seems to be a factor.
I configured my S3/object storage via the Admin External Storages GUI and set the following:
- enable SSL
- enable path style
- enable previews
- enable sharing
- enabled for one user
Is this the first time you’ve seen this error?: No
Steps to replicate it:
- Create an s3 object storage bucket on Linode or Digital Ocean.
- Start a file sync either to the bucket or from the bucket with desktop Sync for a large number of files (thousands of files/dirs, about 150GB).
The output of your Nextcloud log in Admin > Logging:
[webdav] Fatal: Sabre\DAV\Exception\BadRequest: Expected filesize of 1024000 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 0 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side. at <<closure>>
0. /var/www/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php line 156
OCA\DAV\Connector\Sabre\File->put(null)
1. /var/www/nextcloud/apps/dav/lib/Upload/UploadFolder.php line 47
OCA\DAV\Connector\Sabre\Directory->createFile("0000000000000000-0000000001023999", null)
2. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 1096
OCA\DAV\Upload\UploadFolder->createFile("0000000000000000-0000000001023999", null)
3. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php line 525
Sabre\DAV\Server->createFile("uploads/eliana/ ... 9", null, null)
4. <<closure>>
Sabre\DAV\CorePlugin->httpPut(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
5. /var/www/nextcloud/3rdparty/sabre/event/lib/EventEmitterTrait.php line 105
call_user_func_array([Sabre\DAV\CorePlugin {},"httpPut"], [Sabre\HTTP\Requ ... }])
6. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 479
Sabre\Event\EventEmitter->emit("method:PUT", [Sabre\HTTP\Requ ... }])
7. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 254
Sabre\DAV\Server->invokeMethod(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
8. /var/www/nextcloud/apps/dav/lib/Server.php line 319
Sabre\DAV\Server->exec()
9. /var/www/nextcloud/apps/dav/appinfo/v2/remote.php line 35
OCA\DAV\Server->exec()
10. /var/www/nextcloud/remote.php line 165
require_once("/var/www/nextcl ... p")
PUT /nextcloud/remote.php/dav/uploads/eliana/e42540920a2c9b703c8aac50ace3320d/0000000000000000-0000000001023999
from 172.56.42.1 by eliana at 2020-08-25T17:04:07+00:00
[index] Error: Exception: Argument 1 passed to OCA\Text\Controller\SessionController::sync() must be of the type int, null given, called in /var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php on line 170 at <<closure>>
0. /var/www/nextcloud/lib/private/AppFramework/App.php line 125
OC\AppFramework\Http\Dispatcher->dispatch(OCA\Text\Controller\SessionController {}, "sync")
1. /var/www/nextcloud/lib/private/AppFramework/Routing/RouteActionHandler.php line 47
OC\AppFramework\App::main("OCA\\Text\\Cont ... r", "sync", OC\AppFramework\ ... {}, {_route: "text.Session.sync"})
2. <<closure>>
OC\AppFramework\Routing\RouteActionHandler->__invoke({_route: "text.Session.sync"})
3. /var/www/nextcloud/lib/private/Route/Router.php line 299
call_user_func(OC\AppFramework\ ... {}, {_route: "text.Session.sync"})
4. /var/www/nextcloud/lib/base.php line 1008
OC\Route\Router->match("/apps/text/session/sync")
5. /var/www/nextcloud/index.php line 38
OC::handleRequest()
POST /nextcloud/apps/text/session/sync
from 73.83.235.112 by tony at 2020-08-25T21:08:33+00:00
[webdav] Fatal: Sabre\Xml\ParseException: This should never happen (famous last words) at <<closure>>
0. /var/www/nextcloud/3rdparty/sabre/xml/lib/Element/Base.php line 86
Sabre\Xml\Reader->parseInnerTree()
1. <<closure>>
Sabre\Xml\Element\Base::xmlDeserialize(Sabre\Xml\Reader ... ]})
2. /var/www/nextcloud/3rdparty/sabre/xml/lib/Reader.php line 246
call_user_func(["Sabre\\Xml\\El ... "], Sabre\Xml\Reader ... ]})
3. /var/www/nextcloud/3rdparty/sabre/xml/lib/Reader.php line 71
Sabre\Xml\Reader->parseCurrentElement()
4. /var/www/nextcloud/3rdparty/sabre/xml/lib/Service.php line 118
Sabre\Xml\Reader->parse()
5. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php line 713
Sabre\Xml\Service->parse("", "/nextcloud/remo ... /", null)
6. <<closure>>
Sabre\DAV\CorePlugin->httpReport(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
7. /var/www/nextcloud/3rdparty/sabre/event/lib/EventEmitterTrait.php line 105
call_user_func_array([Sabre\DAV\CorePlugin {},"httpReport"], [Sabre\HTTP\Requ ... }])
8. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 479
Sabre\Event\EventEmitter->emit("method:REPORT", [Sabre\HTTP\Requ ... }])
9. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 254
Sabre\DAV\Server->invokeMethod(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
10. /var/www/nextcloud/apps/dav/lib/Server.php line 319
Sabre\DAV\Server->exec()
11. /var/www/nextcloud/apps/dav/appinfo/v2/remote.php line 35
OCA\DAV\Server->exec()
12. /var/www/nextcloud/remote.php line 165
require_once("/var/www/nextcl ... p")
REPORT /nextcloud/remote.php/dav/calendars/[redacted]
from 73.83.235.112 by tony at 2020-08-25T15:23:21+00:00
The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):
<?php
$CONFIG = array (
'passwordsalt' => [redacted],
'secret' => [redacted],
'trusted_domains' =>
array (
0 => 'localhost',
1 => [redacted],
),
'datadirectory' => '/home/yunohost.app/nextcloud/data',
'dbtype' => 'mysql',
'version' => '18.0.5.1',
'overwrite.cli.url' => [redacted],
'dbname' => 'nextcloud',
'dbhost' => 'localhost',
'dbport' => '',
'dbtableprefix' => 'oc_',
'dbuser' => [redacted],
'dbpassword' => [redacted],
'installed' => true,
'instanceid' => 'ocdsdbdhvn6i',
'ldapIgnoreNamingRules' => false,
'ldapProviderFactory' => 'OCA\\User_LDAP\\LDAPProviderFactory',
'updatechecker' => false,
'memcache.local' => '\\OC\\Memcache\\APCu',
'integrity.check.disabled' => true,
'filelocking.enabled' => true,
'memcache.locking' => '\\OC\\Memcache\\Redis',
'redis' =>
array (
'host' => 'localhost',
'port' => '6379',
'timeout' => '0.0',
'password' => '',
),
'hashing_default_password' => true,
'logout_url' => [redacted],
'mail_smtpmode' => 'smtp',
'mail_smtpsecure' => 'tls',
'mail_sendmailmode' => 'smtp',
'mail_from_address' => 'no-reply',
'mail_domain' => [redacted],
'mail_smtpauthtype' => 'LOGIN',
'mail_smtphost' => [redacted],
'mail_smtpport' => '587',
'maintenance' => false,
'preview_max_x' => '2048',
'preview_max_y' => '2048',
'jpeg_quality' => '60',
);
The output of your Apache/nginx/system log in /var/log/____: nginx error log
2020/08/25 16:08:33 [error] 1888#1888: *336558 lua entry thread aborted: runtime error: /usr/share/ssowat/helpers.lua:382: bad argument #1 to 'find' (string expected, got nil)
stack traceback:
coroutine 0:
[C]: in function 'find'
/usr/share/ssowat/helpers.lua:382: in function 'authenticate'
/usr/share/ssowat/access.lua:361: in function </usr/share/ssowat/access.lua:1>, client: 73.83.235.112, server: nextcloud.opensourceit.org, request: "HEAD /remote.php/webdav/InstantUpload/Screenshots/Screenshot_20200824-225543.png HTTP/1.1", host: "nextcloud.opensourceit.org"
2020/08/25 16:08:33 [error] 1888#1888: *336561 lua entry thread aborted: runtime error: /usr/share/ssowat/helpers.lua:382: bad argument #1 to 'find' (string expected, got nil)
stack traceback:
coroutine 0:
[C]: in function 'find'
/usr/share/ssowat/helpers.lua:382: in function 'authenticate'
/usr/share/ssowat/access.lua:361: in function </usr/share/ssowat/access.lua:1>, client: 73.83.235.112, server: nextcloud.opensourceit.org, request: "PUT /remote.php/webdav/InstantUpload/Screenshots/Screenshot_20200824-225543.png HTTP/1.1", host: "nextcloud.opensourceit.org"
2020/08/25 16:08:35 [error] 1888#1888: *336564 lua entry thread aborted: runtime error: /usr/share/ssowat/helpers.lua:382: bad argument #1 to 'find' (string expected, got nil)
stack traceback:
coroutine 0:
[C]: in function 'find'
/usr/share/ssowat/helpers.lua:382: in function 'authenticate'
/usr/share/ssowat/access.lua:361: in function </usr/share/ssowat/access.lua:1>, client: 73.83.235.112, server: nextcloud.opensourceit.org, request: "HEAD /remote.php/webdav/InstantUpload/IMG_20200823_155850.jpg HTTP/1.1", host: "nextcloud.opensourceit.org"
2020/08/25 16:08:35 [error] 1888#1888: *336565 lua entry thread aborted: runtime error: /usr/share/ssowat/helpers.lua:382: bad argument #1 to 'find' (string expected, got nil)
stack traceback:
coroutine 0:
[C]: in function 'find'
...