Primary storage Swift - Internal server error - 401: Unauthorized

Nextcloud version: 13.0.0
Operating system and version: Debian stable
Apache or nginx version: Apache/2.4.25 (Debian)
PHP version: PHP 7.0.27-0+deb9u1

Sometimes - it seems ad random - the connection between Nextcloud and the Objectstore fails.

  • In a browser: “Internal server error”
  • In a client: unable to sync (don’t know exact error, will add if relevant)
  • In the logs: objectstore Error; Webdav Fatal (logs added below)

I am not 100% positive, but it seems to happen for about an hour per day, and I am under the impression that the situation shifts one hour per day. (So 24h without errors, the 25th hour = error). Afterwards, without an intervention, everything works again as before (for 24h at least).

Checked with openstack/swift provider - they report the system as operational. During the “downtime” I can log into the object store, and see all objects (although they are not meaningful obviously without the nextcloud-metadata).

Any tips, things to check or do?

Thank you very much

Nextcloud logs:

Fatal	webdav	

Sabre\DAV\Exception: Client error response [status code] 401 [reason phrase] Unauthorized [url] https://private_id.objectstore.eu/nextcloud_swift2
    /var/www/nextcloud13/apps/dav/lib/Connector/Sabre/File.php - line 186: OCA\DAV\Connector\Sabre\File->convertToSabreException(Object(Guzzle\Http\Exception\ClientErrorResponseException))
    /var/www/nextcloud13/apps/dav/lib/Connector/Sabre/Directory.php - line 151: OCA\DAV\Connector\Sabre\File->put(Resource id #21)
    /var/www/nextcloud13/3rdparty/sabre/dav/lib/DAV/Server.php - line 1096: OCA\DAV\Connector\Sabre\Directory->createFile('private_file', Resource id #21)
    /var/www/nextcloud13/3rdparty/sabre/dav/lib/DAV/CorePlugin.php - line 525: Sabre\DAV\Server->createFile('private_file', Resource id #21, NULL)
    [internal function] Sabre\DAV\CorePlugin->httpPut(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))
    /var/www/nextcloud13/3rdparty/sabre/event/lib/EventEmitterTrait.php - line 105: call_user_func_array(Array, Array)
    /var/www/nextcloud13/3rdparty/sabre/dav/lib/DAV/Server.php - line 479: Sabre\Event\EventEmitter->emit('method PUT', Array)
    /var/www/nextcloud13/3rdparty/sabre/dav/lib/DAV/Server.php - line 254: Sabre\DAV\Server->invokeMethod(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))
    /var/www/nextcloud13/apps/dav/lib/Server.php - line 283: Sabre\DAV\Server->exec()
    /var/www/nextcloud13/apps/dav/appinfo/v2/remote.php - line 35: OCA\DAV\Server->exec()
    /var/www/nextcloud13/remote.php - line 164: require_once('/var/www/nextcl...')
    {main}


Error 	objectstore	

Could not create object urn:oid:71460 for private_file.ocTransferId1101220213.part: {"Exception":"Guzzle\\Http\\Exception\\ClientErrorResponseException","Message":"Client error response\n[status code] 401\n[reason phrase] Unauthorized\n[url] https:\/\/private_id.objectstore.eu\/nextcloud_swift2","Code":0,"Trace":"#0 \/var\/www\/nextcloud13\/3rdparty\/guzzle\/guzzle\/src\/Guzzle\/Http\/Message\/Request.php(145): Guzzle\\Http\\Exception\\BadResponseException::factory(Object(Guzzle\\Http\\Message\\EntityEnclosingRequest), Object(Guzzle\\Http\\Message\\Response))\n#1 [internal function]: Guzzle\\Http\\Message\\Request::onRequestError(Object(Guzzle\\Common\\Event), 'request.error', Object(Symfony\\Component\\EventDispatcher\\EventDispatcher))\n#2 \/var\/www\/nextcloud13\/3rdparty\/symfony\/event-dispatcher\/EventDispatcher.php(212): call_user_func(Array, Object(Guzzle\\Common\\Event), 'request.error', Object(Symfony\\Component\\EventDispatcher\\EventDispatcher))\n#3 \/var\/www\/nextcloud13\/3rdparty\/symfony\/event-dispatcher\/EventDispatcher.php(44): Symfony\\Component\\EventDispatcher\\EventDispatcher->doDispatch(Array, 'request.error', Object(Guzzle\\Common\\Event))\n#4 \/var\/www\/nextcloud13\/3rdparty\/guzzle\/guzzle\/src\/Guzzle\/Http\/Message\/Request.php(589): Symfony\\Component\\EventDispatcher\\EventDispatcher->dispatch('request.error', Object(Guzzle\\Common\\Event))\n#5 \/var\/www\/nextcloud13\/3rdparty\/guzzle\/guzzle\/src\/Guzzle\/Http\/Message\/Request.php(378): Guzzle\\Http\\Message\\Request->processResponse(Array)\n#6 \/var\/www\/nextcloud13\/3rdparty\/guzzle\/guzzle\/src\/Guzzle\/Http\/Message\/EntityEnclosingRequest.php(49): Guzzle\\Http\\Message\\Request->setState('complete', Array)\n#7 \/var\/www\/nextcloud13\/3rdparty\/guzzle\/guzzle\/src\/Guzzle\/Http\/Curl\/CurlMulti.php(286): Guzzle\\Http\\Message\\EntityEnclosingRequest->setState('complete', Array)\n#8 \/var\/www\/nextcloud13\/3rdparty\/guzzle\/guzzle\/src\/Guzzle\/Http\/Curl\/CurlMulti.php(244): Guzzle\\Http\\Curl\\CurlMulti->processResponse(Object(Guzzle\\Http\\Message\\EntityEnclosingRequest), Object(Guzzle\\Http\\Curl\\CurlHandle), Array)\n#9 \/var\/www\/nextcloud13\/3rdparty\/guzzle\/guzzle\/src\/Guzzle\/Http\/Curl\/CurlMulti.php(227): Guzzle\\Http\\Curl\\CurlMulti->processMessages()\n#10 \/var\/www\/nextcloud13\/3rdparty\/guzzle\/guzzle\/src\/Guzzle\/Http\/Curl\/CurlMulti.php(211): Guzzle\\Http\\Curl\\CurlMulti->executeHandles()\n#11 \/var\/www\/nextcloud13\/3rdparty\/guzzle\/guzzle\/src\/Guzzle\/Http\/Curl\/CurlMulti.php(105): Guzzle\\Http\\Curl\\CurlMulti->perform()\n#12 \/var\/www\/nextcloud13\/3rdparty\/guzzle\/guzzle\/src\/Guzzle\/Http\/Curl\/CurlMultiProxy.php(91): Guzzle\\Http\\Curl\\CurlMulti->send()\n#13 \/var\/www\/nextcloud13\/3rdparty\/guzzle\/guzzle\/src\/Guzzle\/Http\/Client.php(282): Guzzle\\Http\\Curl\\CurlMultiProxy->send()\n#14 \/var\/www\/nextcloud13\/3rdparty\/guzzle\/guzzle\/src\/Guzzle\/Http\/Message\/Request.php(198): Guzzle\\Http\\Client->send(Object(Guzzle\\Http\\Message\\EntityEnclosingRequest))\n#15 \/var\/www\/nextcloud13\/3rdparty\/rackspace\/php-opencloud\/lib\/OpenCloud\/ObjectStore\/Service.php(117): Guzzle\\Http\\Message\\Request->send()\n#16 \/var\/www\/nextcloud13\/lib\/private\/Files\/ObjectStore\/Swift.php(152): OpenCloud\\ObjectStore\\Service->createContainer('nextcloud_swift...')\n#17 \/var\/www\/nextcloud13\/lib\/private\/Files\/ObjectStore\/Swift.php(245): OC\\Files\\ObjectStore\\Swift->init()\n#18 \/var\/www\/nextcloud13\/lib\/private\/Files\/ObjectStore\/ObjectStoreStorage.php(406): OC\\Files\\ObjectStore\\Swift->writeObject('urn:oid:71460', Resource id #27)\n#19 \/var\/www\/nextcloud13\/lib\/private\/Files\/ObjectStore\/ObjectStoreStorage.php(306): OC\\Files\\ObjectStore\\ObjectStoreStorage->writeBack('\/tmp\/oc_tmp_iSV...', 'private_file')\n#20 [internal function]: OC\\Files\\ObjectStore\\ObjectStoreStorage->OC\\Files\\ObjectStore\\{closure}(*** sensitive parameters replaced ***)\n#21 \/var\/www\/nextcloud13\/3rdparty\/icewind\/streams\/src\/CallbackWrapper.php(109): call_user_func(Object(Closure))\n#22 [internal function]: Icewind\\Streams\\CallbackWrapper->stream_close()\n#23 \/var\/www\/nextcloud13\/apps\/dav\/lib\/Connector\/Sabre\/File.php(162): fclose(Resource id #25)\n#24 \/var\/www\/nextcloud13\/apps\/dav\/lib\/Connector\/Sabre\/Directory.php(151): OCA\\DAV\\Connector\\Sabre\\File->put(Resource id #21)\n#25 \/var\/www\/nextcloud13\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(1096): OCA\\DAV\\Connector\\Sabre\\Directory->createFile('private_file', Resource id #21)\n#26 \/var\/www\/nextcloud13\/3rdparty\/sabre\/dav\/lib\/DAV\/CorePlugin.php(525): Sabre\\DAV\\Server->createFile('private_file', Resource id #21, NULL)\n#27 [internal function]: Sabre\\DAV\\CorePlugin->httpPut(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#28 \/var\/www\/nextcloud13\/3rdparty\/sabre\/event\/lib\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\n#29 \/var\/www\/nextcloud13\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(479): Sabre\\Event\\EventEmitter->emit('method:PUT', Array)\n#30 \/var\/www\/nextcloud13\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(254): Sabre\\DAV\\Server->invokeMethod(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#31 \/var\/www\/nextcloud13\/apps\/dav\/lib\/Server.php(283): Sabre\\DAV\\Server->exec()\n#32 \/var\/www\/nextcloud13\/apps\/dav\/appinfo\/v2\/remote.php(35): OCA\\DAV\\Server->exec()\n#33 \/var\/www\/nextcloud13\/remote.php(164): require_once('\/var\/www\/nextcl...')\n#34 {main}","File":"\/var\/www\/nextcloud13\/3rdparty\/guzzle\/guzzle\/src\/Guzzle\/Http\/Exception\/BadResponseException.php","Line":43}

nextcloud config

{
"system": {
"objectstore": {
"class": "OC\Files\ObjectStore\Swift",
"arguments": {
"username": "xx",
"password": "REMOVED SENSITIVE VALUE",
"bucket": "nextcloud_swift",
"autocreate": true,
"region": "xx",
"url": "https://identity.stack.cloudvps.com/v2.0",
"tenantName": "xx",
"serviceName": "swift"
}
},
"passwordsalt": "REMOVED SENSITIVE VALUE",
"secret": "REMOVED SENSITIVE VALUE",
"trusted_domains": [
"DOMAIN"
],
"datadirectory": "xx",
"overwrite.cli.url": "xx",
"dbtype": "mysql",
"version": "xx",
"dbname": "xx",
"dbhost": "localhost",
"dbport": "",
"dbtableprefix": "oc_",
"dbuser": "REMOVED SENSITIVE VALUE",
"dbpassword": "REMOVED SENSITIVE VALUE",
"installed": true,
"forcessl": true,
"logtimezone": "UTC",
"trashbin_retention_obligation": "auto",
"mail_from_address": "info",
"mail_smtpmode": "smtp",
"mail_domain": "xx",
"mail_smtpsecure": "tls",
"mail_smtpauthtype": "xx",
"mail_smtpauth": 1,
"mail_smtphost": "xx",
"mail_smtpport": "xx",
"mail_smtpname": "REMOVED SENSITIVE VALUE",
"mail_smtppassword": "REMOVED SENSITIVE VALUE",
"app.mail.smtp.timeout": 10,
"memcache.local": "\OC\Memcache\Redis",
"filelocking.enabled": "true",
"memcache.distributed": "\OC\Memcache\Redis",
"memcache.locking": "\OC\Memcache\Redis",
"redis": {
"host": "localhost",
"port": 6379,
"timeout": 0,
"dbindex": 0
},
"appstore.experimental.enabled": true,
"maintenance": false,
"theme": "",
"loglevel": 0,
"updater.secret": "REMOVED SENSITIVE VALUE",
"instanceid": "xx"
}
}