Hi,
I have been using nextcloud through Docker for years now.
I started using it with raid divices and it worked fine. Because I change my hardware last year, I migrated all the data from local raid to local Minio S3.
Minio is working fine. The migration whent fine.
Since then, It worekd mostly fine, but I saw weared things happenning:
- It took me some time to understand that occ files:scan does not work with S3.
- There is no way to recolect the database and the data in the S3 storage witch lead to error messages saying file does not exist, is corrupted or unknown error. here is an exemple:
{
"reqId": "276khjKeG0DuDBs0K2QQ",
"level": 3,
"time": "2024-03-04 16:20:34",
"remoteAddr": "",
"user": "--",
"app": "objectstore",
"method": "",
"url": "/cron.php",
"message": "Could not delete object urn:oid:2078085 for files_trashbin/files/IMG_20230131_043452.jpg.d1702237217",
"userAgent": "--",
"version": "27.1.6.2",
"exception": {
"Exception": "Aws\\S3\\Exception\\S3Exception",
"Message": "Error executing \"DeleteObject\" on \"https://s3.site.fr/nextcloud/urn%3Aoid%3A2078085\"; AWS HTTP error: Server error: `DELETE https://s3.site.fr/nextcloud/urn%3Aoid%3A2078085` resulted in a `500 Internal Server Error` response:\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, pleas (truncated...)\n InternalError (server): We encountered an internal error, please try again.: cause(file is corrupted) - <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(file is corrupted)</Message><Key>urn:oid:2078085</Key><BucketName>nextcloud</BucketName><Resource>/nextcloud/urn:oid:2078085</Resource><Region>fr-paris-07</Region><RequestId>17B99C2920CB70D2</RequestId><HostId>46efbbb7efbd81c7d995bde03cc6fabf60c12f80d4e074c1c972dbc4d583c3d4</HostId></Error>",
"Code": 0,
"Trace": [
{
"file": "/var/www/html/3rdparty/aws/aws-sdk-php/src/WrappedHttpHandler.php",
"line": 96,
"function": "parseError",
"class": "Aws\\WrappedHttpHandler",
"type": "->",
"args": [
"*** sensitive parameters replaced ***",
[
"GuzzleHttp\\Psr7\\Request"
],
[
"Aws\\Command"
],
[]
]
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php",
"line": 204,
"function": "Aws\\{closure}",
"class": "Aws\\WrappedHttpHandler",
"type": "->",
"args": [
"*** sensitive parameters replaced ***"
]
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php",
"line": 169,
"function": "callHandler",
"class": "GuzzleHttp\\Promise\\Promise",
"type": "::",
"args": [
2,
"*** sensitive parameters replaced ***",
null
]
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/promises/src/RejectedPromise.php",
"line": 42,
"function": "GuzzleHttp\\Promise\\{closure}",
"class": "GuzzleHttp\\Promise\\Promise",
"type": "::",
"args": [
"*** sensitive parameters replaced ***"
]
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/promises/src/TaskQueue.php",
"line": 48,
"function": "GuzzleHttp\\Promise\\{closure}",
"class": "GuzzleHttp\\Promise\\RejectedPromise",
"type": "::",
"args": [
"*** sensitive parameters replaced ***"
]
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php",
"line": 159,
"function": "run",
"class": "GuzzleHttp\\Promise\\TaskQueue",
"type": "->",
"args": []
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php",
"line": 184,
"function": "tick",
"class": "GuzzleHttp\\Handler\\CurlMultiHandler",
"type": "->",
"args": []
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php",
"line": 248,
"function": "execute",
"class": "GuzzleHttp\\Handler\\CurlMultiHandler",
"type": "->",
"args": [
true
]
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php",
"line": 224,
"function": "invokeWaitFn",
"class": "GuzzleHttp\\Promise\\Promise",
"type": "->",
"args": []
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php",
"line": 269,
"function": "waitIfPending",
"class": "GuzzleHttp\\Promise\\Promise",
"type": "->",
"args": []
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php",
"line": 226,
"function": "invokeWaitList",
"class": "GuzzleHttp\\Promise\\Promise",
"type": "->",
"args": []
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php",
"line": 269,
"function": "waitIfPending",
"class": "GuzzleHttp\\Promise\\Promise",
"type": "->",
"args": []
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php",
"line": 226,
"function": "invokeWaitList",
"class": "GuzzleHttp\\Promise\\Promise",
"type": "->",
"args": []
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php",
"line": 62,
"function": "waitIfPending",
"class": "GuzzleHttp\\Promise\\Promise",
"type": "->",
"args": []
},
{
"file": "/var/www/html/3rdparty/aws/aws-sdk-php/src/AwsClientTrait.php",
"line": 58,
"function": "wait",
"class": "GuzzleHttp\\Promise\\Promise",
"type": "->",
"args": []
},
{
"file": "/var/www/html/3rdparty/aws/aws-sdk-php/src/AwsClientTrait.php",
"line": 86,
"function": "execute",
"class": "Aws\\AwsClient",
"type": "->",
"args": [
[
"Aws\\Command"
]
]
},
{
"file": "/var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php",
"line": 183,
"function": "__call",
"class": "Aws\\AwsClient",
"type": "->",
"args": [
"deleteObject",
[
[
"nextcloud",
"urn:oid:2078085"
]
]
]
},
{
"file": "/var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php",
"line": 225,
"function": "deleteObject",
"class": "OC\\Files\\ObjectStore\\S3",
"type": "->",
"args": [
"urn:oid:2078085"
]
},
{
"file": "/var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php",
"line": 217,
"function": "rmObject",
"class": "OC\\Files\\ObjectStore\\ObjectStoreStorage",
"type": "->",
"args": [
[
"OC\\Files\\Cache\\CacheEntry"
]
]
},
{
"file": "/var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php",
"line": 265,
"function": "unlink",
"class": "OC\\Files\\ObjectStore\\ObjectStoreStorage",
"type": "->",
"args": [
"files_trashbin/files/IMG_20230131_043452.jpg.d1702237217"
]
},
{
"file": "/var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php",
"line": 265,
"function": "unlink",
"class": "OC\\Files\\Storage\\Wrapper\\Wrapper",
"type": "->",
"args": [
"files_trashbin/files/IMG_20230131_043452.jpg.d1702237217"
]
},
{
"function": "unlink",
"class": "OC\\Files\\Storage\\Wrapper\\Wrapper",
"type": "->",
"args": [
"files_trashbin/files/IMG_20230131_043452.jpg.d1702237217"
]
},
{
"file": "/var/www/html/apps/files_trashbin/lib/Storage.php",
"line": 193,
"function": "call_user_func",
"args": [
[
[
"OCA\\Files_Antivirus\\AvirWrapper",
[
"OC\\Files\\Cache\\Cache"
],
null,
null,
null,
null
],
"unlink"
],
"files_trashbin/files/IMG_20230131_043452.jpg.d1702237217"
]
},
{
"file": "/var/www/html/apps/files_trashbin/lib/Storage.php",
"line": 102,
"function": "doDelete",
"class": "OCA\\Files_Trashbin\\Storage",
"type": "->",
"args": [
"files_trashbin/files/IMG_20230131_043452.jpg.d1702237217",
"unlink"
]
},
{
"file": "/var/www/html/lib/private/Files/View.php",
"line": 1161,
"function": "unlink",
"class": "OCA\\Files_Trashbin\\Storage",
"type": "->",
"args": [
"files_trashbin/files/IMG_20230131_043452.jpg.d1702237217"
]
},
{
"file": "/var/www/html/lib/private/Files/View.php",
"line": 702,
"function": "basicOperation",
"class": "OC\\Files\\View",
"type": "->",
"args": [
"unlink",
"/user1/files_trashbin/files/IMG_20230131_043452.jpg.d1702237217",
[
"delete"
]
]
},
{
"file": "/var/www/html/lib/private/Files/Node/File.php",
"line": 133,
"function": "unlink",
"class": "OC\\Files\\View",
"type": "->",
"args": [
"/user1/files_trashbin/files/IMG_20230131_043452.jpg.d1702237217"
]
},
{
"file": "/var/www/html/apps/files_trashbin/lib/Trashbin.php",
"line": 695,
"function": "delete",
"class": "OC\\Files\\Node\\File",
"type": "->",
"args": []
},
{
"file": "/var/www/html/apps/files_trashbin/lib/Trashbin.php",
"line": 908,
"function": "delete",
"class": "OCA\\Files_Trashbin\\Trashbin",
"type": "::",
"args": [
"IMG_20230131_043452.jpg",
"user1",
1702237217
]
},
{
"file": "/var/www/html/apps/files_trashbin/lib/BackgroundJob/ExpireTrash.php",
"line": 80,
"function": "deleteExpiredFiles",
"class": "OCA\\Files_Trashbin\\Trashbin",
"type": "::",
"args": [
[
[
"OC\\Files\\FileInfo"
]
],
"user1"
]
},
{
"file": "/var/www/html/lib/private/User/Manager.php",
"line": 632,
"function": "OCA\\Files_Trashbin\\BackgroundJob\\{closure}",
"class": "OCA\\Files_Trashbin\\BackgroundJob\\ExpireTrash",
"type": "->",
"args": [
"*** sensitive parameters replaced ***"
]
},
{
"file": "/var/www/html/apps/files_trashbin/lib/BackgroundJob/ExpireTrash.php",
"line": 74,
"function": "callForSeenUsers",
"class": "OC\\User\\Manager",
"type": "->",
"args": [
[
"Closure"
]
]
},
{
"file": "/var/www/html/lib/public/BackgroundJob/Job.php",
"line": 81,
"function": "run",
"class": "OCA\\Files_Trashbin\\BackgroundJob\\ExpireTrash",
"type": "->",
"args": [
null
]
},
{
"file": "/var/www/html/lib/public/BackgroundJob/TimedJob.php",
"line": 103,
"function": "start",
"class": "OCP\\BackgroundJob\\Job",
"type": "->",
"args": [
[
"OC\\BackgroundJob\\JobList"
]
]
},
{
"file": "/var/www/html/lib/public/BackgroundJob/TimedJob.php",
"line": 93,
"function": "start",
"class": "OCP\\BackgroundJob\\TimedJob",
"type": "->",
"args": [
[
"OC\\BackgroundJob\\JobList"
]
]
},
{
"file": "/var/www/html/cron.php",
"line": 152,
"function": "execute",
"class": "OCP\\BackgroundJob\\TimedJob",
"type": "->",
"args": [
[
"OC\\BackgroundJob\\JobList"
],
[
"OC\\Log"
]
]
}
],
"File": "/var/www/html/3rdparty/aws/aws-sdk-php/src/WrappedHttpHandler.php",
"Line": 195,
"Previous": {
"Exception": "GuzzleHttp\\Exception\\ServerException",
"Message": "Server error: `DELETE https://s3.site.fr/nextcloud/urn%3Aoid%3A2078085` resulted in a `500 Internal Server Error` response:\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, pleas (truncated...)\n",
"Code": 500,
"Trace": [
{
"file": "/var/www/html/3rdparty/guzzlehttp/guzzle/src/Middleware.php",
"line": 69,
"function": "create",
"class": "GuzzleHttp\\Exception\\RequestException",
"type": "::",
"args": [
"*** sensitive parameters replaced ***"
]
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php",
"line": 204,
"function": "GuzzleHttp\\{closure}",
"class": "GuzzleHttp\\Middleware",
"type": "::",
"args": [
"*** sensitive parameters replaced ***"
]
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php",
"line": 153,
"function": "callHandler",
"class": "GuzzleHttp\\Promise\\Promise",
"type": "::",
"args": [
1,
"*** sensitive parameters replaced ***",
"*** sensitive parameters replaced ***"
]
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/promises/src/TaskQueue.php",
"line": 48,
"function": "GuzzleHttp\\Promise\\{closure}",
"class": "GuzzleHttp\\Promise\\Promise",
"type": "::",
"args": [
"*** sensitive parameters replaced ***"
]
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php",
"line": 159,
"function": "run",
"class": "GuzzleHttp\\Promise\\TaskQueue",
"type": "->",
"args": []
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php",
"line": 184,
"function": "tick",
"class": "GuzzleHttp\\Handler\\CurlMultiHandler",
"type": "->",
"args": []
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php",
"line": 248,
"function": "execute",
"class": "GuzzleHttp\\Handler\\CurlMultiHandler",
"type": "->",
"args": [
true
]
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php",
"line": 224,
"function": "invokeWaitFn",
"class": "GuzzleHttp\\Promise\\Promise",
"type": "->",
"args": []
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php",
"line": 269,
"function": "waitIfPending",
"class": "GuzzleHttp\\Promise\\Promise",
"type": "->",
"args": []
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php",
"line": 226,
"function": "invokeWaitList",
"class": "GuzzleHttp\\Promise\\Promise",
"type": "->",
"args": []
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php",
"line": 269,
"function": "waitIfPending",
"class": "GuzzleHttp\\Promise\\Promise",
"type": "->",
"args": []
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php",
"line": 226,
"function": "invokeWaitList",
"class": "GuzzleHttp\\Promise\\Promise",
"type": "->",
"args": []
},
{
"file": "/var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php",
"line": 62,
"function": "waitIfPending",
"class": "GuzzleHttp\\Promise\\Promise",
"type": "->",
"args": []
},
{
"file": "/var/www/html/3rdparty/aws/aws-sdk-php/src/AwsClientTrait.php",
"line": 58,
"function": "wait",
"class": "GuzzleHttp\\Promise\\Promise",
"type": "->",
"args": []
},
{
"file": "/var/www/html/3rdparty/aws/aws-sdk-php/src/AwsClientTrait.php",
"line": 86,
"function": "execute",
"class": "Aws\\AwsClient",
"type": "->",
"args": [
[
"Aws\\Command"
]
]
},
{
"file": "/var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php",
"line": 183,
"function": "__call",
"class": "Aws\\AwsClient",
"type": "->",
"args": [
"deleteObject",
[
[
"nextcloud",
"urn:oid:2078085"
]
]
]
},
{
"file": "/var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php",
"line": 225,
"function": "deleteObject",
"class": "OC\\Files\\ObjectStore\\S3",
"type": "->",
"args": [
"urn:oid:2078085"
]
},
{
"file": "/var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php",
"line": 217,
"function": "rmObject",
"class": "OC\\Files\\ObjectStore\\ObjectStoreStorage",
"type": "->",
"args": [
[
"OC\\Files\\Cache\\CacheEntry"
]
]
},
{
"file": "/var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php",
"line": 265,
"function": "unlink",
"class": "OC\\Files\\ObjectStore\\ObjectStoreStorage",
"type": "->",
"args": [
"files_trashbin/files/IMG_20230131_043452.jpg.d1702237217"
]
},
{
"file": "/var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php",
"line": 265,
"function": "unlink",
"class": "OC\\Files\\Storage\\Wrapper\\Wrapper",
"type": "->",
"args": [
"files_trashbin/files/IMG_20230131_043452.jpg.d1702237217"
]
},
{
"function": "unlink",
"class": "OC\\Files\\Storage\\Wrapper\\Wrapper",
"type": "->",
"args": [
"files_trashbin/files/IMG_20230131_043452.jpg.d1702237217"
]
},
{
"file": "/var/www/html/apps/files_trashbin/lib/Storage.php",
"line": 193,
"function": "call_user_func",
"args": [
[
[
"OCA\\Files_Antivirus\\AvirWrapper",
[
"OC\\Files\\Cache\\Cache"
],
"*** sensitive parameters replaced ***",
"*** sensitive parameters replaced ***",
"*** sensitive parameters replaced ***",
"*** sensitive parameters replaced ***"
],
"unlink"
],
"files_trashbin/files/IMG_20230131_043452.jpg.d1702237217"
]
},
{
"file": "/var/www/html/apps/files_trashbin/lib/Storage.php",
"line": 102,
"function": "doDelete",
"class": "OCA\\Files_Trashbin\\Storage",
"type": "->",
"args": [
"files_trashbin/files/IMG_20230131_043452.jpg.d1702237217",
"unlink"
]
},
{
"file": "/var/www/html/lib/private/Files/View.php",
"line": 1161,
"function": "unlink",
"class": "OCA\\Files_Trashbin\\Storage",
"type": "->",
"args": [
"files_trashbin/files/IMG_20230131_043452.jpg.d1702237217"
]
},
{
"file": "/var/www/html/lib/private/Files/View.php",
"line": 702,
"function": "basicOperation",
"class": "OC\\Files\\View",
"type": "->",
"args": [
"unlink",
"/user1/files_trashbin/files/IMG_20230131_043452.jpg.d1702237217",
[
"delete"
]
]
},
{
"file": "/var/www/html/lib/private/Files/Node/File.php",
"line": 133,
"function": "unlink",
"class": "OC\\Files\\View",
"type": "->",
"args": [
"/user1/files_trashbin/files/IMG_20230131_043452.jpg.d1702237217"
]
},
{
"file": "/var/www/html/apps/files_trashbin/lib/Trashbin.php",
"line": 695,
"function": "delete",
"class": "OC\\Files\\Node\\File",
"type": "->",
"args": []
},
{
"file": "/var/www/html/apps/files_trashbin/lib/Trashbin.php",
"line": 908,
"function": "delete",
"class": "OCA\\Files_Trashbin\\Trashbin",
"type": "::",
"args": [
"IMG_20230131_043452.jpg",
"user1",
1702237217
]
},
{
"file": "/var/www/html/apps/files_trashbin/lib/BackgroundJob/ExpireTrash.php",
"line": 80,
"function": "deleteExpiredFiles",
"class": "OCA\\Files_Trashbin\\Trashbin",
"type": "::",
"args": [
[
[
"OC\\Files\\FileInfo"
]
],
"user1"
]
},
{
"file": "/var/www/html/lib/private/User/Manager.php",
"line": 632,
"function": "OCA\\Files_Trashbin\\BackgroundJob\\{closure}",
"class": "OCA\\Files_Trashbin\\BackgroundJob\\ExpireTrash",
"type": "->",
"args": [
"*** sensitive parameters replaced ***"
]
},
{
"file": "/var/www/html/apps/files_trashbin/lib/BackgroundJob/ExpireTrash.php",
"line": 74,
"function": "callForSeenUsers",
"class": "OC\\User\\Manager",
"type": "->",
"args": [
[
"Closure"
]
]
},
{
"file": "/var/www/html/lib/public/BackgroundJob/Job.php",
"line": 81,
"function": "run",
"class": "OCA\\Files_Trashbin\\BackgroundJob\\ExpireTrash",
"type": "->",
"args": [
"*** sensitive parameters replaced ***"
]
},
{
"file": "/var/www/html/lib/public/BackgroundJob/TimedJob.php",
"line": 103,
"function": "start",
"class": "OCP\\BackgroundJob\\Job",
"type": "->",
"args": [
[
"OC\\BackgroundJob\\JobList"
]
]
},
{
"file": "/var/www/html/lib/public/BackgroundJob/TimedJob.php",
"line": 93,
"function": "start",
"class": "OCP\\BackgroundJob\\TimedJob",
"type": "->",
"args": [
[
"OC\\BackgroundJob\\JobList"
]
]
},
{
"file": "/var/www/html/cron.php",
"line": 152,
"function": "execute",
"class": "OCP\\BackgroundJob\\TimedJob",
"type": "->",
"args": [
[
"OC\\BackgroundJob\\JobList"
],
[
"OC\\Log"
]
]
}
],
"File": "/var/www/html/3rdparty/guzzlehttp/guzzle/src/Exception/RequestException.php",
"Line": 113
},
"message": "Could not delete object urn:oid:2078085 for files_trashbin/files/IMG_20230131_043452.jpg.d1702237217",
"CustomMessage": "Could not delete object urn:oid:2078085 for files_trashbin/files/IMG_20230131_043452.jpg.d1702237217"
}
}
- The process to make sure the data is well Written and sync with the database seems not robust enought. From my client (mainly android), I sometime get an error saying that the file could not be written, then I start an infinit loop to try to write the file. To avoid getting the infinit loop message, I am force to ingore it or to delete the file.
- Antivirus ClamAV and S3 primary storage does not work.
- The database grows.
Because of the amount of data, It difficult to be certain that all the data is there. Because files are not only in one places, I do not care to much for now. But I start to feel uncertain about what is going on with my files.
So the question is: Do you think Nextcloud with S3 primary storage is mature enought to be used? What are your thought on this?
Regards
Pierre