50 gb deleted for no reason deleted, signature fault when restored

ℹ️ Support
,
1

Good afternoon,

In the past 2 days my Nextcloud instance removed 50GB of data from my server. I don’t know why.
So today I noticed that and tried to restore the folders. Restoring of the files didn’t go as planned.
I can see the restored folders in the webview. Some of them with the name (restored) after it and some not. and some are duplicated. So the same folder with and without “(restored)” appended to it.

The files in the folders which have been restored (with and without “(restored)” appended to them) I cannot open.
.txt files just open as blanc files in the webview
When I try to download the files I get a HTTP ERROR 503.

In the errors I see signature failed.

I can download those files from the server(with ssh) and decrypt them myself with the corresponding encryption files. and they are correct then. but this will be a tedious thing to do with all my files.

Nextcloud version (eg, 18.0.2): 19.0.0
Nextcloud client windows : 3.0.1
Operating system and version (eg, Ubuntu 20.04): Ubuntu 20.04
Apache or nginx version (eg, Apache 2.4.25):
PHP version (eg, 7.1):

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:
I have no steps on how to replicate this

Someone an idea on how to fix the “corrupted” items? and on how to get rid of the duplicate folders?

And maybe also an answer on why the files were removed in the first place. other account/client it didn’t happen to.
There is also no logic in which folders were removed. some top folders some lower level folders.

Log file:

restoring error log(formatted):

[webdav] Fatal: Doctrine\DBAL\Exception\DriverException: An exception occurred while executing 'UPDATE `filecache` SET `storage` = ?, `path_hash` = MD5(CONCAT(?, SUBSTR(`path`, ?))), `path` = CONCAT(?, SUBSTR(`path`, ?)) WHERE (`storage` = ?) AND (`path` LIKE ?)' with params [3, "files_encryption\/keys\/files\/Niels\/Factorio (restored)", 64, "files_encryption\/keys\/files\/Niels\/Factorio (restored)", 64, 3, "files\\_encryption\/keys\/files\\_trashbin\/files\/Factorio.d1600288347\/%"]:

SQLSTATE[HY000]: General error: 1206 The total number of locks exceeds the lock table size at <<closure>>

 0. /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DBALException.php line 169
    Doctrine\DBAL\Driver\AbstractMySQLDriver->convertException("An exception oc ... e", Doctrine\DBAL\Dr ... ]})
 1. /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DBALException.php line 145
    Doctrine\DBAL\DBALException::wrapException(Doctrine\DBAL\Driver\PDOMySql\Driver {}, Doctrine\DBAL\Dr ... ]}, "An exception oc ... e")
 2. /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php line 1063
    Doctrine\DBAL\DBALException::driverExceptionDuringQuery(Doctrine\DBAL\Driver\PDOMySql\Driver {}, Doctrine\DBAL\Dr ... ]}, "UPDATE `filecac ... )", {1: 3,2: "files_ ... "})
 3. /var/www/html/lib/private/DB/Connection.php line 214
    Doctrine\DBAL\Connection->executeUpdate("UPDATE `filecac ... )", [3,"files_encryp ... "], [1,2,1,2,1,1,2])
 4. /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Query/QueryBuilder.php line 203
    OC\DB\Connection->executeUpdate("UPDATE `filecac ... )", {dcValue1: "file ... "}, {dcValue1: 2,dcV ... 2})
 5. /var/www/html/lib/private/DB/QueryBuilder/QueryBuilder.php line 216
    Doctrine\DBAL\Query\QueryBuilder->execute()
 6. /var/www/html/lib/private/Files/Cache/Cache.php line 647
    OC\DB\QueryBuilder\QueryBuilder->execute()
 7. /var/www/html/lib/private/Files/Cache/Cache.php line 585
    OC\Files\Cache\Cache->moveFromCache(OC\Files\Cache\HomeCache {}, "files_encryptio ... 7", "files_encryptio ... )")
 8. /var/www/html/lib/private/Files/Cache/Updater.php line 196
    OC\Files\Cache\Cache->move("files_encryptio ... 7", "files_encryptio ... )")
 9. /var/www/html/lib/private/Files/View.php line 335
    OC\Files\Cache\Updater->renameFromStorage(OCA\Files_Trashb ... }}, "files_encryptio ... 7", "files_encryptio ... )")
10. /var/www/html/lib/private/Files/View.php line 830
    OC\Files\View->renameUpdate(OCA\Files_Trashb ... }}, OCA\Files_Trashb ... }}, "files_encryptio ... 7", "files_encryptio ... )")
11. /var/www/html/lib/private/Encryption/Keys/Storage.php line 277
    OC\Files\View->rename("/niels/files_en ... 7", "/niels/files_en ... )")
12. /var/www/html/lib/private/Files/Storage/Wrapper/Encryption.php line 293
    OC\Encryption\Keys\Storage->renameKeys("/niels/files_tr ... 7", "/niels/files/Niels/Factorio (restored)")
13. /var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php line 278
    OC\Files\Storage\Wrapper\Encryption->rename("files_trashbin/ ... 7", "files/Niels/Factorio (restored)")
14. /var/www/html/lib/private/Files/View.php line 816
    OC\Files\Storage\Wrapper\Wrapper->rename("files_trashbin/ ... 7", "files/Niels/Factorio (restored)")
15. /var/www/html/apps/files_trashbin/lib/Trashbin.php line 452
    OC\Files\View->rename("/files_trashbin ... 7", "/files/Niels/Factorio (restored)")
16. /var/www/html/apps/files_trashbin/lib/Trash/LegacyTrashBackend.php line 85
    OCA\Files_Trashbin\Trashbin::restore("/Factorio.d1600288347", "Factorio", 1600288347)
17. /var/www/html/apps/files_trashbin/lib/Trash/TrashManager.php line 65
    OCA\Files_Trashbin\Trash\LegacyTrashBackend->restoreItem(OCA\Files_Trashbin\Trash\TrashItem {})
18. /var/www/html/apps/files_trashbin/lib/Sabre/AbstractTrash.php line 94
    OCA\Files_Trashbin\Trash\TrashManager->restoreItem(OCA\Files_Trashbin\Trash\TrashItem {})
19. /var/www/html/apps/files_trashbin/lib/Sabre/RestoreFolder.php line 76
    OCA\Files_Trashbin\Sabre\AbstractTrash->restore()
20. /var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php line 164
    OCA\Files_Trashbin\Sabre\RestoreFolder->moveInto("Factorio.d1600288347", "trashbin/niels/ ... 7", OCA\Files_Trashbin\Sabre\TrashFolder {})
21. /var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php line 641
    Sabre\DAV\Tree->move("trashbin/niels/ ... 7", "trashbin/niels/ ... 7")
22. /var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php line 89
    Sabre\DAV\CorePlugin->httpMove(Sabre\HTTP\Request {}, Sabre\HTTP\Response {})
23. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 474
    Sabre\DAV\Server->emit("method:MOVE", [Sabre\HTTP\Requ ... }])
24. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 251
    Sabre\DAV\Server->invokeMethod(Sabre\HTTP\Request {}, Sabre\HTTP\Response {})
25. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 319
    Sabre\DAV\Server->start()
26. /var/www/html/apps/dav/lib/Server.php line 320
    Sabre\DAV\Server->exec()
27. /var/www/html/apps/dav/appinfo/v2/remote.php line 35
    OCA\DAV\Server->exec()
28. /var/www/html/remote.php line 167
    require_once("/var/www/html/a ... p")

MOVE /remote.php/dav/trashbin/niels/trash/Factorio.d1600288347
from 86.94.249.172 by niels at 2020-09-18T10:57:46+00:00
2

Hey,
I think that your database is corrupted. Try the occ commands to repair the database and scan files

3

Thanks
I didn’t know about these commands. I will try them this weekend.

4

The repair commands had not much todo.

the files:scan doensn’t work/exist in the installation.

Any other options? I personally think that the client on my computer failed.

repair output:

 - Repair MySQL collation
     - All tables already have the correct collation -> nothing to do
 - Repair mime types
 - Clean tags and favorites
     - 0 tags of deleted users have been removed.
     - 1 tags for delete files have been removed.
     - 0 tag entries for deleted tags have been removed.
     - 1 tags with no entries have been removed.
 - Repair invalid shares
 - Move .step file of updater to backup location
 - Fix potential broken mount points
     - No mounts updated
 - Add log rotate job
 - Clear frontend caches
     - Image cache cleared
     - SCSS cache cleared
     - JS cache cleared
 - Clear every generated avatar on major updates
 - Add preview background cleanup job
 - Queue a one-time job to cleanup old backups of the updater
 - Cleanup invalid photocache files for carddav
 - Add background job to cleanup login flow v2 tokens
 - Remove potentially over exposing share links
     - No need to remove link shares.
 - Clear access cache of projects
 - Reset generated avatar flag
 - Fix component of birthday calendars
     - 6 birthday calendars updated.
 - Regenerating birthday calendars to use new icons and fix old birthday events without year
     - Repair step already executed
 - Fix broken values of calendar objects
    0 [->--------------------------]
 - Registering building of calendar search index as background job
     - Repair step already executed
 - Registering background jobs to update cache for webcal calendars
     - Added 0 background jobs to update webcal calendars
 - Registering building of calendar reminder index as background job
     - Repair step already executed
 - Clean up orphan event and contact data
     - 0 events without a calendar have been cleaned up
     - 0 properties without an events have been cleaned up
     - 0 changes without a calendar have been cleaned up
     - 0 cached events without a calendar subscription have been cleaned up
     - 0 changes without a calendar subscription have been cleaned up
     - 0 contacts without an addressbook have been cleaned up
     - 0 properties without a contact have been cleaned up
     - 0 changes without an addressbook have been cleaned up
 - Remove activity entries of private events
     - Removed 0 activity entries
 - Write default encryption module configuration to the database
 - Copy default images to the app data directory
     - Icon %s already exists
     - Icon %s already exists
     - Icon %s already exists
 - Fix the share type of guest shares when migrating from ownCloud
 - Copy the share password into the dedicated column
 - Set existing shares as accepted
 - Update OAuth token expiration times
 - Switches from default updater server to the customer one if a valid subscription is available
     - Repair step already executed
 - Add background job to check for backup codes
 - Populating added database structures for workflows

occ help
occ list:

 Nextcloud 19.0.0

Usage:
  command [options] [arguments]

Options:
  -h, --help            Display this help message
  -q, --quiet           Do not output any message
  -V, --version         Display this application version
      --ansi            Force ANSI output
      --no-ansi         Disable ANSI output
  -n, --no-interaction  Do not ask any interactive question
      --no-warnings     Skip global warnings, show command output only
  -v|vv|vvv, --verbose  Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug

Available commands:
  check                               check dependencies of the server environment
  help                                Displays help for a command
  list                                Lists commands
  status                              show some status information
  upgrade                             run upgrade routines after installation of a new release. The release has to be installed before.
 app
  app:check-code                      check code to be compliant
  app:disable                         disable an app
  app:enable                          enable an app
  app:getpath                         Get an absolute path to the app directory
  app:install                         install an app
  app:list                            List all available apps
  app:remove                          remove an app
  app:update                          update an app or all apps
 background
  background:ajax                     Use ajax to run background jobs
  background:cron                     Use cron to run background jobs
  background:webcron                  Use webcron to run background jobs
 broadcast
  broadcast:test                      test the SSE broadcaster
 config
  config:app:delete                   Delete an app config value
  config:app:get                      Get an app config value
  config:app:set                      Set an app config value
  config:import                       Import a list of configs
  config:list                         List all configs
  config:system:delete                Delete a system config value
  config:system:get                   Get a system config value
  config:system:set                   Set a system config value
 db
  db:add-missing-columns              Add missing optional columns to the database tables
  db:add-missing-indices              Add missing indices to the database tables
  db:convert-filecache-bigint         Convert the ID columns of the filecache to BigInt
  db:convert-mysql-charset            Convert charset of MySQL/MariaDB to use utf8mb4
  db:convert-type                     Convert the Nextcloud database to the newly configured one
 encryption
  encryption:change-key-storage-root  Change key storage root
  encryption:decrypt-all              Disable server-side encryption and decrypt all files
  encryption:disable                  Disable encryption
  encryption:enable                   Enable encryption
  encryption:encrypt-all              Encrypt all files for all users
  encryption:list-modules             List all available encryption modules
  encryption:set-default-module       Set the encryption default module
  encryption:show-key-storage-root    Show current key storage root
  encryption:status                   Lists the current status of encryption
 group
  group:add                           Add a group
  group:adduser                       add a user to a group
  group:delete                        Remove a group
  group:list                          list configured groups
  group:removeuser                    remove a user from a group
 integrity
  integrity:check-app                 Check integrity of an app using a signature.
  integrity:check-core                Check integrity of core code using a signature.
  integrity:sign-app                  Signs an app using a private key.
  integrity:sign-core                 Sign core using a private key.
 l10n
  l10n:createjs                       Create javascript translation files for a given app
 log
  log:file                            manipulate logging backend
  log:manage                          manage logging configuration
 maintenance
  maintenance:data-fingerprint        update the systems data-fingerprint after a backup is restored
  maintenance:mimetype:update-db      Update database mimetypes and update filecache
  maintenance:mimetype:update-js      Update mimetypelist.js
  maintenance:mode                    set maintenance mode
  maintenance:repair                  repair this installation
  maintenance:theme:update            Apply custom theme changes
  maintenance:update:htaccess         Updates the .htaccess file
 migrations
  migrations:execute                  Execute a single migration version manually.
  migrations:generate
  migrations:generate-from-schema
  migrations:migrate                  Execute a migration to a specified version or the latest available version.
  migrations:status                   View the status of a set of migrations.
 security
  security:certificates               list trusted certificates
  security:certificates:import        import trusted certificate
  security:certificates:remove        remove trusted certificate
 twofactorauth
  twofactorauth:cleanup               Clean up the two-factor user-provider association of an uninstalled/removed provider
  twofactorauth:disable               Disable two-factor authentication for a user
  twofactorauth:enable                Enable two-factor authentication for a user
  twofactorauth:enforce               Enabled/disable enforced two-factor authentication
  twofactorauth:state                 Get the two-factor authentication (2FA) state of a user
 user
  user:add                            adds a user
  user:delete                         deletes the specified user
  user:disable                        disables the specified user
  user:enable                         enables the specified user
  user:info                           show user info
  user:lastseen                       shows when the user was logged in last time
  user:list                           list configured users
  user:report                         shows how many users have access
  user:resetpassword                  Resets the password of the named user
  user:setting                        Read and modify user settings
5

The encryption app makes restores very complicated and you need the right database dump to the data set. For that reason it is important, not only to make backups but as well test and do restores. There are a few topics here and on github. The problem is that the files are signed with information from the database. If you don’t have the database, some managed to ignore a wrong signature to be able to decrypt the file.

6

Yea I was able to manually decrypt a folder. I think the rest will work to. but I would have to find out what is duplicate and what not. and if i have everything. it seems like i will have to just decrypt everything and then restore the files. I was still setting up backups… my last backup is a few months old. As i went to a new system 2 months ago.

//edit
It is decrypting everything as we speak.
For that I use: https://github.com/syseleven/nextcloud-tools
i’ll just copy all the files over my existing files and reupload everything