Catastrophic failure removing file encryption (in CacheQueryBuilder::whereFileId)

ℹ️ Support
,
1

I have been attempting to remove server encryption from the file store of a deployment.

I have run the administrative command occ encryption:decrypt-all.

After several attempts, recovering the entire deployment from backup each time, I am confirming a failure occurs consistently for the same file.

I have further attempted the process after a sequence of operations, including files:cleanup, files:repair-tree, and files:scan, but such preparations produce no improvement.

The printed stack trace from the console is reproduced below, and then further below, a tabified rendering of the server log (through occ log:tail).

Once the error occurs, I must restore the entire deployment from backup, because even though the file store is only partially decrypted, subsequent calls to the same command for decryption report that the store is no longer encrypted, and performs no action.

The message shown for subsequent attempts to decrypt is as follows:

Server side encryption not enabled. Nothing to do

I recently upgraded the main software to recent releases, including MariaDB 10.2 (10.2.31), PHP 8.0 (8.0.26), and Nextcloud 27 (27.0.1.2)

The system is CentOS 7 (7.9.2009).

I am seeking support for completing the decryption process. The issue is exacerbated by the lack of fault tolerance, in the sense of the impossibility of recovering the instance after the failed operation.

Note that an issue has been tracked for the same problem, which has never been resolved, despite being formally closed.

An unhandled exception has been thrown:
TypeError: OC\Files\Cache\CacheQueryBuilder::whereFileId(): Argument #1 ($fileId) must be of type int, null given, called in /www/wwwroot/nextcloud/lib/private/Files/Cache/Cache.php on line 739 and defined in /www/wwwroot/nextcloud/lib/private/Files/Cache/CacheQueryBuilder.php:85
Stack trace:
#0 /www/wwwroot/nextcloud/lib/private/Files/Cache/Cache.php(739): OC\Files\Cache\CacheQueryBuilder->whereFileId()
#1 /www/wwwroot/nextcloud/lib/private/Files/Cache/Cache.php(651): OC\Files\Cache\Cache->moveFromCache()
#2 /www/wwwroot/nextcloud/lib/private/Files/Cache/Updater.php(203): OC\Files\Cache\Cache->move()
#3 /www/wwwroot/nextcloud/lib/private/Files/View.php(306): OC\Files\Cache\Updater->renameFromStorage()
#4 /www/wwwroot/nextcloud/lib/private/Files/View.php(804): OC\Files\View->renameUpdate()
#5 /www/wwwroot/nextcloud/lib/private/Encryption/DecryptAll.php(264): OC\Files\View->rename()
#6 /www/wwwroot/nextcloud/lib/private/Encryption/DecryptAll.php(228): OC\Encryption\DecryptAll->decryptFile()
#7 /www/wwwroot/nextcloud/lib/private/Encryption/DecryptAll.php(187): OC\Encryption\DecryptAll->decryptUsersFiles()
#8 /www/wwwroot/nextcloud/lib/private/Encryption/DecryptAll.php(98): OC\Encryption\DecryptAll->decryptAllUsersFiles()
#9 /www/wwwroot/nextcloud/core/Command/Encryption/DecryptAll.php(152): OC\Encryption\DecryptAll->decryptAll()
#10 /www/wwwroot/nextcloud/3rdparty/symfony/console/Command/Command.php(298): OC\Core\Command\Encryption\DecryptAll->execute()
#11 /www/wwwroot/nextcloud/3rdparty/symfony/console/Application.php(1040): Symfony\Component\Console\Command\Command->run()
#12 /www/wwwroot/nextcloud/3rdparty/symfony/console/Application.php(301): Symfony\Component\Console\Application->doRunCommand()
#13 /www/wwwroot/nextcloud/3rdparty/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun()
#14 /www/wwwroot/nextcloud/lib/private/Console/Application.php(211): Symfony\Component\Console\Application->run()
#15 /www/wwwroot/nextcloud/console.php(100): OC\Console\Application->run()
#16 /www/wwwroot/nextcloud/occ(11): require_once('...')

 ------- ------------ ---------------------------------------------- --------------------------- 
  Level   App          Message                                        Time                       
 ------- ------------ ---------------------------------------------- --------------------------- 
  Debug   encryption   /appinfo/app.php is deprecated, use            2023-08-02T14:01:03+00:00  
                       \OCP\AppFramework\Bootstrap\IBootstrap on                                 
                       the application class instead.                                            
                                                                                                 
  Debug   encryption   /appinfo/app.php is deprecated, use            2023-08-02T14:01:03+00:00  
                       \OCP\AppFramework\Bootstrap\IBootstrap on                                 
                       the application class instead.                                            
                                                                                                 
  Debug   encryption   /appinfo/app.php is deprecated, use            2023-08-02T14:01:31+00:00  
                       \OCP\AppFramework\Bootstrap\IBootstrap on                                 
                       the application class instead.                                            
                                                                                                 
  Debug   encryption   /appinfo/app.php is deprecated, use            2023-08-02T14:01:33+00:00  
                       \OCP\AppFramework\Bootstrap\IBootstrap on                                 
                       the application class instead.                                            
                                                                                                 
  Debug   encryption   /appinfo/app.php is deprecated, use            2023-08-02T14:01:49+00:00  
                       \OCP\AppFramework\Bootstrap\IBootstrap on                                 
                       the application class instead.                                            
                                                                                                 
  Debug   encryption   /appinfo/app.php is deprecated, use            2023-08-02T14:01:59+00:00  
                       \OCP\AppFramework\Bootstrap\IBootstrap on                                 
                       the application class instead.                                            
                                                                                                 
  Debug   encryption   /appinfo/app.php is deprecated, use            2023-08-02T14:02:01+00:00  
                       \OCP\AppFramework\Bootstrap\IBootstrap on                                 
                       the application class instead.                                            
                                                                                                 
  Debug   encryption   /appinfo/app.php is deprecated, use            2023-08-02T14:02:02+00:00  
                       \OCP\AppFramework\Bootstrap\IBootstrap on                                 
                       the application class instead.                                            
                                                                                                 
  Debug   encryption   /appinfo/app.php is deprecated, use            2023-08-02T14:02:08+00:00  
                       \OCP\AppFramework\Bootstrap\IBootstrap on                                 
                       the application class instead.                                            
                                                                                                 
  Debug   serverDI     The requested alias "SystemConfig" is          2023-08-02T14:02:09+00:00  
                       deprecated. Please request "OC\SystemConfig"                              
                       directly. This alias will be removed in a                                 
                       future Nextcloud version.
2

Is this a file you can temporarily move off-line and delete from your NC server prior to running occ encryption:decrypt-all? Or is the situation that the file exists in the NC’s database, but doesn’t reside on the filesystem in your data directory so there’s sort of mismatch/inconsistency? The latter is one of my suspicions because one of the main ways I can see (from the code) that unhandled exception arising is if the source file isn’t a file or isn’t readable

Others have found helpful for encryption recovery:

3

@brainchild Please do not use files:scan for encrypted files. The tool does not support the server-side encryption and might break values in the database for newly scanned files.

4

I can try to remove some files before the process, but I have no way to predict in advance whether the same problem would appear for certain other files, much less which ones in particular.

I hope that if I delete a file through the files application, then it would be removed from the file system if it exists, and also removed from the database even if it does not.

I had understood that inconsistencies between file tree and database metadata would be repaired by the other utilities I mentioned.

5

@brainchild Not when you’re using the server-side encryption. For the signature check of the server-side encryption to work properly, the encrypted field in the database needs to contain the correct file version number (which increments with every edit). The files:scan sets this field to 0, though.

As the maintainer of nextcloud/encryption-recovery-tools I’d personally suggest you give the recovery script a try as it considers edge cases that encryption:decrypt-all does not.

7

I completed decryption by removing the part of the file tree that was problematic.

However, the discussion reveals a variety of important issues.

As follows are several, listed in order of escalating relevance:

  1. Some of the administrative commands are actively destructive, if encryption is in use, but no warning is given in the documentation, or check made by the utility.
  2. The decryption feature declines to achieve a best possible recovery of the file data, by continuing to process any remaining after encountering loss.
  3. When the process discovers data loss, it fails by leaving the data set in an incoherent state. Even if the utility is not exiting gracefully, the process should have some kind of overall transactional safety. Files should be marked individually as encrypted versus not, and as long as any remain encrypted, the overall state should be considered as encryption being in use.

@jtr @yahesh