Errors while deleting & moving files on external storage

Nextcloud version: 13.0.5.2
Operating system and version: Ubuntu 18.04
Apache or nginx version: (using snap image)
PHP version: (using snap image)

The issue you are facing:
Yesterday I setup an external WebDAV storage on my nextcloud instance. It holds much more storage capacity than my Nextcloud instance itself so I decided to move my ~15GB of pictures onto the external storage ‘folder’ that Nextcloud had created.
To see if things worked as expected I started by moving a small folder with roughly 500MB worth of pictures to the external storage. This took some time, as expected, but worked perfectly. I could see that the folder had been ‘removed’ from my Nextcloud storage and appeared as expected in the external storage. File size and everything was correct.
Then I started to move over the ~15GB worth of files to the external storage. I let it do it’s thing and after a while I could see all the pictures were now on the external storage. But this is where things get weird. The folder in my Nextcloud instance that held the images appeared to still be there, unchanged (unmoved).
However, when I SSH-ed into the server I found that the images were in fact gone. Trying to download any image in the web interface resulted in a 404. I found out that I had the option to use the nextcloud.occ files:scan --all command to trigger a rescan. And sure enough, that made everything work as intended again, I could now see the image folder only in the external storage and no longer on the Nextcloud instance.

Another bug that I encountered was when I almost deleted the ~15GB worth of images. So I was at the point where I thought that the move had failed because the Nextcloud folder still held the images in it. What I did was I deleted the pictures on the external storage to retry the move operation.
However, to my shock, during this time I found out that the pictures were also no longer in the Nextcloud instance when SSH-ing into the machine. Freaking out over the fact that I at this moment was deleting nearly 5 years worth of pictures I immediately rebooted my server to stop the process.
When I checked the external storage it seemed like nothing had been deleted. So I started an experiment where I copied a file on the external storage and then deleted it. And lo-and-behold the nextcloud log was spammed with errors (shared below). However the file did get deleted in the end.

TL;DR:

  1. Moving large folders to external storage causes Nextcloud to not update the ‘current’ state of files.
  2. Deleting files on external storage causes errors but deletes the file nonetheless.

Is this the first time you’ve seen this error?: Yes

Steps to replicate it:

  1. Hook up a WebDAV external storage
  2. Move over a large amount of files (not sure on size, ~15GB worked for me) to the external storage
  3. Look at the Nextcloud folder that had the files in it. They should still be there.

The output of your Nextcloud log in Admin > Logging:
Link to the JSON. Note that these errors are for 1 file being deleted, and were repeated at least 50 times during the operation.

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'apps_paths' => 
  array (
    0 => 
    array (
      'path' => '/snap/nextcloud/current/htdocs/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 => 
    array (
      'path' => '/var/snap/nextcloud/current/nextcloud/extra-apps',
      'url' => '/extra-apps',
      'writable' => true,
    ),
  ),
  'supportedDatabases' => 
  array (
    0 => 'mysql',
  ),
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'memcache.local' => '\\OC\\Memcache\\Redis',
  'redis' => 
  array (
    'host' => '/tmp/sockets/redis.sock',
    'port' => 0,
  ),
  'passwordsalt' => 'REDACTED',
  'secret' => 'REDACTED',
  'trusted_domains' => 
  array (
    0 => 'localhost',
    1 => 'nextcloud.homs.codes',
  ),
  'datadirectory' => '/var/snap/nextcloud/common/nextcloud/data',
  'overwrite.cli.url' => 'http://localhost',
  'dbtype' => 'mysql',
  'version' => '13.0.5.2',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost:/tmp/sockets/mysql.sock',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud',
  'dbpassword' => 'REDACTED',
  'installed' => true,
  'instanceid' => 'ocdcds1ms1mu',
  'mail_from_address' => 'REDACTED',
  'mail_smtpmode' => 'php',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_domain' => 'REDACTED',
);

The output of your Apache/nginx/system log in /var/log/____:
No logs were appended for Apache/systemlog during any of the operations.

Appending this issue with more weird behavior. I’m trying to move back all the pictures to my Nextcloud instance from the external storage and it doesn’t work. When tailing the Nextcloud logs I see a whole lot of error messages appear but no files are actually being transferred.
Here’s a link to the error messages that are appearing while trying to move the pictures back. Note that the logs are being spammed with these messages, hundreds of times per minute.

As of right now I’m unable to access any of the pictures as they are encrypted on the external storage and seem to be unable to be moved or accessed.

Found some more information regarding issues with external storage. I tried to download a single image from the external storage and the download simply fails. It does start downloading but eventually just stops with the “Failed” error message. (using Firefox)
If I look at the logs I see the same behavior as the previous 2 logs I posted at first, although at the end of the log I can see it can’t decrypt the file due to a “Missing Signature”. Then after that it tries to set some HTTP headers and fails because it has already sent the response. The logs for this are here.

Well it just keeps getting worse and worse. I had encryption turned on on both the external storage and on my Nextcloud instance. I thought I’d try to decrypt all the files to see if Nextcloud would actually decrypt the external storage as well. So I put my instance in maintenance mode, ran nextcloud.occ encryption:decrypt-all and waited for the operation to complete.
As soon as the decryption was completed I turned maintenance mode off and browsed to the webinterface. Literally all of my files are now unable to open anymore. I can’t download them, and when I rsync them to my local machine they’re still encrypted!

The log files I had shared are now gone as they were all on my nextcloud instance. However I’ll paste the raw format for what happens when I try to download a file through the webinterface:

{
   "reqId":"rDkjvNFzBj2o2FUqkzVJ",
   "level":3,
   "time":"2018-09-05T12:58:11+00:00",
   "remoteAddr":"<my ip>",
   "user":"<me>",
   "app":"no app in context",
   "method":"GET",
   "url":"\/index.php\/apps\/files\/ajax\/download.php?dir=%2FvakantieDenemarken%2FTom&files=20150721_125351.jpg&downloadStartSecret=f7ehx5ifseq",
   "message":"Exception: {\"Exception\":\"OCP\\\\Encryption\\\\Exceptions\\\\GenericEncryptionException\",\"Message\":\"Bad Signature\",\"Code\":0,\"Trace\":\"#0 \\\/snap\\\/nextcloud\\\/8267\\\/htdocs\\\/apps\\\/encryption\\\/lib\\\/Crypto\\\/Crypt.php(465): OCA\\\\Encryption\\\\Crypto\\\\Crypt->checkSignature('nNZQPvZ+8EJ0SQM...', '`\\\\xE4\\\\xF3\\\\xE3\\\\xAB\\\\x7F\\\\xB8\\\\xF5R\\\\x8B*\\\\xDD\\\\xC5gW...', '6118ddb04dfbbe2...')\\n#1 \\\/snap\\\/nextcloud\\\/8267\\\/htdocs\\\/apps\\\/encryption\\\/lib\\\/Crypto\\\/Encryption.php(380): OCA\\\\Encryption\\\\Crypto\\\\Crypt->symmetricDecryptFileContent('nNZQPvZ+8EJ0SQM...', '`\\\\xE4\\\\xF3\\\\xE3\\\\xAB\\\\x7F\\\\xB8\\\\xF5R\\\\x8B*\\\\xDD\\\\xC5gW...', 'AES-256-CTR', 0, 0)\\n#2 \\\/snap\\\/nextcloud\\\/8267\\\/htdocs\\\/lib\\\/private\\\/Files\\\/Stream\\\/Encryption.php(464): OCA\\\\Encryption\\\\Crypto\\\\Encryption->decrypt(*** sensitive parameters replaced ***)\\n#3 \\\/snap\\\/nextcloud\\\/8267\\\/htdocs\\\/lib\\\/private\\\/Files\\\/Stream\\\/Encryption.php(295): OC\\\\Files\\\\Stream\\\\Encryption->readCache()\\n#4 [internal function]: OC\\\\Files\\\\Stream\\\\Encryption->stream_read(8192)\\n#5 \\\/snap\\\/nextcloud\\\/8267\\\/htdocs\\\/apps\\\/files_external\\\/3rdparty\\\/icewind\\\/streams\\\/src\\\/Wrapper.php(83): fread(Resource id #17, 8192)\\n#6 \\\/snap\\\/nextcloud\\\/8267\\\/htdocs\\\/apps\\\/files_external\\\/3rdparty\\\/icewind\\\/streams\\\/src\\\/CallbackWrapper.php(91): Icewind\\\\Streams\\\\Wrapper->stream_read(8192)\\n#7 [internal function]: Icewind\\\\Streams\\\\CallbackWrapper->stream_read(8192)\\n#8 \\\/snap\\\/nextcloud\\\/8267\\\/htdocs\\\/lib\\\/private\\\/Files\\\/View.php(425): fread(Resource id #20, 8192)\\n#9 \\\/snap\\\/nextcloud\\\/8267\\\/htdocs\\\/lib\\\/private\\\/legacy\\\/files.php(310): OC\\\\Files\\\\View->readfile('\\\/vakantieDenema...')\\n#10 \\\/snap\\\/nextcloud\\\/8267\\\/htdocs\\\/lib\\\/private\\\/legacy\\\/files.php(122): OC_Files::getSingleFile(Object(OC\\\\Files\\\\View), '\\\/vakantieDenema...', '20150721_125351...', Array)\\n#11 \\\/snap\\\/nextcloud\\\/8267\\\/htdocs\\\/apps\\\/files\\\/ajax\\\/download.php(64): OC_Files::get('\\\/vakantieDenema...', '20150721_125351...', Array)\\n#12 \\\/snap\\\/nextcloud\\\/8267\\\/htdocs\\\/lib\\\/private\\\/Route\\\/Route.php(155): require_once('\\\/snap\\\/nextcloud...')\\n#13 [internal function]: OC\\\\Route\\\\Route->OC\\\\Route\\\\{closure}(*** sensitive parameters replaced ***)\\n#14 \\\/snap\\\/nextcloud\\\/8267\\\/htdocs\\\/lib\\\/private\\\/Route\\\/Router.php(297): call_user_func(Object(Closure), Array)\\n#15 \\\/snap\\\/nextcloud\\\/8267\\\/htdocs\\\/lib\\\/base.php(999): OC\\\\Route\\\\Router->match('\\\/apps\\\/files\\\/aja...')\\n#16 \\\/snap\\\/nextcloud\\\/8267\\\/htdocs\\\/index.php(42): OC::handleRequest()\\n#17 {main}\",\"File\":\"\\\/snap\\\/nextcloud\\\/8267\\\/htdocs\\\/apps\\\/encryption\\\/lib\\\/Crypto\\\/Crypt.php\",\"Line\":485,\"Hint\":\"Bad Signature\"}",
   "userAgent":"Mozilla\/5.0 (X11; Linux x86_64; rv:62.0) Gecko\/20100101 Firefox\/62.0",
   "version":"13.0.5.2"
}

Encryption is disabled at the moment, yet the files are still encrypted after running the ‘decrypt-all’ command.

I think I face with almost the same issue.

Do you solved this?

1 Like