Nextcloud client deleted local files that it was supose to ignore

Support intro

Sorry to hear you’re facing problems :slightly_frowning_face:

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can :heart:

Nextcloud version (eg, 20.0.5): 21.0.1
Operating system and version (eg, Ubuntu 20.04): Ubuntu 20.04.2
Apache or nginx version (eg, Apache 2.4.25): Apache 2.4.41
PHP version (eg, 7.4): 7.4.3

The issue you are facing:
I noticed today that Nextcloud had deleted over 300GB of video files. I have set up the desktop client to sync a specific folder and I have ignored a sub-folder in this. I have been running it this way for about a year now.

For some reason it seems that the NC client tried to sync this folder, I say this because I can see the folder in the web-interface but it is empty. Somehow all the files are deleted and I cannot find them anywhere locally or on the server.

I have tried to use the TestDisk utility to recover the files but it was not possible

Is this the first time you’ve seen this error? (Y/N):
Yes

Steps to replicate it:

  1. Do not know how to replicate this

The output of your Nextcloud log in Admin > Logging:

{"reqId":"gd3qy6QLXc3Cw7PwORX7","level":3,"time":"2021-04-19T19:13:25+00:00","remoteAddr":"185.125.169.27","user":"bigboss","app":"no app in context","method":"PUT","url":"/remote.php/dav/uploads/bigboss/3596350639/0000000000000055","message":{"Exception":"Sabre\\DAV\\Exception\\BadRequest","Message":"Expected filesize of 53643174 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 12402688 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.","Code":0,"Trace":[{"file":"/var/www/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php","line":156,"function":"put","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->"},{"file":"/var/www/nextcloud/apps/dav/lib/Upload/UploadFolder.php","line":46,"function":"createFile","class":"OCA\\DAV\\Connector\\Sabre\\Directory","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":1098,"function":"createFile","class":"OCA\\DAV\\Upload\\UploadFolder","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":504,"function":"createFile","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpPut","class":"Sabre\\DAV\\CorePlugin","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":253,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":321,"function":"start","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/apps/dav/lib/Server.php","line":332,"function":"exec","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/apps/dav/appinfo/v2/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/remote.php","line":167,"args":["/var/www/nextcloud/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/nextcloud/apps/dav/lib/Connector/Sabre/File.php","Line":246,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (Linux) mirall/3.2.0-20210410.072919.14fbf1b10-1.0~focal1 (Nextcloud, ubuntu-5.8.0-50-generic ClientArchitecture: x86_64 OsArchitecture: x86_64)","version":"21.0.1.1","id":"608843b790737"}

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => 'xxxxxxxxxx',
  'passwordsalt' => 'xxxxxxxx',
  'secret' => 'xxxxxxxxxxxx',
  'trusted_domains' => 
  array (
    0 => 'xxxxxxxxx',
    1 => 'xxxxxxxxxx',
    2 => 'xxxxxxxxx',
    3 => 'localhost',
  ),
  'datadirectory' => '/mnt/cloud/data',
  'dbtype' => 'mysql',
  'version' => '21.0.1.1',
  'overwrite.cli.url' => 'xxxxxxxx',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'xxxxxxx',
  'dbpassword' => 'xxxxxxx',
  'installed' => true,
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.local' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' => 
  array (
    'host' => 'localhost',
    'port' => 6379,
  ),
  'maintenance' => false,
  'theme' => '',
  'loglevel' => 2,
  'updater.release.channel' => 'stable',
  'default_phone_region' => 'xx',
);

The output of your Apache/nginx/system log in /var/log/____:

[Tue Apr 27 00:00:01.235935 2021] [mpm_prefork:notice] [pid 980] AH00163: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1f configured -- resuming normal operations
[Tue Apr 27 00:00:01.235947 2021] [core:notice] [pid 980] AH00094: Command line: '/usr/sbin/apache2'

Português:
Estou com um problema parecido, passei uma pasta de fotos do celular para meu desktop, ao clicar no botão para sincronizar os arquivos novos com o NC começou a sincronizar as novas fotos e também começou a aparecer mensagens dizendo que eu havia excluído algumas fotos, mas eu não exclui fotos nenhuma, apenas adicionei uma pasta nova de fotos.
O problema foi resolvido?

Inglês:
I have a similar problem, I moved a photo folder from my phone to my desktop, when I clicked on the button to synchronize the new files with the NC, I started to synchronize the new photos and also started to appear messages saying that I had deleted some photos, but I don’t delete any photos, I just added a new photo folder.
The problem was solved?

No, I have not yet found a solution. It’s very strange

It seems to be a problem with the NC client on the desktop.

Yes it seems so, I think the safest thing to do for now is not use it, though that forces us to do manual syncing which is not ideal. Do you maybe have some tips to how to recover the missing files?

Exactly, the best thing to do is not to use the NC client for the time being until the problem is solved, something else, you can make a copy of the files to a folder that is not synchronized with the NC so that if the files are deleted , you don’t miss them.
The files I lost were not very important, but in a way I didn’t want to lose them, I used the EaseUs Data Recovery Wizard program, I managed to recover a part, it took 2 hours to scan a 500 GB hard drive completely in search of the files.

Thanks for the tip, I’ll check it out and see if I can recover my files

by default deleted files remain in NC trashbin (check your sever/web interface). if you are not under resource pressure your media files should survive without issues - defaults rely on your storage quota and last for many months of regular operation.

  • check if your data exists on NC trashbin
    • if yes recover it
    • observe your local client
  • if recovery is successful
    • try to repro the initial issue and describe it with more details

In my case, the trashbin does not contain any of the deleted files

So far I have tried TestDisk which recovered 200KB of the 300GB. I’ll be testing PhotoRec next

bad situation… once you are try to recover recover with testdisk and photorec I assume no backup is available? try to recover from both hard disks - server and client side. and avoid any write to the disks until you finish the recovery or give up…

No that’s just the thing, this is a folder that was not supposed to be on the server, it was marked to be ignored and strangely got deleted locally without ever being uploaded. It seems that the NC client tried to sync the ignored folder and could not do it so it just deleted it instead. Everything on the server is backed up. But yeah I’ll continue to do some reading on recovery and if I cant find a solution I’ll just have to accept it is gone.

Hello @nimkha

Sorry to hear that you probably lost data. Nevertheless, here are a few thoughts on this, in order to hopefully avoid such a situation in the future, also for others who are reading this.

You moved a folder on your computer, that you don’t want to synchronize with your Nextcloud into a folder, that is already synchronizing it’s content with the Nextcloud, only to exclude it from the synchronization afterwards? That doesn’t make a lot of sense to me. As I understand the “ignore” function, it is primay meant to ignore folders on the server, so that the sync client does not try to download them to devices, that otherwise would run out of storage space.

What exactly happened and what can you do?

The files in this folder were probably never fully uploaded . So there is no point in trying to restore the files to the server. The sync client probably started to sync as soon as you created or moved the folder and then canceled the sync when you set it to “ignore”. Therefore only 200KB were found on the server. So if you have a chance to restore the files at all, it would be on your computer.

How to avoid this in the future:

  1. Only move files into a folder that automatically syncs with Nextcloud if you want them to be synced with your Nextcloud and keep everything else in separate folders outside of that folder.
  2. Only use the ignore function to prevent folders from beeing downloaded and not the other way arround
  3. Always backup important stuff
  4. Always backup important stuff
  5. Always backup important stuff
1 Like

Hi @bb77 thanks for the response but I probably should describe my situation better.

I am using the NC desktop client to sync my local files and folder to the server.
After installing the desktop client I choose which folders to sync with the server.
When I chose a folder I can then uncheck any sub-folder that I do not wish to upload and sync with the server.
In my case, I have one sub-folder that is unchecked and therefore is not uploaded.
This has worked fine and as intended since last summer.
Now, a couple of days ago I noticed the one sub-folder that is unchecked is empty. I then check the server and see that the sub-folder which is unchecked is uploaded but with only 3GB of the content.
I do not know how and why, all of the sudden, the NC desktop client tried to upload the sub-folder which is unchecked.

In the image, it is the last folder which is the unchecked one that has lost its content. This setup/structure has been the same since I setup the NC desktop client

I would not use it like that. But it is of course up to you.

Why not put all the folders that have to be synced in a seperate folder and leave the ones that you don’t want to get synced outside?

The problem with your setup is, that anytime you create a new folder in /media/deskto/Seagate it begins with synchronisation right away and you have to uncheck the folder afterwards.

From now on I will absolutely do it like you suggest.

I just don’t understand how this happened now and not earlier since I have been using it like this for almost a year now creating/deleting/editing files and folders, and if anybody might have known where the files might be or how I could get them back.

I can only guess…

Maybe an an incomplete upload of the folder was accidentally synchronized back somehow. What I don’t understand is why it still shows 3,2GB if the folder is empty?

Now I finally got the issue: you had a sub-folder within Nextcloud folder on the client, which you excluded from sync and this got deleted! My apologies.

My personal stand point would be not to setup it this way… sounds non-intuitive for me… maybe not even supported or at least not expected by developers. If you feel this is valid scenario maybe it’s worth opening an issue on GitHub to clarify if this scenario is supported and at least make the devs aware this case exists…

Since the the desktop client started the sync process by itself it managed to upload 3,2GB of one file (which is btw still incomplete), and also uploaded some empty folders before ignoring and deleting the rest.

The thing is these are very large files and folders, between 10-50GB, I think if they were smaller it would maybe have managed to upload them without issue. I have noticed before when trying to upload large files above 10GB that it causes problems even though I have no restriction on file size.

Just to avoid any confusion I still did not ask the desktop client to do any of this

I honestly can’t understand how that even worked, the way you did it. When I check out a folder, it disappears locally in my file system and is only visible in the Nextcloud web interface.

The explanation for the disappearance of the files could therefore be, that the check mark was set again in the meantime and an incomplete upload was then synced back from the Nextcloud to your local filessystem.

Yeah I’m starting to understand that I have misunderstood the concept of unchecking sub-folders. I will avoid any form of unchecking sub-folders in the future.

Do you have any suggestion of how I can exclude any files in the future that might be part of a folder. For instance I was working on a project regarding password analysis and wrote some scripts to perform some tests. In this folder I had a file containing passwords that was 30GB and I want to ignore just this file?