Massive automatic file deletion after Nextcloud upgrade

ℹ️ Support
1

Nextcloud version: 24.0.4.1
Operating system and version: Debian 11.4
Apache or nginx version: Apache 2.4.54
PHP version: 7.4.30

config.php: <?php$CONFIG = array ( 'passwordsalt' => 'xxxxxxxxxxxxxxxxxxxxxxxxxx', ' - Pastebin.com
Log file (isolated around the issue): {"reqId":"NtqlTVXVZAUY0SmYJJ72","level":0,"time":"August 12, 2022 23:45:01","rem - Pastebin.com

Hi,

New to this forum - hoping to find help to a somewhat serious issue I have.

Background:

Personal use (two users) Nextcloud installation on a Debian ‘stable’ machine. Using Nextcloud for quite some time now, initially started when it was still Owncloud.

Not that much data stored inside - perhaps 25GB, 5000 files or so. Using Nextcloud for file storage, calendar and contacts. Mobile clients (iOS and Android), desktop clients (windows 10) in use. And, perhaps most relevant, also some folders mounted on the same Debian box using five davfs mounted folders (via systemd mount/automount scripts).

So, I guess two months ago, I noticed a complete deletion of ~300 files & directories in Nextcloud. All in my user and all from a few of these davfs mounted folders. At the time I simply restored these from the trashcan inside Nextcloud and didn’t investigate any further. Yesterday… same thing happened, so spend time on restoring this all again.

Worth mentioning: last Friday I’ve upgraded from 24.0.3 to 24.0.4 early in the morning. From the logs, it seems like the removal of all these files started at the end of Friday, at 23:45 or so. Perhaps also worth mentioning: all the deletions happened within a davfs mounted directory - but one davfs mount was not affected. No idea why, as the mount scripts are nearly identical and there is nothing that stands out on this one not-affected directory. I’m not sure, but it is possible that also the previous time I had this issue was after a Nextcloud upgrade.

Partial theory I have at the moment:
Perhaps the davfs folders are not actively mounted at some point in time; then the mount points are empty directories and perhaps some Nextcloud cron job then decides to synchronoize (and delete from the nextcloud). If so… I don’t understand why this one davfs mount is not affected (looking at the current mount status, no davfs mount is active right now - it gets automatically mounted when accessing a directory). Also don’t understand why this massive deletion doesn’t happen more often.

So, log files. I’ve downloaded the nextcloud.log, isolated the file a bit (starting from 23:45 when the Nextcloud cron job starts and ending when the first file is deleted). Looking into this:

  • at some point, I see a message “CLI cron call has selected job with ID 963”. What does this ID 963 mean?

  • then the cron job seems to scan several of my directories within my nextcloud with message “No public access to this resource., No ‘Authorization: Basic’ header found. Either the client didn’t send one, or the server is misconfigured, No ‘Authorization: Bearer’ header found. Either the client didn’t send one, or the server is mis-configured, No ‘Authorization: Basic’ header found. Either the client didn’t send one, or the server is misconfigured”

  • Following each of above messages, it also complains “Token is not valid: Token does not exist: token does not exist”. I’ve seen complaints about this message in my Google search results, but not related to a massive deletion action.

  • After all complaints about several folders, it starts deleting files. Message with the deletion is again “Token is not valid: Token does not exist: token does not exist”

Furthermore: no errors in the log file (well, only a few during my attempt to move files from trashcan back to nextcloud, but that was half a day after the deletion).

I’m hoping someone can explain a bit what can be seen in the log and perhaps link it to a likely root cause for this issue.

2

Your description is little cumbersome to isolate or understand the issue. I hit similar problems while back

from my feeling the problem was due to “broken local client”… it look the client and especially VFS matured in the meanwhile so I never hit the issue again (but others did according to GH).

Sorry I can’t provide many useful advice - try searching the forum and Github using you log entries - hopefully you hit useful reports…

3

Are you saying you have a davfs mounted on the client, and the client is running a sync on this remote folder? If so then this is plausible.

The client really should have some kind of sanity check before wiping stuff… but I would still say running a sync on a remote mount is asking for trouble. Could you maybe set this up in Nextcloud as external storage instead?

4

The mount is using davfs2, as described here. So, that IS a supported construction I would say. Main difference is that I’m using systemd for automounting the directories (mount when needed, unmount when idle). Of course I can try to disable the unmount-when-idle, or simply mount at startup and don’t use the automount system.

But, I’m hoping to get some information on why nextcloud deletes the files/directories first, to get a better feeling for the root cause. For me, the messages I see in the log are a bit cryptic, not clear (e.g. “Token is not valid: Token does not exist: token does not exist” - is that something to be worried about? What does it mean?)

“running a sync on a remote mount” - well, not that remote. :smile: The mount is on the same machine as Nextcloud itself.

“The client really should have some kind of sanity check before wiping stuff”
I’ve given this some more thought: when not accessing the mounts for 1 minute or so, it gets automatically unmounted. So I don’t thing the deleting was done client-side: at the time I see all these ‘DELETE’ in the log file, all should be unmounted as I’m pretty sure I was sleeping at the time, not accessing data in the mounted directories. Seems to me that the deletion was done server-side for reasons I don’t understand…

5

Sorry, could you clarify this? You have a folder from your Nextcloud server mounted on your computer via davfs2, and then you’re syncing that mounted folder back to Nextcloud in the sync client…?

Remember that when not mounted, there is simply an empty folder at the mount path, or no folder at all. What should the sync client think other than everything was deleted?

1 Like
6

Ah, apologies for not being clear on this:

  • In my house I’ve got a few mobile phones, a desktop, two laptops where a Nextcloud client is running on.

  • I’ve got this one machine (a.k.a. ‘server’) with Debian Linux on it, running headless 24/7, which I’m using as webserver, mailserver, NAS, some network functions and whatever else I can think of. This is where I’ve installed Nextcloud server on.

  • On this same machine, I also regularly access it via SSH - so accessing via command line from a remote location (e.g. when I’m at work, from a work laptop, SSH into the server at home). I want to be able to easily access documents stored in my Nextcloud server, or add new documents. So within my home directory on this server, I’ve mounted a Nextcloud folder using davfs2 to a local directory

So…yes, I guess you’re right: I’m mounting a folder from my Nextcloud server to a folder on that same machine. But:

…be aware that there is NO explicit sync client. The davfs2 file system driver is the all-in-one thing here: you tell it to mount e.g. https://…/dav/files/vanaalten/MyDB to directory /home/vanaalten/MyDB and the same driver takes care of syncing between the directory and Nextcloud. So when davfs2 unmounts the directory, it will not sync anymore.

Reason for this, perhaps complicated, approach is that:
a) I cannot use a GUI based client,
b) I could use command line based nextcloud-cmd, but that only does a one-shot sync: each time you run the command it will sync, it is not continuously monitoring and syncing.

7

In the case of WebDAV, it’s not actually a sync. It’s directly accessing the remote files. This should be fine so long as you’re mounting a Nextcloud WebDAV path or share and not directly accessing the data folder.

However, that also means anything able to access this Debian mount (runaway script, fat finger, etc.) can now blow out your Nextcloud data. It’s a risk, and virtually impossible to guess what may have happened. Keep in mind it could be something that happened in Nextcloud, or any client syncing/mounting that folder.

Did you look in the Nextcloud activity app to see if it said anything? Should be something in your Nextcloud log as well.

1 Like
8

You’re right - that’s indeed a risk and I should consider a different solution. Perhaps use ‘nextcloud-cmd’ and making it a cron-job every five minutes or so.

But…

…this got interesting. I’ve never looked into the activity-app before and with this being my first time, already saw interesting results:
It seems that either the Nextcloud server software, or the davfs2 file system driver, does some cycle each night around 02:00 or so. In the activity app, I see some weird things happening:

You created 2021_12_24_Factuur.pdf
You deleted 2021_12_24_Factuur.pdf
You created 2021_11_23_Factuur.pdf and 2021_10_25_Factuur.pdf
You deleted 2021_11_23_Factuur.pdf
You created 2021_09_24_Factuur.pdf and 2021_08_25_Factuur.pdf
You deleted 2021_10_25_Factuur.pdf and 2021_09_24_Factuur.pdf
...

These are all in the davfs mounted folders. And while not giving any problems, I really dislike the idea of deleting files, then adding them again.

Then I traced this back in the nextcloud.log and also there: first two times a ‘GET’, then ‘DELETE’, then ‘PUT’. And ‘userAgent’: davfs2/1.6.0

Now looking again in the nextcloud.log at the time of the massive deletion - with a bit more knowledge on all this - I can see lots of DELETE at the issue-time, but no PUT. And, the ‘DELETE’ are from userAgent davfs2.

So this doesn’t exactly explain WHY this happens, it does suggest pretty strongly that the davfs2 driver is causing the issue.

For now I’ve enabled debug logging of the davfs2 driver. And either I will patiently wait until this issue happens again (and debug a bit more), or move to a different solution.

1 Like
9

Sounds like you have narrowed it down somewhat.

I don’t know exactly what your use case is on the davfs2 mount, but there could be an alternative there as well. Specific files can be uploaded and downloaded with curl for example, no mounting needed. Also consider external storage.

1 Like