Content wiped out from database after storage unmounted

🏷️ General
1

I have the following setup.

  • Nextcloud 19 (docker image) on a Raspbian Buster.
  • RAID1 via mdadm using two 4TB hard drives.
  • RAID storage added as a local external storage in Nextcloud.
  • Nextcloud desktop client which is syncing from my machine to NextCloud.

I had content synced into the external storage folder and everything was working fine. I was messing with the nextcloud server system and rebooted it. When the system was back online, the RAID disk/space was not mounted and available.

I believe the desktop client app was running during this time and deleted my local files because Nextcloud thought its copies were removed. What it thought were deletions were synced back to the desktop client.

Is it expected that Nextcloud does not understand the difference between an external storage not being mounted versus the contents contained within having been removed? Does it require deletion events or just the mere fact that the files no longer exist where it expects them on the server?

Well, the server files were never lost since the RAID device was not mounted in the first place when this happen. However, the Nextcloud database now had no evidence of those files having existed and even with the RAID mounted, the web interface showed an empty folder.

I had to run “files:scan” and then resync everything back down to the local desktop machine. Quite the scare.

How do I avoid this situation in the future? Is there some configuration available to prevent this from happening?

Thanks.

2

In a .../index.php/settings/admin/externalstorages you can set if your RAID should be rescanned by access or never. I think you can select never and simply added cron job to rescan externals only periodically. Ether by command:

sudo -u www-data occ files:scan --path user_id/files/RiadFolder

Or, if you have more external storage’s, simply use

Also you can added a line that will check before if storage is mounted.

I never had problem when external storage is unavailable - client will not wipe data out. But it is different for Local storage. If you point Nextcloud to the root mounting point, then from the Nextcloud view all data being wiped when you not mount your RAID.

E.g. your mounting point is/mnt/raid and you added /mnt/raid as external storage to the nextcloud. In this case when nothing mounted, /mnt/raid is still exist as empty folder and Nextcloud will, from my point of view, correctly think that no data exist anymore and delete it when rescanned.

If you added subfolder (e.g. /mnt/raid/share) on the raid to the nextcloud as external storage, then it should work. If RAID mount on /mnt/raid fails, then no sub-folder /mnt/raid/share exist, nextcloud will give an error and will not wipe your data out. If RAID mounted, then Sub-folder will be accessible. This is my theory :thinking:

3

gas85,

Really appreciate your response. I think your theory makes perfect sense and I think that is what happened based on my observation. My mount point is indeed /mnt/raid1 and for some reason I also have subfolders under it (e.g. /mnt/raid1/pictures, /mnt/raid1/documents) - I don’t recall creating them but maybe I did? The subfolders are exactly the same ones I have in the actual raid device. I will remove them when raid is not mounted to avoid this situation.

I had issues running your script but made an update that worked for me. I had to change the following from

if [ LogFilePath = “” ]; then

to

if [ “$LogFilePath” = “” ]; then

I’ll need to rebuild my custom docker image to get this into www-data’s cron job but that should be the easy part now with your help.

Thanks!

1 Like
4

I thinks you can easily added

docker exec --user www-data CONTAINER_ID

in front of each line starting with $PHP $COMMAND.
And delete few lines. It could be like this, but I never test it:

#!/bin/bash

# By Georgiy Sitnikov.
#
# Will do external ONLY shares rescan for nextcloud and put execution information in NC log.
# If you would like to perform WHOLE nextcloud rescan, please add --all to command, e.g.:
# ./nextcloud-file-sync.sh --all
#
# AS-IS without any warranty

# Adjust to your NC installation
	# Your NC OCC Command path
COMMAND=occ
	# Your NC log file path
ConfigDirectory=""
DataDirectory=""
LOGFILE=""

CONTAINER_ID="nextcloud"

CRONLOGFILE=/var/log/next-cron.log
	# If you want to perform cache cleanup, please change CACHE value to 1
CACHE=0
	# Your PHP location
PHP=php

. /etc/nextcloud-scripts-config.conf

# Live it like this
OPTIONS="files:scan"
LOCKFILE=/tmp/nextcloud_file_scan
KEY="$1"
SECONDS=0

if [ -f "$LOCKFILE" ]; then
	# Remove lock file if script fails last time and did not run longer than 10 days due to lock file.
	find "$LOCKFILE" -mtime +10 -type f -delete
	exit 1
fi

# Put output to Logfile and Errors to Lockfile as per https://stackoverflow.com/questions/18460186/writing-outputs-to-log-file-and-console
#exec 3>&1 1>>${CRONLOGFILE} 2>>${CRONLOGFILE}

touch $LOCKFILE

reqId=$(< /dev/urandom tr -dc A-Za-z0-9 | head -c20)

echo \{\"reqId\":\"$reqId\",\"app\":\"$COMMAND $OPTIONS\",\"message\":\""+++ Starting Cron Filescan +++"\",\"level\":1,\"time\":\"`date "+%Y-%m-%dT%H:%M:%S%:z"`\"\} >> $LOGFILE

date >> $CRONLOGFILE

# scan all files of selected users
#$PHP $COMMAND $OPTIONS [user_id] >> $CRONLOGFILE
# e.g. php $COMMAND $OPTIONS user1 >> $CRONLOGFILE


# scan all EXTERNAL files of selected users
# how to get mounted externals? --> sudo -u www-data php occ files_external:list | awk -F'|' '{print $8 $3}'
#sudo -u www-data php occ files_external:list | awk -F'|' '{print $8"/files"$3}'| tail -n +4 | head -n -1 | awk '{gsub(/ /, "", $0); print}'
# "user_id/files/path"
#   or
# "user_id/files/mount_name"
#   or
# "user_id/files/mount_name/path"

if [ "$KEY" != "--all" ]; then
	# get ALL external mounting points and users
	docker exec --user www-data $CONTAINER_ID $PHP $COMMAND files_external:list | awk -F'|' '{print $8"/files"$3}'| tail -n +4 | head -n -1 | awk '{gsub(/ /, "", $0); print}' > $LOCKFILE
		
	# rescan all shares
	cat $LOCKFILE | while read line ; do docker exec --user www-data $CONTAINER_ID $PHP $COMMAND $OPTIONS --path="$line"; done
else
	# scan all files of all users (Takes ages)
	if [ "$KEY" == "--all" ]; then
		docker exec --user www-data $CONTAINER_ID $PHP $COMMAND $OPTIONS --all
	fi
fi

duration=$SECONDS

echo \{\"reqId\":\"$reqId\",\"app\":\"$COMMAND $OPTIONS\",\"message\":\""+++ Cron Filescan Completed. Execution time: $(($duration / 60)) minutes and $(($duration % 60)) seconds +++"\",\"level\":1,\"time\":\"`date "+%Y-%m-%dT%H:%M:%S%:z"`\"\} >> $LOGFILE

# OPTIONAL
### Start Cache cleanup

if [ "$CACHE" -eq "1" ]; then
	echo \{\"reqId\":\"$reqId\",\"app\":\"$COMMAND $OPTIONS\",\"message\":\""+++ Starting Cron Files Cache cleanup +++"\",\"level\":1,\"time\":\"`date "+%Y-%m-%dT%H:%M:%S%:z"`\"\} >> $LOGFILE
	SECONDS=0
	date >> $CRONLOGFILE
	docker exec --user www-data $CONTAINER_ID $PHP $COMMAND files:cleanup >> $CRONLOGFILE
	duration=$SECONDS
	echo \{\"reqId\":\"$reqId\",\"app\":\"$COMMAND $OPTIONS\",\"message\":\""+++ Cron Files Cache cleanup Completed. Execution time: $(($duration / 60)) minutes and $(($duration % 60)) seconds +++"\",\"level\":1,\"time\":\"`date "+%Y-%m-%dT%H:%M:%S%:z"`\"\} >> $LOGFILE
fi
### FINISCH Cache cleanup

rm $LOCKFILE

exit 0
5

Thanks. I was able to add a cron task to run the script (as www-data) in the container so it seems to be working now. I was able to test it and print out the list of local external storages it detected.

1 Like
6

I have stumbled upon this problem as well.

A faulty USB cable caused my Nextcloud instance to silently remove a lot of files from the database just because of I/O errors, even thought the files remained present on the data folder.

I’ve opened an issue and described the problem here: