Nextcloud Docker image deleted all my files during update (take care about volume mounts)

ℹ️ Support
, , , , ,
1

Just writing to warn other people about what could happen, I don’t have any faith about recovering my files (well, actually yes, but in a non-perfect-way).

First of all, the details of my Nextcloud installation:

SO: Debian 12
NC version: 28.0.5.1
Type of installation: Docker Compose
Container images:

  • mariadb:10.11
  • nextcloud:stable-fpm
  • nginx:alpine
  • redis:alpine
  • lucaslorentz/caddy-docker-proxy:ci-alpine

The situation is simple. The installation has been working fine for a year or so. Yesterday I wanted to update (as I did 5 or 6 times before) so I used watchower to check for any update for the docker images and it did the job (as always) and updated all the images. After updating I ran the scripts that installs SMB into the container so it can access one NAS (the other 2 are mounted using NFS in the host SO so they were already accessible). Then I rebooted Nextcloud container to enable the SMB acess to the third NAS and checked that the web interface was working fine but it was throwing a 502 error. Went to the container logs and it kept saying:

Initializing nextcloud 29.0.6.1 …
Upgrading nextcloud from 28.0.5.1 …
rsync: [generator] delete_file: rmdir(backupsnas12tb) failed: Device or resource busy (16)

Then I went to the NAS to check that everything was ok and then realized that this folder (backupsnas12Tb) was completely empty. At first I thought it was an error, but after some time I was sure that everything (3Tb of docs, videos, photos, etc) was gone.
In this moment I heard the disks of the other NAS with NFS access from Nextcloud spining up and after checking it, half the folders inside this folder was also gone. I inmediately shutdown Nextcloud VM to avoid losing more data and just confirmed that it also deleted half of the information in this NAS.

I’m not sure why that happened. I’m pretty sure I didn’t do anything wrong because I followed the same procedure as before. Maybe something changed in this version regarding the updates or maybe I didn’t configure the access to the folders in the right way, the only thing I’m 100% sure is that from now on, Nextcloud will be a forbiden word in my house and of course, never again will deal with it.

After that I took the disk from the NAS and used a data recovery software to recover the information, but of course, every single file now it’s called “lost name file (1)”, “lost name file (2)”, etc. What a nightmare…

If somebody experienced something similar, I would love to know the reasong why that happened and if any of you know about any data recovery software that could recover the entire folder tree with the original names.

2

I’m really sorry to hear about your data loss, that’s a nightmare for anyone relying on such systems.
But one thing I would strongly recommend: always make a full backup before every update. That should be as obvious as the “amen” in church! It’s the best (if not the only) way to protect yourself from situations like this.

What happened to you is indeed strange, and I’m sure there’s some explanation for it. However, one thing is certain: Nextcloud is not a backup solution. It’s cloud software, and the data should be backed up according to all best practices.

Much and good luck,
ernolf

2 Likes
3

Completely agree, a backup is the only way to go regarding data keeping. The point with this problem is that the folder was mounted using NFS in order to choose which folder is available in Nextcloud (one with some media that my girlfriend listens to while driving and no value in terms of data), that’s why maybe my mistake was mounting the root folder instead of the final folder with just the media, allowing Nextcloud to access this data (lazyness you can call it, but lesson learned).
Also, the NAS where half of the data was deleted, was the backup destination for some folders in the completely empty folder…
Bad luck I guess and for sure, something to learn from.

4

Without seeing your Compose file, it’s hard to say, but a wild guess: was this folder perhaps mounted within Nextcloud’s installation folder somewhere (anywhere under /var/www/html). See here: https://github.com/nextcloud/docker/#custom-volumes

1 Like
5

Sorry, but what you’re describing is not a backup, even if sharing just the media folder, preferable as read-only, would probably have limited or prevented the damage in this case.

Data that only exists in one place is by definition not backed up. Storing it on a NAS doesn’t change that, i.e, you always need at least two, better three copies of your important data stored on a different devices, or at least on different hard drives/storage pools, and ideally one of the copies is stored off-site.

Also, ideally, users and the application from which the data originates should not have access to the backup data, i.e. the backup device should pull the data from its original location, so there’s no way a user or bug in the application can mess with the backups.

1 Like
6

It seems like you are completely right.

  nextcloud:
    image: nextcloud:stable-fpm
    container_name: nextcloud-app
    networks:
      - nextcloud_network
    volumes:
      - nextcloud_data:/var/www/html:z
      - ./php-fpm-www.conf:/usr/local/etc/php-fpm.d/www.conf:ro
      - ./smbclient.sh:/var/www/html/smbclient.sh
      - /mnt/2big:/var/www/html/2big
      - /mnt/backupsnas12tb:/var/www/html/backupsnas12tb
      - ./nextcloud.ini:/usr/local/etc/php/conf.d/nextcloud.ini
    environment:
      - MYSQL_USER
      - MYSQL_PASSWORD
      - MYSQL_DATABASE
      - MYSQL_HOST
      - REDIS_HOST
      - OVERWRITEPROTOCOL
      - OVERWRITEHOST
      - TRUSTED_PROXIES
      - APACHE_DISABLE_REWRITE_IP
    restart: unless-stopped
    depends_on:
      - caddy
      - db
      - redis

Don’t know why I never saw this part in the documentation regarding the path where you shouldn’t mount the volumes (or exclude them in upgrade.exclude file), but now everything makes sense and seems like I’m the only one to blame. One more lesson to learn after i-don’t-know-how-many-years working in IT.

Thank you for your help.

1 Like
7

Apart from learning what you did wrong in this specific case, the main takeaway should be to establish a better backup concept, because mistakes happen to even the most experienced IT people, and are likely to happen again at some point in the future, and in such a case it is extremely helpful to have a recent copy of your data on a separate NAS, hard drive, cloud storage, wherever, from which you can just copy it back… :wink:

1 Like
8

This topic was automatically closed 8 days after the last reply. New replies are no longer allowed.