🔥 OVH fire disaster: file storage lost, database preserved, how to recover?

Hi,

I have a NextCloud instance running on 3 servers:

  • one server running nginx + postgres
  • two servers storing files, each on two disk mirroring each other (RAID)

Despite a lot a security deployed, two servers storing files went up in smoke after a terrible fire that entirely destroyed one of OVH data-center in France. Both servers were synchronized mirrors of each other, and located in different rooms in the data center to prevent data loose. I did not expect to have both rooms ruined at the same time.

So, the situation for now is I have a webserver and a databose still alive, but no more user files.

Question is: is there a way to force NextCloud clients to re-upload local files to the server?

5 posts were split to a new topic: Crossposting in two languages

That’s a very wrong statement ! For redundancy and fail-over it has to be in physical different datacenters for that exact reason !! Myself I had a server in SBG1 but backup were done on a server in Roubaix to avoid such situations !
I don’t think you have a way to force update like that from clients and don’t forget also that you might not have all files stored in NC synchronised by customers !

Hi Vincent,

The thing is one server was not a backup of the other, both were live and duplicating files in real-time to have a the service up and running even in case one disk or server would stop or crash. This implied to have both servers located close to each others, ie in the same data-center. I tried to place one in Strasbourg and the other one in Roubaix but it was a disaster in terms of performances, due to latency in vRacks networks.

All the files are synchronized by users: that’s the company strategy and users are informed that only files in their NextCloud directories are protected by a near real-time backup. They may make the choice not to save their files there, but at their own risk and they know it. For now, no local file is missing, hopefully.

But my question is not about wrong statement or local backup strategy. My question is about re-synchronizing local files to remote server.

[quote=“MaximeM, post:6, topic:110671”]
The thing is one server was not a backup of the other, both were live and duplicating files in real-time to have a the service up and running even in case one disk or server would stop or crash. This implied to have both servers located close to each others, ie in the same data-center. I tried to place one in Strasbourg and the other one in Roubaix but it was a disaster in terms of performances, due to latency in vRacks networks.[quote]
So yep basically you had no backup of your datas as redundancy like that is not a real backup !

I think the best way is to get the whole archive of your nextcloud files from one of the clients (considering there is also the problem of personal NC directory of each user). Then you rebuild your NC and get all clients to synchronise on it which should go fast as they are basically going to delete all local files. Once done you put back in NC your archive and then let clients resynchronise all datas !

There were cases where the sync client deleted all files in the corresponding folders. So before doing anything or syncing, I’d ask everybody to backup their data on some offline storage before trying to sync to a new server.

1 Like

Thanks for the advice, I will. :+1:

1 Like

isn’t it enough to move all files on the clients pc into another folder before you bring up the new nextcloud server?

the real problem would be coordinate this.

p.s.: I use S3 compatible storage at a diffferent cloud provider to backup all files (using restic). nextcloud@hetzner.nürnberg → backup@digitalocean.london :wink: if that fails, well it’s real bad karma.

I thought about this solution (renaming the local storage folder, create a new empty one with the same name, then move files from one folder to the other) but the thing is clients PC has copies of personal files (owned by user itself) but also copies of shared files (owned by other users). Restoring the server, I would like the shares not be lost, to prevent files duplicates, keep synchronization active when updates are made on shared files or folders.

My guess is that, when the client will try to synchronize the local computer with the server, the server will compare information about files from the database, not from the storage itself. Thus, unchanged files on the client will not trigger any upload or download, considering the files are the same on both sides (even if they are missing physically). I wonder if updating some information (but don’t which one) in the database would make the server consider the files it have are outdated and trigger an upload from client to server.

i always assumed that the server site is “master”. so nextcloud checks if the file “in the database” is present at the server, but isn’t, the client file will be deleted.

how many user you have and how skilled (and computer friendly) are they?

I have about 20 users. They are not much computer friendly, some are not at all. I will proceed with the maintenance on their computers.

Oh… I just though the database was used as a master and the storage only used to deliver and store user files. Indeed if missing files on storage (but still referenced in database) lead to a deletion client side, this is a real concern. As the server will not be made available until next Friday, I will install a temporary instance of NextCloud on a local computer and play with files and database entries to see how it behaves.

Yes. Best is to try.

can you collect their local files with an external usb drive?

Yes. But if you have set up different sync points, you still might have problems and in general I think it is a good idea to have some offline backup. Especially when already something did go wrong.

1 Like

I could collect them but I cannot access physically the server. Instead, I’m able to transfer their files through Internet.

Hi all,
Since OVH fire in Strasbougr datacenter, I actually experience the same situation as MaximeM.

I tell you what I did since now and where I am stucked !

  • First of all, all sync are off from clients
  • I loaded my database backup into MySQL fresh install. This is ok
  • Reinstalled my nextcloud folder from backup (was up-to-date) and reconfigured my Virtual Host (I use Apache). It is accessible from web browser -even if I don’t want to log-in right now to prevent any weird behaviors and irreversible actions.
  • I set all permissions for using nextcloud correctly

I do have, on my computer, all data I need to sync them again on cloud server. My plan was to :

  • do scan files into the server, to “repair” files / database associations :

sudo -u www-data php ./occ files:scan --all

  • and finally create a new connection on client side to sync again my files and ask again to my collaborators to do the same

But the command sudo -u www-data php ./occ files:scan --all returns the following errors for each user :

I’m looking for a solution, maybe an alternative !
OK : just read this - which gives a solution : Home storage for user not writable - #6 by steinbach

if you sync them from your client via nextcloud client all shares would be lost. or? if you never shared any file with anyone that’s ok.

i would use the client files as a kind of backup of the nextcloud data files/folders and copy them back to the server via scp/rsync or similar.

so: install nextcloud. restore the user files collected from the clients. restore the database and run occ maintenance:data-fingerprint

after that run occ files:scan --all

if the users shared files to each other or use group folders. that might be of course a bit tricky.

and i’m pretty sure users keep working with there files on their clients. so some shared files might be inconsistent between users.

You’re right, shared files are lost, but that’s ok., we are a small team. We can handle this manually.

Done on “my way” already… Are you sure that would preserve shared files ? I have some doubt because ownership would be lost anyway !

It would be nice to have an occ command to be able to recreate files structure based on database information and a given data folder. Maybe not that easy …

Many thanks for your reply @Reiner_Nippes !

Maybe of topic here, but I’m surprised that one doesn’t keep a backup on a completely different location - especially for such important servers.

I also learnt it the hard way several years ago, but for anyone else seeing this: Keep backups on the backups, you never know.

It’s sad to hear the story about OVH, I feel for those involved. :`(

1 Like

@enoch85 : of course, you’re right. Actually both my VPS and my additional HDD were destroyed by fire … We are all guilty and all sad fore sure. AT least, I hope, we will all learn from this event.

3 Likes

Hello,

So, here is a beginning of solution, from a theoretical point of view.

Context reminder: the NextCloud web portal and the database are safe. The file storage is entirely lost. Files keep existing only a client computers.

Actions are as follow:

  1. remove trashed files and versioned files from database
  2. rebuilt user mount points
  3. recreate files tree
  4. temporary allow write on all shares
  5. copy users files, user per user, using a WebDAV client.
  6. restore share rights, user per user, once file restoration is achieved.

In the next answer to this pose are the code fragments for steps 1 to 4 + script to restore share rights, user per user.

I successfully ran this scenario on a test platform, I should be able to try it with a larger set of data on next week, once the web portal is back on line.

1 Like