two servers storing files, each on two disk mirroring each other (RAID)
Despite a lot a security deployed, two servers storing files went up in smoke after a terrible fire that entirely destroyed one of OVH data-center in France. Both servers were synchronized mirrors of each other, and located in different rooms in the data center to prevent data loose. I did not expect to have both rooms ruined at the same time.
So, the situation for now is I have a webserver and a databose still alive, but no more user files.
Question is: is there a way to force NextCloud clients to re-upload local files to the server?
That’s a very wrong statement ! For redundancy and fail-over it has to be in physical different datacenters for that exact reason !! Myself I had a server in SBG1 but backup were done on a server in Roubaix to avoid such situations !
I don’t think you have a way to force update like that from clients and don’t forget also that you might not have all files stored in NC synchronised by customers !
The thing is one server was not a backup of the other, both were live and duplicating files in real-time to have a the service up and running even in case one disk or server would stop or crash. This implied to have both servers located close to each others, ie in the same data-center. I tried to place one in Strasbourg and the other one in Roubaix but it was a disaster in terms of performances, due to latency in vRacks networks.
All the files are synchronized by users: that’s the company strategy and users are informed that only files in their NextCloud directories are protected by a near real-time backup. They may make the choice not to save their files there, but at their own risk and they know it. For now, no local file is missing, hopefully.
But my question is not about wrong statement or local backup strategy. My question is about re-synchronizing local files to remote server.
[quote=“MaximeM, post:6, topic:110671”]
The thing is one server was not a backup of the other, both were live and duplicating files in real-time to have a the service up and running even in case one disk or server would stop or crash. This implied to have both servers located close to each others, ie in the same data-center. I tried to place one in Strasbourg and the other one in Roubaix but it was a disaster in terms of performances, due to latency in vRacks networks.[quote]
So yep basically you had no backup of your datas as redundancy like that is not a real backup !
I think the best way is to get the whole archive of your nextcloud files from one of the clients (considering there is also the problem of personal NC directory of each user). Then you rebuild your NC and get all clients to synchronise on it which should go fast as they are basically going to delete all local files. Once done you put back in NC your archive and then let clients resynchronise all datas !
There were cases where the sync client deleted all files in the corresponding folders. So before doing anything or syncing, I’d ask everybody to backup their data on some offline storage before trying to sync to a new server.
isn’t it enough to move all files on the clients pc into another folder before you bring up the new nextcloud server?
the real problem would be coordinate this.
p.s.: I use S3 compatible storage at a diffferent cloud provider to backup all files (using restic). nextcloud@hetzner.nürnberg → backup@digitalocean.london if that fails, well it’s real bad karma.
I thought about this solution (renaming the local storage folder, create a new empty one with the same name, then move files from one folder to the other) but the thing is clients PC has copies of personal files (owned by user itself) but also copies of shared files (owned by other users). Restoring the server, I would like the shares not be lost, to prevent files duplicates, keep synchronization active when updates are made on shared files or folders.
My guess is that, when the client will try to synchronize the local computer with the server, the server will compare information about files from the database, not from the storage itself. Thus, unchanged files on the client will not trigger any upload or download, considering the files are the same on both sides (even if they are missing physically). I wonder if updating some information (but don’t which one) in the database would make the server consider the files it have are outdated and trigger an upload from client to server.
i always assumed that the server site is “master”. so nextcloud checks if the file “in the database” is present at the server, but isn’t, the client file will be deleted.
how many user you have and how skilled (and computer friendly) are they?
I have about 20 users. They are not much computer friendly, some are not at all. I will proceed with the maintenance on their computers.
Oh… I just though the database was used as a master and the storage only used to deliver and store user files. Indeed if missing files on storage (but still referenced in database) lead to a deletion client side, this is a real concern. As the server will not be made available until next Friday, I will install a temporary instance of NextCloud on a local computer and play with files and database entries to see how it behaves.
Yes. But if you have set up different sync points, you still might have problems and in general I think it is a good idea to have some offline backup. Especially when already something did go wrong.
Hi all,
Since OVH fire in Strasbougr datacenter, I actually experience the same situation as MaximeM.
I tell you what I did since now and where I am stucked !
First of all, all sync are off from clients
I loaded my database backup into MySQL fresh install. This is ok
Reinstalled my nextcloud folder from backup (was up-to-date) and reconfigured my Virtual Host (I use Apache). It is accessible from web browser -even if I don’t want to log-in right now to prevent any weird behaviors and irreversible actions.
I set all permissions for using nextcloud correctly
I do have, on my computer, all data I need to sync them again on cloud server. My plan was to :
do scan files into the server, to “repair” files / database associations :
sudo -u www-data php ./occ files:scan --all
and finally create a new connection on client side to sync again my files and ask again to my collaborators to do the same
But the command sudo -u www-data php ./occ files:scan --all returns the following errors for each user :
You’re right, shared files are lost, but that’s ok., we are a small team. We can handle this manually.
Done on “my way” already… Are you sure that would preserve shared files ? I have some doubt because ownership would be lost anyway !
It would be nice to have an occ command to be able to recreate files structure based on database information and a given data folder. Maybe not that easy …
@enoch85 : of course, you’re right. Actually both my VPS and my additional HDD were destroyed by fire … We are all guilty and all sad fore sure. AT least, I hope, we will all learn from this event.
So, here is a beginning of solution, from a theoretical point of view.
Context reminder: the NextCloud web portal and the database are safe. The file storage is entirely lost. Files keep existing only a client computers.
Actions are as follow:
remove trashed files and versioned files from database
rebuilt user mount points
recreate files tree
temporary allow write on all shares
copy users files, user per user, using a WebDAV client.
restore share rights, user per user, once file restoration is achieved.
In the next answer to this pose are the code fragments for steps 1 to 4 + script to restore share rights, user per user.
I successfully ran this scenario on a test platform, I should be able to try it with a larger set of data on next week, once the web portal is back on line.
if that fails, well it’s real bad karma.