Best Hot-Restore Strategy

Hello everyone,

I’m using owncloud/nextcloud for a very long time and I’m very happy with it. I’m running it for such a long time, that i got multiple disc-crashes after wich I have to restore my system from my backups. The disc-crashes are not the fault of my nextcloud instance. :wink:

The restore process is annoying, takes much time and I’m always not sure what happend to my data that are processed on my clients while the restoring process took place.

But first some details about my setup.

My Setup

Typical LAMP (LinuxApacheMySqlPHP) setup with latest nextcloud version.

Linux Clients:
Ubuntu 20.04 LTS with Thunderbird + CardBook for Calendar and Addressbook; Nextcloud Desktop Client for file sync.

Android Clients:
Pixel Android with DAVX5 and Nextcloud Android Client for file (mainly pictures) sync.

Extensive usage of the files feature. Almost all my data files (pictures, documents, music, videos … > 200GB) are synced across my clients. Some examples:

  • Pictures taken on android, automaticly uploaded to the server, linux clients downloads them, then ordering the picutres in some other picture folders (directly or via share)
  • Music bought on amazon (yes, i still do this) is put in music folder, synced to the server and all linux clients pull it from there (directly or via share)
  • Documents edited on Linux client are pushed to the server and then every linux client downloads it from there (directly or via share)
  • Many calendars (work, school, kid1, kid2 …) are shared and heavily used on the Android phones and mainly only synced for viewing to the thunderbird installations
  • Many addressbooks (at least one for every family member) are shared and synced accross all clients (Android <-> Linux)

Every 3 days a full backup of my database and the server installation directory.
Every week an incremental Backup of my files on my Linux Clients.

As you can see, my family is using nextcloud a lot, even if you dont know that nextcloud has its hands in the game.

In an ideal world, I would recognize the server failure instantly. All client users stop working without any delay (not changing any data). I start my restore process, which only tooks a few seconds, and after just a few minutes downtime, everything is in exact the state everybody left it the last time and all is working on with no problems.

Now for the real world:

  • It needs some time for me to recognize, that the server is down. And even if I recognize it the time it happens, I dont have time instantly to solve id.
  • While the server is down, everybody is working with there clients: Putting more schedules in the calendar, moving given schedules, editing addressbook entries, taking pictures, moving and editing files …
  • If I finally got time to restore the server from the last backup (which may be up to a week ago by now) it took a very long time to receive all the data from the backup.

So what is this all about?

As I mentioned, it may take some time, until I can start restoring the backup. And the backup I got, may be some days in the past. Eg Backup on the 1st, editing data (calendar, files …) on the 2nd, crash on the 3th, still editing data on the clients, restoring backup from 1st on the 5th. Process done on the 6th.
So what will happen with the calendars / files i edited between the last backup and the crash (3th)? What will happen with the data I edited after the server shutdown?
And do I have to restore the files from the backup? Or would it be enough to provide the storage structure on the server (eg /var/nextcloud/user/files) and then let the client push everything to the server?

I would be very thankful, if someone could answer me, what would be the best “hot restore” strategy (hot = clients still manipulate data, while server is down) and what could go wrong.

Thanks so far!

You’ll need to setup a monitoring tool on an external server. Something like Kuma Uptime so you’ll be notified once the server goes offline.

Also, do you keep a backup server? Fwiw. We thrive on details. Please provide explicit info on software and versions as opposed to “typical lamp stack.” Thanks!

Hello “just” and thank u for your reply.

Well, my main problem is not that I will not be noticed if the server is down. As I said: even if I get informed of the server problems, the second they happened, I dont have instantly time to start the restoration process.

Do you mean a server to handle my backups or a backup nextcloud server which could come up instantly? The short answer to both is no.
My backups are created automaticly (just dumping the db, zipping the installation directory) on a regular base and distributed on several harddiscs on a regular but manual base. And there is no chance to get a second nextcloud server as a replica for my main server which could come up instantly (if that would be your proposal).

I think that is not so important for my question, but you asked for it: Ubuntu 20.04 Server, Apache 2.4.41, MySQL 8.0, php 7.4

So to come back to my main questeions:
What will happen to data (calendars, files …) that where successfully submitted to the server after the last backup and before the crash, after restoring the last backup? The data are still on the clients, so will the clients just “resend” them after the server is back on? I think that may be different for calendars / files?