Now What? Long term safeguards on a fully completed setup

Ok…So now I’m the proud owner of a fully completed Nextcloud server complete with a separate turn server and a Collabora server. All checks passed. A+ on a security check. All OSS imaged…and data is on a separate drive.
But now what? Let’s suppose that the OS that Nextcloud is on craps out and I’m left with rebuilding the machine. Are their instructions on how to do that to make sure everything is aligned correctly?
Thanks in advance.

Hi.

Would say you should sacrifice a chicken to the god of neverhappenbutitcould. This might be a first good step.

The seconde good step would be:

  • doing regular database and data folder backups.
    or
  • just doing a basic data backup, and if the god of neverhappenbutitcould strike you, just rebuilding a server, moving data’s on it, and doing an occ file:scan
    or like me
  • building a second HA nextcloud server.

Or, sacrificing 2 chickens !!! All depends of the data’s value.

Data will be backed up to a remote device in my son-in-laws house once a week using rsync on a WireGuard VPN.
I’m just curious to see if there is a set of rebuild instructions somewhere that I can bookmark…just in case.
Nextcloud took me a lot of research from various sources to get just right. A LOT of research. I’d hate to lose it.
As for the data, I’ve got about 20yrs of photos, my own art for a watch making hobby I have, and all my Google stuff….and most importantly a recipe for Hess’s strawberry pie.

Setup local backups for every hour and then have that offload to the remote server once a week.

Have your remote server keep it’s own backups as well.
Ideally, keep backups in three different locations.

Fine with me, I accept the sacrifice…

Rsync backup of what exactly, only data?
Or both data and db ?

Db might be saved too if you use apps like calendar or colabora…

There are two questions for you, a backup to have the data, and the recovery of a Nextcloud server.

Like for your photos over 20 years, you probably want to keep them and multiple backups at multiple locations, will do the trick. But you don’t need the photos all the time, if you can’t access them during 1 or 2 weeks, there won’t be a problem (at least in my case).

To restore the Nextcloud server, that is something different. There is a documentation: https://docs.nextcloud.com/server/stable/admin_manual/maintenance/restore.html and if you don’t feel sure, it a very good idea to locally spin up a virtual server and try to recover the whole Nextcloud server. Then you will know if you have all required data, you can go through the process slowly and check everything out. And if the problem happens, you really know what to do.

For the speed, it can range from:

  • if the server fails, you buy replacement parts (worst case: new system) and you restore afterwards (offline time: couple of days?)
  • you already have a fully set up second server in spare (perhaps you sync the data on a regular basis) → offline time: depends how fast you can switch the systems

Raid is not really a backup or anything. But for reliability, you can cover a disk failure, since one disk fails, the server continues working and you can replace the broken disk in the background.

Some info about reliability/backup…my experience.

I’ve been using nextcloud since … 2017.

My personal server has always been based on Debian and a soft raid6 lvm.

The hardware most of the time is based on regular “gaming” hardware except Hdd who are Proversion of Seagate ironwolf for Nextcloud data and a SSD for OS.

My raid6 array grown from 6TB to 10TB (LVM)…

My HA backup is much more simpler, and have only a 12TB HDD, rsync every 6 hours, hosted by my brother ( we are happy FttH users with 10G boxes free.fr )

I did change my hardware twice since 2017.
First time 5 years ago, rebuilt next 13.

Rebuilt my OS once, rebuilding next 17.

And after 5 years, as I had to change the SSD after smart alert. Decided also to update my server totaly. Rebuilding OS and Nextcloud to 23 two weeks ago.

My raid6 had 1 disk failure in 6 years.

We are 21 users on it … Both personal and professional use…

Everytime I had to work on the server, I had the possibility to go to maintenance, activate HA manually…(luck, I guess… My 2 chickens sacrifice…)

And has i have a 3 months limite for share links, I am not very concerned by them…

My HA server is 4 years old, the HDD is new…

I could have gone for S3 external backup, but long haul pricing is bad…

I do something similar on my home fiber network. However I used rsnapshot to be able to go back in time with my backups.

Some operators share statistics like 2020 Hard Drive Reliability Report by Make and Model
I was lucky that mostly the hard drive failures were announced by degrading SMART values, so I could change them ahead of time.

Thanks for all the great replies. The initial jist was to back up my laptop to a drive which had/has all my data from a Mycloud that crapped out on me. The Mycloud lasted a long time so I can’t complain…but I didn’t want another. So I built a backup machine…and than I added a remote back up as well. Nextcloud was actually an afterthought for the first backup machine using the external storage option in Nextcloud.
After I put it up I actually have found some really good uses for it in my life other than a visual of my laptop backup data…thus the question of the best way to back it up. Once I have a back up file I can sync it to the remote backup.
I installed the “Backup” app and was wondering if that’s the best solution or if there’s a better solution?

in addition to all of the good hints - you should train the restore procedure from time to time… you should plan with the worst case - and verify if you can restore your backup on fresh installed system (e.g use a VM or another hardware box, install OS, applications from scratch and verify if your backup is sufficient to bring the system in service with all data and configs). Doing this you know how long it takes to restore and know if there are other drawbacks e.g.missing software dependencies, passwords, config files etc…

3 Likes

It was in a 5 years span, not 6.

Thank you for the backblaze stats.

When i first bought my HDD, i spent time looking at the figure before making a choose. But, it was merely impossible to find any of the “good” PIN through regular EU shops …

Had to guess which was the closest model respecting my pricing ratio. I end up choosing the Pro Ironwolf model from Seagate ( PIN ST4000NE0025 )

I bought my disks in 4 shots ( 2 disks each time ) to avoid having same batches. Still having a brand new tested HDD in spare (6 power-cycle, 4 hours spin… brand-new !!! ) .

The Ironwolf seagate were all model: ST4000NE0025 with different built date and firmware ( 5 disk had the same date/firmware, not from the same factory… 3 had same date but another firmware, also not from same factory. ). I did not tried to mess with firmware and used them “as it”…
MTBF for those drive are “1.200.000 Hours” with a Maximum rate of 300TB/year ! Witch i am absolutely below ( around 193TB ).

The failing disk had a " Power On Hours = 83593", almost 3495 spinning days.
I experienced multiples smart error:

  • Reallocated Sector Ct blown over 100 ( bad bad bad )
  • G-Sense Error Rate was reaching over 7000, the others disks were closed to 900.
  • ECC-Hardware had reach alarm value.

Hopefully, at that time, i was still eligible for the 5 years warranty and i exchanged the failing disk.

Today, i have 4 disks over 92000 power on hours, one around 7000 power on hours…
The warranty program will be reach between december 2022 and February 2023. Not hoping for any failures before the end of the program.

My next server update-rebuild will be in 2 to 4 years… At that time, i will buy a full array of new disks…

Curious why nobody mentioned Ansible. In our shop, you cannot deploy any new OS (CentoOs or Debian) or application e.g. Nextcloud without a script (playbook). We have learned the hard way that too many errors happens the manual way. Over time this has resulted in a nice library of standard setup functions to rapidly deploy a server. A nice side effect is that when you do this is scripts and look at other people’s work, you often learn a better or more complete way of doing things. There are several ready to run “playbooks” to perform a NextCloud installation if you don’t want to do your own customization.

PS Thank you for sharing your approach for backup. Very similar to ours.

Well if you have a backup to restore, you don’t need to start with a new system. Systems like ansible are great but you have to learn them as well. And for beginners, you then easily end up with a huge list of things to learn at the same time.

@jet438
Also the Backup App does not solve all your possible errors in the future.

You need knowledge. IT knowledge. If you have less knowledge you can learn only backup and restore and on error you follow your less knowledge. If you increase your IT knowledge you can perhaps solve some Nextcloud problems without backup/restore of all.
If you do not have enough knowledge or you do not want to learn you can buy a Managed Nextcloud. But you will never find a manual that solves all problems.

It’s a bit like going to the hairdresser’s. There are experts for that, too, if the hairstyle is important.

2 Likes

My son is into coding and he mentioned Ansible as well. I have a very small system. 3RPis sitting in a 3D case I designed utilizing Bitscope blades, power supply, 5 port switch, some big ass fans (Ha), and a 3T drive. Nifty little all in one. I also have the remote drive. It used to be a Rockpi device but that went South on me so I’m now using an ArgonEon Drive.
Long story short, after I set eveything up I made images of all the SD cards. Technically, sans Nextcloud and a short supply of Raspberry Pis, I can be back up and running in a matter of minutes. So while Ansible looks like a really great solution for a large-er topology it’s probably way over the top for my little pea shooter setup…or at least that was my initial take on it.
I checked out the Nextcloud Backup app. Besides it saying that it takes a lot of drive space (which I’m not worried about, it performs an encrypted backup (Good things if your storing passwords in Nextcloud)…but here’s the interesting thing, at least to me…It recommends to put the key in a safe place. Other than painstakingly writing it down I’m not sure what that means. Ha.