Usage of nc-snapshot vs. nc-backup

[zweiflamme]

Hey all,

I am following the steps for backing up my Cloud data provided here:

nextcloud/nextcloudpi

Still there are 2 open questions in my mind:

1.) What’s the main difference between using nc-backup including my data, and nc-snapshot? Can I use nc-backup excluding data and then doing nc-snapshot, having all my relevant data together and ready to easily restore?
2.) If so - why does nc-snapshot by default (without any UI option to change that) create the snapshot on the same drive the datadir is on? Is it safe to just move the ncp-snapshots directory to my backup drive? And can I, without interfering with anything, alter etc/nextcloudpi-config.d/nc-snapshot.sh to create the snapshot directly on my backup drive?

Probably this is also related to Auto Backup inclufing nextcloud PI image

Best regards
Dennis

[nachoparker]

1.) What’s the main difference between using nc-backup including my data, and nc-snapshot?

nc-snapshot uses no additional space, while backups with data store a whole copy of your data, which could be many gigabytes. You can have hundreds of snapshots of a 20GB datadir in a 100 GB harddrive, but only five data backups.

Can I use nc-backup excluding data and then doing nc-snapshot, having all my relevant data together and ready to easily restore?

Yes, I recommend doing that

2.) If so - why does nc-snapshot by default (without any UI option to change that) create the snapshot on the same drive the datadir is on? Is it safe to just move the ncp-snapshots directory to my backup drive? And can I, without interfering with anything, alter etc/nextcloudpi-config.d/nc-snapshot.sh to create the snapshot directly on my backup drive?

Due to the nature of BTRFS snapshots, the snapshots need to be created in the same drive. There is no option to change the path for simplicity.

If you want a second copy of your snapshots in another hard drive, or in another machine, you can use nc-snapshot-sync.

Cheers

[zweiflamme]

Thank you so much for making these points very clear! I’ll make up a backup strategy using your recommendations. Cheers
Dennis

[Agricola]

Due to the nature of BTRFS snapshots, the snapshots need to be created in the same drive. There is no option to change the path for simplicity.

If you want a second copy of your snapshots in another hard drive, or in another machine, you can use nc-snapshot-sync.

This tid bit was what I was looking for. It answered one major question for me. I have been trying to set up this backup strategy you have written about, before entering any data at all.
Now When I try to set up nc-snapshot-sync I get an error:

stat: cannot read file system information for ‘/media/hc2disk2/ncp-snapshots’: No such file or directory
/media/hc2disk2/ncp-snapshots is not in a BTRFS filesystem

Running btrfs filesystem show hc2disk shows up just like my data location. If I run ls On my data location ncp-snapshots directory shows up. What am I doing wrong?

Update: I found that leaving off the /ncp-snapshots in the destination path allows the sync to take place, and snapshots are showing up, but they are not in their own folder like in the data location. Is that the way it should be?

[crustulumtheone]

Hey, could you maybe help me stting my nc-snapshot-sync up? Im only a beginner but for me it seems as you have quite a bit of knowledge. Nobody else is answering my question so you are my last hope…
Here is my Question.
I would appreciate any help!

[michuvon]

@nachoparker

Due to the nature of BTRFS snapshots, the snapshots need to be created in the same drive. There is no option to change the path for simplicity.

@nachoparker Is this still the case and there is no workaround?

If we flash the ncp-image to a USB media the drive is formatet as ext4 and snapshots are not possible.

[nachoparker]

not possible, snapshots are a BTRFS feature

[michuvon]

So do I understand BTRFS snapshots correctly:

Snapshots can only be made if the data I want to take a snapshot from is on a BTRFS filesystem.

[mad73]

I’ve been trying to wrap my head around this whole backup strategy for days…

I have two 480GB SSDs I want to use; the first as the primary data storage, and the second as the backup. Given that snapshots appear to be the best way to go from a data efficiency standpoint, but given that

• snapshots only work on a BTRFS formatted drive, and
• you cannot install NCPi on a BTRFS formatted drive [<- this is my understanding from NP’s post above]

…my perception is I need to install NCPi on an SD card and boot from that, and then change the nc-datadir to the primary SSD. I can then enable ncsnapshots (as well as a NCPi config backup) on the primary drive, and use nc-snapshot-sync to push a backup to the secondary SSD.

My main challenge so far was trying to figure out how to format the SSDs into BTRFS using the NCPi nc-format-USB, given that I had a bunch of data on one of them I needed to transfer into NC before formatting into BTRFS. However, prior to reading this thread, I thought I could get away installing NCPi on one SSD and using the second as the back up for snapshots. My preference for booting and running NCPi from the SSD is I’ve read it’s faster. But I’d rather have a robust backup strategy.

Is my understanding of how this all works correct? Am I missing anything or is there a factor I’m not considering / aware of?

[mad73]

In your reply to @zweiflamme you encourage using nc-backup in combination with nc-snapshot. I’m trying to understand the best way to actually perform that.

I currently have NCPi booting from an SD card on an RPi 4, connected to two Kingston SSDs (primary and backup). I’ve set nc-datadir to a folder on the primary ssd. I’ve also activated nc-snapshot-auto and nc-snapshop-sync, with sync pushing to my backup SSD.

My question is what / how should I use nc-backup with this configuration? Should the destination directory be on my primary SSD, or the backup SSD? Should I run nc-backup to both SSDs? If I run nc-backup to the primary SSD, is there a way to have that automatically sync with the backup SSD?

Lastly, I still don’t quite understand how snapshots work as a back up for data? I understand conceptually that it’s an incremental save for data that has changed since the last snapshot… but what about the baseline data? If I upload 20GB of data, where / how is that 20GB of data backed up (assuming I run nc-backup without checking the include data box)? If a snapshot is only storing a data change referenced against the full 20GB of full data, how does a full data restore work if the SD or SSD that contains the data directory fails, and all you have are the snapshots that have been sync’d to the backup SSD?

[michuvon]

@mad73

Is my understanding of how this all works correct? Am I missing anything or is there a factor I’m not considering / aware of?

I think you understand correctly. If you flash the ncp img to a SSD and have no SD, there is no way to use snapshot.

To format a USB drive, the drive has to be empty. You can’t keep any data.

@mad73

My question is what / how should I use nc-backup with this configuration? Should the destination directory be on my primary SSD, or the backup SSD? Should I run nc-backup to both SSDs? If I run nc-backup to the primary SSD, is there a way to have that automatically sync with the backup SSD?

use nc-backup with a path directly to the backup SSD. No need to have it first on the datadir.

I’m working with the SSD boot setup and have set a full-auto-backup every 5 days to be started some time late at night. As I use USB 3 it only takes about 3-4h for over 400 GB of data. So I can live with that. I’ll just be frustrated If I have a hardware failure on day 5 before the backup. hehe…

@mad73

Lastly, I still don’t quite understand how snapshots work as a back up for data?

I also still don’t fully the point you describe about snapshots.
But as I have a spare Pi… Some day I will play around with it and find out. If by that time there is still no foolproof explanation, I will write one.

[mad73]

Thank you @michuvon. I’ve got my system set up now as you describe. Insofar as my question about the snapshots… my understanding is the snapshots are capturing the incremental changes to my data that occur over time. What is less clear to me is where the “foundational” data lives? For example, I’ve set up Nextcloud and uploaded 15GB of data within a 24 hours period. Does the first 24 snapshots “store” that 15GB, and then each subsequent snapshot whatever additional data is added (or changed) going forward? I guess what I’m trying to understand is it sounds like the snapshots are a relative reference to a main data source, but I don’t quite understand how that relationship works functionally.

[OliverV]

@mad73

how that relationship works functionally.

Found this to be a decent explanation Introduction to Btrfs Filesystem – Linux Hint
But there is a lot more out there

[johnnyboy]

Hi
I’m new to this Nexcloud system (and not that proficient with linux systems) and have recently built a Nexcloudpi instance running on a bootable 240gb ssd drive. Having done this, and all seeming well, i’m now trying to get my head around a backing up strategy - hence my reading this thread.

My original thought was to initially back up to a duplicate 240gb SSD to guard against disk failure and to utilise the snapshot facility to guard against idiotic/accidental deletions.

Reading around, it seems to me that in order to use the snapshot facility, the instance has to be on a btrfs formatted disk. As I understand it, my installation process has created ext4 volume for the data, from the image file I originally wrote to the SSD to begin the installation - is this correct?

So, firstly - is/was there any way to have a bootable usb drive installation and have it create a btrfs volume instead of an ext4 one?

2. With my current setup, is it possible to resize the ext4 volume and create a btrfs volume on the same physical SSD, and then migrate the required data folders over to the btrfs volume so as to enable snapshots?

3. If I were to attach a 2nd SSD drive to the raspberry pi and have it formatted as btrfs and then migrate the required data folders over to it, thus allowing snapshots to work, could I then sync/copy these back to the original ext4 drive, so as to have backup copies of the snapshots together with fuller backup files in case the 2nd SSD drive failed?

i.e does it matter if the snapshots are subsequently copied and stored on an ext4 volume?

regards

[michuvon]

@johnnyboy

As I understand it, my installation process has created ext4 volume for the data, from the image file I originally wrote to the SSD to begin the installation - is this correct?

yes

@johnnyboy

So, firstly - is/was there any way to have a bootable usb drive installation and have it create a btrfs volume instead of an ext4 one?

If you use the Raspi image proviede by @nachoparker , no, that is not possible.

@johnnyboy

With my current setup, is it possible to resize the ext4 volume and create a btrfs volume on the same physical SSD, and then migrate the required data folders over to the btrfs volume so as to enable snapshots?

This goes in to advanced areas, where I’m not feeling comfortable to give advice. So so take it with a grain of salt.
– Maybe you could try to use Gparted to do that. Using another Linux machine or the command line. After that you would have to manually set the path for your directory… i guess

@johnnyboy

If I were to attach a 2nd SSD drive to the raspberry pi and have it formatted as btrfs and then migrate the required data folders over to it, thus allowing snapshots to work, could I then sync/copy these back to the original ext4 drive, so as to have backup copies of the snapshots together with fuller backup files in case the 2nd SSD drive failed?

Sounds very complicated… and a waste of resources. If you have time try the one above (2) and please document it to post your steps here in the forum. If you get it to work we could even add it to the NCP Documentation page.

[johnnyboy]

Hi
Thanks for the reply and the information.

just thought i’d update my investigations into this in case it’s of use to anyone

I had a go, last night, experimenting with another instance, that i had originally written to a 32gb bootable usb stick, to see how i could try and achieve this.
So, using gparted, on a seperate raspbian installation, and then connecting up the usb drive, i shrank the ext4 partition to 10gb, then added btrfs functionality to gparted (sudo apt-get install btrfs-tools) and then created a btrfs partition in the space i had released (sda3 - 19gb), Labelled it ‘btrfs1’ and then booted up using the usb.

The Nextcloudpi instance seemed to still run fine with the resized ext4 partition, so i then first used the setup wizzard in the web interface to enable usb storage, but did not format when asked. I then used nc-datadir to change the data location, whereby it threw up some error messages about permissions, but showed the location of the data as now being /media/btrfs1/ncdata in the settings, and i confirmed this location actually existed via command line.

However, there still seemed to be issues of permission and mention of the location not being properly prepared (i should have cut and pasted the errors but didn’t). It still showed several of these errors in nc-datadir even after several reboots.

I then tried the wizzard again,this time selecting the format usb option which seems to have simply removed the btrfs (sda3) partition along with the other partitions on the usb.

When i have more time i may investigate further.