Beta of peer-to-peer Nextcloud Backup app now available for testing!

📰 News
1

Originally published at: Beta of peer-to-peer Nextcloud Backup app now available for testing! - Nextcloud

Nextcloud Backup logo - dot with open circular arrow around it

It is our mission to put you back in control over your data. While not everybody can run their own Nextcloud server, hundreds of thousands of people do. Making it easier for them to keep their data safe is important. If you are one of them, Nextcloud Backup is for you!

With Nextcloud Backup, you can find a friend or family member who has their own Nextcloud server and ask them to create a user account for you. Then, you can setup Backup to regularly store compressed, encrypted backups of your data on their Nextcloud server! If something goes wrong, you can restore your Nextcloud, or even bring back individual files and folders. Of course, it might make sense to, in exchange, create an account for them on your server to store their backups!

What makes Nextcloud Backup different from other backup solutions? Its ease of use! If you are using a backup technology that you are comfortable with, keep using it. Nextcloud Backup is for those users who have found it difficult and time consuming to setup another backup solution.

What it can do

Nextcloud Backup is in beta and designed for Nextcloud 23, so it isn’t fully baked yet. But it already does a lot of what you might need it to do – and quite some things more! Let’s walk through it.


App settings

App settings (continued)

Getting recovery key from app settings

The app can:

  • Store backups on another Nextcloud server. The other server can but doesn’t have to have the Backup app installed
  • Store backups on external storage, like FTP, SMB, WebDAV or any other Nextcloud-supported protocol. You can also store them on a local drive, like an USB drive attached to your server!
  • Do manual and/or automatic, scheduled backups in time slots you can configure
  • Store the entire Nextcloud installation, including configuration, apps and their data. Note: non-local data, including external storage, is NOT backed-up. Maybe a feature for the future!
  • Do differential or full backups in the background (short maintenance mode needed)
  • Do optional (but recommended and enabled by default) compression and encryption
  • Export encryption key and other configuration information to a file or clipboard for safe-keeping
  • Give notifications in the Activities app about backup activity
  • Exclude folders from backup if you add a .nobackup file in the folder.
  • Be controlled by the command line to search for and restore single files, initiate full restore, or list and create backups

Backups happen in the background – but a short downtime is needed when a snapshot is made. Nextcloud will be in maintenance mode during this phase. Once the snapshot is made, it is compressed, split in 100MB parts and encrypted, a process which happens in the background and does not require maintenance mode. These are then stored on the designated backup system, be it local or remote.

Note: the Backup app requires quite some free storage. When a full backup is made, a complete copy of Nextcloud and its data is made, after which this gets compressed and encrypted and stored on the designated backup system.

Get started

Right now, the backup app is in beta and only available for Nextcloud 23. A final release is planned to coincide with the Nextcloud release later this year. You can get involved with testing! Find the app in github. A direct download of the beta tarball is here. You can extract that in your apps folder in Nextcloud 23 beta 2 install and (after setting the correct user ownership!) the app will show up in the list of not enabled apps in your instance. Turn it on and try it out! You can leave feedback in the forums below or in the Github repo.. The app should also appear in the app store for Nextcloud 23 in the next few hours.

You can also read our full press release for some more background information.

11 Likes
2

Is the app using rsync + restic?

3

Sounds like an awesome feature.

Some technical background information would be helpful, like:

  • which mechanism is used in the background (like @just asked)
  • what are the hardware requirements
  • estimated time needed to encrypt and backup a specific amount of data on an example device
    (for example: initial backup of 100 GB, quad core CPU 2GHz base clock with encryption set to XY took 3 hours …).

This would help to understand if this feature could work on our hardware at home.

1 Like
4

@just @Schmu documentation is still in writing, but you might already find the information you are looking for there

2 Likes
6

I really appreciate this this app - good idea addressing a major pain point of missing easy offsite backup solution.

for me the docs look not bad already…

but from what I understand it will perform full/incremental backup every X intervals. Implementing such backup with reasonable amount of data - my instance has about 1TB of data (majority is static - my family photo collection) - sounds problematic for me in terms of used storage and transfer times (over private internet connection)… especially with the idea of other Nextcloud instance as designated target I would really appreciate this app would act more like a sync solution: perform initial sync and just send periodic updates of the files and full DB backup to the destination (file versions of the other instance would keep change history)

7

You can do the updates to an external disk or similar on the local machine, including locations without Nextcloud. Lots of good options for proper backups and supports incremental, so should fit those needs via cron.

8

@wwe From the settings page, you can set the frequency of ‘Full’ backup to 90 days (or above, there is no limit) and keep a small period of few days (or 1 day for daily backup) for ‘incremental’ backup.

9

today I read the docs of this really interesting app and have to admit this doesn’t use restic. From the description of the 1st pass it’s clear the app uses full copy of the files:

Because there is no compression during the first pass, the appdata folder of the Backup App will require at least the same size of your current setup of Nextcloud: the content of the core, its apps and local data.

yes this is true - but this doesn’t change the fact it consumes a lot of space (each time the full size of Data+Apps+DB, little compressed).

Side effect - if you loose your instance you will need to go through restore of the full backup and every single incremental one - so I would keep the number of incremental steps low…

If we talk about 100GB of data and you want to keep 4 weekly full backups you need ~400GB storage at your backup destination (+changes in incremental) and transfer this data to you final backup destination which becomes challenge until you have fiber at home. Uploading 100GB at 10MBit/s takes about 22 hours! with 50MBit/s it still takes 4-5 hours… and this happens for each full backup… Assuming this is automatic and robust process it’s only annoying pain point and not a blocker… but as @just stated modern approach like restic or BorgBackup would improve the process.

For me this sounds like a good and easy solution to allow less trained users to start with backups (maybe first in their life) - but because of the size and transfer times this remains mostly limited to local networks for majority of personal users…

Despite the drawbacks I’m still excited somebody made such important and un-thankful task easier to accomplish!

1 Like
10

This is so wrong, I don’t know who wrote this! :wink:
I need to correct this, you only need to apply the last incremental backup that contains all modification since last full backup

You can already use an external storage for that, but yes. It is noted somewhere in my head to have a look into this and comes with a solution using zlib during the 1st pass. level of compression will be set by an admin to choose between speed and diskspace.

1 Like
11

maybe my understanding is wrong. In my eyes incremental backup includes changes since last backup - a backup holding all changes since last full backup called differential (see this article for details). Restoring differential backup involves 2 steps: restore full backup + latest differential version you want, restoring incremental backup involves multiple steps: restore full backup and every subsequent backup until the point you want to restore (one could automate it but technically there is no other way).

Example:

  • you decide to have one full setup of every 90
  • and going restore the 87th incremental set
  • you have to restore the full backup
  • and 87 incremental backups

(88 steps in total). Depending on the number of changes all the incremental changes might be fast, butt all of them are required to recover the system…

in my eyes the limiting factor is not the size of temporary storage used for 1st pass - the problem is bandwidth and transfer times when the destination is remote (using remote destination for temporary storage aka 1st pass doubles or triples this problem depending on the bandwidth).

@Cult don’t get me wrong, I never mind to blame your app - in opposite I really appreciate it! I believe my long time experience as system engineer allows me to predict some drawbacks, hobby admins might not be aware of - and help users and devs to prevent stress, useless troubleshooting and bug reports caused by wrong expectations.
If you made real-world testing with NC with dozens or hundreds GB of data over personal-grade DSL line and feel the performance good - just ignore this comment or even better post the test results so people know what to expect…

3 Likes
12

Yep. We will be flodded with posts here in the forums, from people who try to backup their movie collections from their Raspis with 18TB disks atached to it. :wink: In addition to that, another Nextcloud is probably the worst backup destination that you can imagine for home users. Not only because of bandwidth issues, but rather because on the other side there is probably also just a Raspi with a USB disk attached to it, which mom then unplugs by mistake while she is vacuuming, just at the time when the backup reached 99% after it was running for a week :wink: But exactly this scenario is advertised in the article, while the useful backup targets are only mentioned as a side note.

Anyway… Take my statetement with a grain of salt :wink: I also appreciate that such an app is coming and it is definitely a useful thing. But maybe Nextcloud should do a little less marketing and a little more technical facts in their announcements… I really hope that S3 compatible storage providers like Backblaze B2 will be supported.

Cheers! :slight_smile:

2 Likes
13

S3, and any other external storage app service, are supported.

2 Likes
14

Ah cool. So then it uses the External Storage Support app to connect to the backup destination. Thanks for the info :slight_smile:

15

The uploading process have a retry-on-fail, meaning that even if the remote instance goes not available during the upload, the uploading process will just try again later and finish its work.

But I understand your concern, smart people will just add a .nobackup files at the root of their folder that contains useless data like movies :slight_smile:

2 Likes
16

Yeah, I did read it. Assuming it must be rsync and would appreciate clarification. Cheers!

17

@just the data transfer is using Nextcloud’s API and require the External Storage App

18

If I not mistaken it could support the two scenarios discussed in Clone Nextcloud

  • cloning
  • synced instances (privat one <–> public / DMZ one) but only manage the private one
19

@wwe sorry, I have not seen your post earlier!

We are on the same track with the same objective and we both know that :slight_smile:
I never took any of your advice seriously as a negative or agressive comment

I will have a look to add some example use case in the documentation.
As an alternate solution, when facing huge data collection and small internet connection, it might be doable, and simple enough, to use your workstation as an external storage.

If you configure your workstation so that your Nextcloud can access one of the folder/harddrive; even on Windows you can find a simple way to run an ftpd.
The result would be that if your workstation is running, the Backup App will upload your backups on it. If your workstation is not running, the Backup App will try again later.

Note: fun thing, few months ago, before working on this app, I was looking for a way to detect/list USB device when plugged on the server. Could be fun to automatically store your backup on an external drive that you plug from time to time…

No, I did not have the right wording, I will fix this. Thanks!

2 Likes
20

@packElend Yes, when restoring a backup, you can rearrange the path for the extraction of each data pack (nextcloud, apps, datadirectory, …) and the localisation of the database. The backup process will also update the config/config.php to fit the new location.
Meaning that you do not even need to move the backups files and using the ./occ command from one instance, you can easily create one or multiple clones.

You can use the app to migrate your nextcloud from a container/vm to an other setup.

From the documentation:

However, from what I understood from the request from the discussion your linked, Just doing a backup before upgrading seems enough

2 Likes
21

my idea is that it would even allow a life-sync, which would be an extension of the peer-to-peer Nextcloud Backup app.

As most of the code is already in this app, it would be great if that app could be extended or another app build upon it that helps in the following scenario:

  1. primary NC instance in your private LAN, you have all your files there and a bunch of apps running. There are files you don’t want to see be available from the www in case the NC instance get compromised.
  2. secondary, public, NC instance, which is located in your DMZ or hosted by NC itself or whoever. This contains files and apps you want to access when you are away or share with others. It could be a read-only clone of your primary NC but with reduced content. You may want friends to upload things as well or change e.g. tasks in the task app using the secondary NC, so it would be welcome if sync would two way.

Based on the description of nextcloud/backup: Backup now. Restore later. (github.com) I dare to say that would be feasible reusing most of the code of this app.