Increasing backup speed from data directory

ℹ️ Support
1

Sorry about this. I edited the original post by accident and lost it.

This post was about increasing rsync backup speed as it takes forever for it to check all the small files in appdata/. . . /preview/

2

2 500 000 MB/(3*3600 s) ~ 230 MB/s
that seems not so bad, not sure what connection you have between the devices.

rsync has some flavors that only sync differences (like rsnapshot), so you only sync on differences (or the update flag -u). But if you have a lot of small files, I don’t know if that brings any benefit. The same with compressing files (-z).

Other strategy, check if all files are required, check out apps data in more detail and about the apps, perhaps you can configure things better to create less files (and/or exclude certain things from the backup, e.g. thumbnails).

1 Like
3

ZFS snapshots and replication, but of course that’s not something that can be added later on if you’re not already using ZFS.

Other than that, I can only second what @tflidd said, you should do incremental backups, and only occasional full backups, maybe on weekends. In theory, you can do this with rsync, but if you want rsnapshot-like behavior, it’s probably easier to just use that rather than trying to replicate it with plain rsync: https://rsnapshot.org/.

I can’t say much about Duplicatii, as I haven’t used it in quite a while and only with relatively small amounts of data back then. But yes, based on what I hear, it’s not necessarily a performance monster :wink:

4

Thanks for the replies. rsync will only sync what has changed. In that sense it is incremental but only every has one version. It just takes forever to check the innumerable files. BUT what I can do is run the full rsync once a week and then exclude the preview files other times. Yes Duplicati is slow. But the gui is great and the development is active.

5

Not sure anything can be done about that, tbh.

I have not tested it, but it should be as simple as running a weekly cron job on e.g. Sundays with the same command you are using now, and then adding another cron job Mon-Sat with the --exclude 'preview/' option.

This will of course still only give you one version of the backup, with the difference that the previews in the backup only get “updated” on Sunday. To change that you could use separate destination folders for daily and weekly backups. This would give you two versions of the backup, but would also require more storage space. See here for examples: Use Rsync for Daily, Weekly and Full Monthly Backups

To avoid wasting space and still keep multiple versions of the backup, you could use the --link-dest option to create hard links to unchanged files from the last backup. See the example below for inspiration…

:warning: The example below was generated by ChatGPT, so please do your own research and only use it on test data until you are sure it works as intended, or use the --dry-run option!

:file_folder: Weekly Full Backup

You run:

rsync -a --delete /source/ /backup/full/

  • Copies everything, including previews.

  • --delete ensures the destination mirrors the source (optional, depending on your needs).

:file_folder: Daily Incremental Backup (Excluding Preview Files)

You run:

rsync -a --delete --exclude='preview/' /source/ /backup/incremental-$(date +%F)/

  • Excludes the previews directory or files.

  • Each daily backup can go into a dated folder.

  • For space savings, you can also use --link-dest to hard-link unchanged files from the last backup (emulating rsnapshot behavior):

rsync -a --delete --exclude='preview/' \
  --link-dest=/backup/incremental-$(date -d yesterday +%F)/ \
  /source/ /backup/incremental-$(date +%F)/

This way:

  • Changed files are copied.

  • Unchanged files are hard-linked from the previous snapshot.

  • Previews are ignored on incrementals but backed up weekly.

Have fun! :slight_smile:

…although at this point it would probably be easier to just use rsnapshot. :wink:

1 Like
6

there is definitely no need to backup appdata (frequently).. the data there belongs to the installation and fluent things like previews - you could backup them less often and focus on your user data, database and config.

in my installation #restic needs ~7-8 min to process 1-2GB of changes on total ~1100k files and upload to the cloud s3 bucket

2 Likes