Backup to a second HDD

OFFICE43 · September 7, 2020, 8:13pm

Hi fellow cloud enthusiasts,

The issue :
I use two HDDs, one for the Datadir (as you guys call your data here) and one for the backups. The first one is labelled STORAGEA and the second one (backup) is labelled STORAGEB. When I try to nc-backup, I get the following error, no matter if I try from the terminal of the Web User Interface.

In the following case I had created a directory call nc-backup inside media/STORAGEB/.
[ nc-backup ] (Mon Sep 7 03:05:09 EEST 2020)
check free space…
Maintenance mode enabled
backup database…
backup files…
tar: /media/STORAGEB/ncp-backups/nextcloud-bkp_20200907_1599437109.tar: Cannot write: Input/output error
tar: Error is not recoverable: exiting now
error generating backup
Maintenance mode disabled

In the next case I did not have the ncp-backups directory. It was just /media/STORAGEB/.
[ nc-backup ] (Mon Sep 7 22:33:47 EEST 2020)
check free space…
Maintenance mode enabled
backup database…
backup files…
tar: /media/STORAGEB//nextcloud-bkp_20200907_1599507227.tar: Cannot write: Read-only file system
tar: Error is not recoverable: exiting now
error generating backup
Maintenance mode disabled

In all cases, when I used nc-backup, either fromthe terminal or the Web User Interface, when the procedure is complete and the error is generated, the backup HDD unmounts(!).

I have to point here that automount is enabled.

I also need to point out that both HDDs are formatted as BTRFS. At least I think so because I used the format tool available by NextCloudPi.

I have made more than 20 tries with no luck. Same errors always. Mostly:
tar: /media/STORAGEB//nextcloud-bkp_20200907_1599507227.tar: Cannot write: Read-only file system

It is not hard to replicate it, just use nc-backup.

I face similar problems when I try to rsync, but let us keep it simple. Let’s see if we can solve the simple backup first.

I wonder if anyone can assist me to the right direction. I am not an IT, so I will read your responses, Google hard, try things and ask back.

ps. It crossed my mind if all else fails to install first Raspberry OS Lite, then NextcloudPI from the terminal instead of using the premade image. But I am too worn out for this, after so many tries with the image. Will probably try it at some point though.

Thanks a lot!

devnull · September 8, 2020, 6:58am

Sorry no real idea. But sometimes the error comes apparently from wrong paths.

a.) Where are you configure your backup-dir? Make a screenshot or Cut-and-Paste
b.) Is it really mounted
Post:
mount
c.) Does the correct user have write access to the backup-dir
ls -l /media/STORAGEB
d.) Compare with
ls -l /media/STORAGEA/ncdata

JimmyKater · September 8, 2020, 7:55am

I’d check the mounting-routine of the 2nd hdd. I remember there was a kind of trick doing that being on ncp. It should be documented at least here on the forum (searchers are king/queen).

If you can’t find any hint I’d mount the 2nd HDD manually (and not automatically) using fstab.

or maybe @OliverV would know more

OFFICE43 · September 8, 2020, 10:29pm

Let’s see, I will try to answer everything.

Datadir directory: /media/STORAGEA/ncdata
Destination directory for the backup: /media/STORAGEB/backup

Note: I closed the RPi, unplugged the STORAGEA HDD, then plugged the STORAGEB HDD, opened RPi, used nc-format tool, labeled the HDD as STORAGEB and completed the format process successfully. Then created a directory called backup inside /media/STORAGEB using teh mkdir command. Then plugged back the STORAGE A and moved on with nc-backup. At all times, nc-automount is active.

Now about if it is really mounted,

here is the outcome of the mount command:

/dev/mmcblk0p2 on / type ext4 (rw,noatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=823904k,nr_inodes=103729,mode=755)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=32,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mmcblk0p1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=191200k,mode=700,uid=1000,gid=1000)
/dev/sdb1 on /media/STORAGEA type btrfs (rw,nosuid,nodev,relatime,space_cache,subvolid=5,subvol=/,uhelper=udisks2)
/dev/sda1 on /media/STORAGEB type btrfs (rw,nosuid,nodev,relatime,space_cache,subvolid=5,subvol=/,uhelper=udisks2)

and here is the outcome of the mount -l command
/dev/mmcblk0p2 on / type ext4 (rw,noatime) [rootfs]
devtmpfs on /dev type devtmpfs (rw,relatime,size=823904k,nr_inodes=103729,mode=755)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=32,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mmcblk0p1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro) [boot]
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=191200k,mode=700,uid=1000,gid=1000)
/dev/sdb1 on /media/STORAGEA type btrfs (rw,nosuid,nodev,relatime,space_cache,subvolid=5,subvol=/,uhelper=udisks2) [STORAGEA]
/dev/sda1 on /media/STORAGEB type btrfs (rw,nosuid,nodev,relatime,space_cache,subvolid=5,subvol=/,uhelper=udisks2) [STORAGEB]

Now about write access to the backup dir the outcome of the commands you taught me is below:
pi@nextcloudpi:~ $ ls -l /media/STORAGEB
total 0
drwxr-xr-x 1 root root 0 Sep 9 01:08 backup
pi@nextcloudpi:~ $ ls -l /media/STORAGEB/backup
total 0
pi@nextcloudpi:~ $ ls -l /media/STORAGEA/ncdata
ls: cannot open directory ‘/media/STORAGEA/ncdata’: Permission denied
pi@nextcloudpi:~ $ sudo ls -l /media/STORAGEA/ncdata
total 156
drwxr-xr-x 1 www-data www-data 128 Sep 6 15:09 appdata_ocrnfg5r4p1o
drwxr-xr-x 1 www-data www-data 20 Sep 6 14:51 Doram
drwxr-xr-x 1 www-data www-data 26 Mar 28 22:40 files_external
drwxr-xr-x 1 www-data www-data 18 Sep 6 16:15 __groupfolders
-rw-r–r-- 1 www-data www-data 0 Mar 28 22:40 index.html
drwxr-xr-x 1 www-data www-data 62 Sep 6 11:57 ncp
drwxr-xr-x 1 www-data www-data 22 Mar 28 22:41 news
-rw-r----- 1 www-data www-data 158204 Sep 9 01:09 nextcloud.log
drwxr-xr-x 1 www-data www-data 500 Sep 9 01:00 ownbackup
drwxr-xr-x 1 www-data www-data 0 Sep 8 18:11 tmp
pi@nextcloudpi:~ $

I hope these give you some kind of clue.

Thanks a lot.

OFFICE43 · September 8, 2020, 10:33pm

Thanks! I already study the online guides for manual BTRFS format. Let’s just see if we can get some more clues from the above post first.

KarlF12 · September 9, 2020, 4:05am

To be more specific, it appears that it may have remounted as read only. Linux often does this to preserve the file system in case a read or write error is detected so that the admin can correct it before more data is corrupted.

For example, an error like that. Are you sure that the backup drive is in good health? You might want to test it.

OFFICE43 · September 9, 2020, 6:41am

That is a good idea, to test disk health.

I used the smartmontools, found directions online, it was pretty straightforward.

Unfortunately it says the disk does not support SMART capabilities. It is detected normally though and it is BTRFS. So this is out of the way.

I wonder if I can somehow fix this by messing with the permissions.

When at home I use this program for SSH access which also allows SFTP and you can change permissions easily by the right click menu. Not sure if I should go there yet though.

I think it narrows down to a permission issue and maybe mount issue as well.

If I find another HDD, different brand, etc, it would be interesting to see if it persists.

Will keep you updated, and if you think of anything else please let me know.

Have a nice day!

KarlF12 · September 9, 2020, 11:50am

I would still test it. The manufacturer should have a utility you can download to run a diagnostic test. Drives formatted as BTRFS can still have mechanical failures. I would be suspicious about any HDD that doesn’t support SMART anyway.

After you get that input/output error, run mount and see if the filesystem is now showing the “ro” option instead of “rw”, meaning it has been remounted read only due to an error. If this is indeed what’s happening, then you still have a problem, either a bad disk or corrupt filesystem.

OFFICE43 · September 11, 2020, 6:18am

I think we are getting somewhere.

I used another HDD, it is my heavily worn out, old and trusty drive I use for media exchanges with coworkers and family. Actually it is a 10 year old SAMSUNG model.

Plugged it in, formatted it with the NC format tool as BTRFS and tried to backup. Labelled it as STORAGEC. Nothing again, same error during backup.

Then, I decided to try something simple. Why not simply copy-paste the ncdata directory to STORAGEC. Again no luck. In the meantime I learned about the -raP suffix to the rsync command which gives you some feedback for every copy-paste attempted. Retried and as you can imagine a list of errors appeared some of which seemed undecipherable while others were permission related errors. Just like when using the nc-backup feature.

Time to try something even more simple: I created a directory in STORAGEC and copy-pasted it inside STORAGEA. It worked.

Then, I compared permissions:
Directories that can be copy-pasted are 775. Directories that cannot be copy-pasted are 770.

Logged in as root, I changed nc-data permission recursively. Then went inside the directory, to the subdirectory where my data are stored and tried to copy-paste those to the STORAGEB.

It worked. OK, to be honest there was a single error about a specific file, but is file related, a case study. All the other files were succesfully copied.

Somewhere at this point I refreshed the web GUI and noticed that there was some kind of error and the server was down.

Could not fix it, so I took this as an opportunity to test restore from snapshot. Worked OK but soon after this, a permission related error appeared. Tried to manually fix it, no luck. Tried to fix it with the UI tool, again no luck.

So, where are we now?

First I will recreate the entire installation.
Trying to minimize risks I will put the system in maintenance mode prior to attempting a manual backup (by saying “backup” I mean copy-paste of my datadir files).
Then, when I want to backup just the data dir I will change permissions and download everything on a second HDD.
I will make sure to reset permissions to default.
Deactivate maintenance mode.
Same procedure for the snapshot folder.

It is not the most elegant procedure, but it will probably work out just fine.

gas85 · September 11, 2020, 7:17am

First of all you can simply copy with saving file permissions with no issues.
Second of all I will use rsync or restic. In case you External drive will be broken, e.g. controller dies, then with restic you still have all data encrypted and can simply send drive to the trash without any worries about. Otherwise it is possible to replace controller and get data back.
E.g. I’m using this script

github.com

GAS85/my_backup_scipts/blob/master/systembackup-restic.sh

#!/bin/bash

# By Georgiy Sitnikov.
#
# Will do system backup and upload encrypted to mega - NEEDS megatools
#
# AS-IS without any warranty
#
# Supported key --upload to upload latest files instead of do backup again

# https://help.nextcloud.com/t/nextcloud-backup-and-restore/51589/6

nonce=$(md5sum <<< $(ip route get 8.8.8.8 | awk '{print $NF; exit}')$(hostname) | cut -c1-5 )

# Please do not use root folder
WORKINGDIR=/var/backups
toBackup="/"

configFile="/etc/restic-systembackup.conf"

This file has been truncated. show original

with following config for nextcloud and even not using maintenance mode.

# By Georgiy Sitnikov.
#
# AS-IS without any warranty

# MUST be saved under /etc with chmod 400

# Please do not use root folder
WORKINGDIR=/mnt/raid/backup/nextcloud

toBackup="/mnt/raid/nextcloud"

export RESTIC_REPOSITORY="$WORKINGDIR"
export RESTIC_PASSWORD="xxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

# Name of Remote config to use with rclone
#remote=mailru

#remote2=nextcloud

# Path to Remote folder to use with rclone
#folder=/Backups/System

#folder2=/Externals/Backup/System

#Folders to be excluded from backup
excludeFromBackup="--exclude=$WORKINGDIR"

OFFICE43 · September 11, 2020, 10:14am

Thanks, I will definitely try the megatools, didn’t know about this until now.

About copying, I tried this the other day: Logged in via SSH as root, and used the cp -rap command which gives you a nice feedback for each and every file you try to copy from the datadir. Some where copied, while some other files gave a read-only fail feedback. Which is very strange. That is why I am so nittpicky about permissions.

Will probably solve this once and for all tonight if there is enough time.

OFFICE43 · September 15, 2020, 6:00pm

Just dropped by to thank you all one more time.

It has been a couple of days that I have set the entire system from scratch and so far all good. I am backing up the Datadir using MobaXterm from a WIndows pc and I am copying the snapshots in a BTRFS external HDD.

wizlon · September 16, 2020, 12:33am

I can see your approach. May I ask: what are you trying to achieve? Even with a Pi, you can do backups multiple ways. I have never seen your particular approach. I can’t figure out what Pi version you have, but if you are trying to have a complete working backup of your instance that’s kept up to date via (say) daily backups, I would suggest a very different method.

But firstly, what are you trying to do?

V/R

Andrew