Backup and restore - Last backup failed!

kevinscottuk · April 16, 2024, 10:22am

I have just logged into AIO interface and can see that my last backup failed. I have tried creating a backup by clicking the Create backup button, and again it fails.

This is the output from the logs:

Performing backup...
Starting the backup...
Creating archive at "/mnt/borgbackup/borg::20240416_101100-nextcloud-aio"
/nextcloud_aio_volumes/nextcloud_aio_nextcloud_data: extended stat (xattrs): [Errno 11] Resource temporarily unavailable: '<FD 6>'
Failed to check free space before committing: [Errno 11] Resource temporarily unavailable: '/mnt/borgbackup/borg'
------------------------------------------------------------------------------
Repository: /mnt/borgbackup/borg
Archive name: 20240416_101100-nextcloud-aio
Archive fingerprint: b766b4a247ee792f7f8a1fadd94bd3d0dd7e850ab73d2bfeedf97e7f440e50bf
Time (start): Tue, 2024-04-16 10:11:01
Time (end):   Tue, 2024-04-16 10:11:18
Duration: 17.38 seconds
Number of files: 32608
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                1.58 GB            477.91 MB            103.14 MB
All archives:              653.83 GB            596.34 GB             51.89 GB

                       Unique chunks         Total chunks
Chunk index:                  104167              1028279
------------------------------------------------------------------------------
Deleting the failed backup archive...
Failed to check free space before committing: [Errno 11] Resource temporarily unavailable: '/mnt/borgbackup/borg'
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
Deleted data:               -1.58 GB           -477.98 MB           -103.16 MB
All archives:              652.25 GB            595.86 GB             51.79 GB

                       Unique chunks         Total chunks
Chunk index:                  102920               995666
------------------------------------------------------------------------------
Backup failed!
You might want to check the backup integrity via the AIO interface.

I am running Nextcloud AIO on an Ubuntu Server and the backup folder is configured as network share to my NAS mounted as /mnt/ncbackup/borg.

If I SSH into the host, I can access that share no problem:

sysadmin@nextcloud:~$ ls /mnt/ncbackup/borg
config data hints.4622 index.4622 integrity.4622 nonce README
sysadmin@nextcloud:~$ cat /mnt/ncbackup/borg/README
This is a Borg Backup repository.
See https://borgbackup.readthedocs.io/

If I check the integrity of a previous backup, it passes, so it suggests that the backup process can actually access the share.

Any help on why the backups keep failing would be much appreciated. TIA

szaimen · April 16, 2024, 11:30am

Hi, can you run sudo df -h and post the output here?

kevinscottuk · April 16, 2024, 12:46pm

@szaimen thank you for replying here is the output:

$ df -h
df: /mnt/ncdata: Resource temporarily unavailable
df: /mnt/ncbackup: Resource temporarily unavailable
Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              392M  1.9M  390M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   39G   17G   20G  47% /
tmpfs                              2.0G     0  2.0G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
/dev/sda2                          2.0G  252M  1.6G  14% /boot
tmpfs                              392M  4.0K  392M   1% /run/user/1000

Obviously it’s saying the two shared folders are temporarily unavailable. However, I can access them ok:

sudo ls /mnt/ncdata/ -l
total 0
drwxrwx--- 2 www-data root 0 Aug  2  2023 admin
drwxrwx--- 2 www-data root 0 Sep 17  2023 appdata_oche8ln7tjcz
-rwxrwx--- 1 www-data root 0 Jan  9  2023 audit.log
drwxrwx--- 2 www-data root 0 Apr  5 04:06 files_external
-rwxrwx--- 1 www-data root 0 Apr  5 04:06 index.html
drwxrwx--- 2 www-data root 0 Jan 19  2023 kscott
-rwxrwx--- 1 www-data root 0 Jan  9  2023 nextcloud.log

sudo ls /mnt/ncbackup -l
total 0
drwxr-xr-x 2 root root 0 Apr 16 11:11  borg
drwxr-xr-x 2 root root 0 Jan 20  2023 '#recycle'

szaimen · April 16, 2024, 1:01pm

This seems to be your issue. Have you tried restarting the server?

kevinscottuk · April 16, 2024, 1:07pm

Agreed. I have restarted the server and still the same. But I can still access the shares, so I am a bit confused

kevinscottuk · April 16, 2024, 2:42pm

After some digging around, I found this post here:

It appears kernel 5.15.0-102-generic breaks CIFS

I have rolled back to 5.15.0-101-generic (grub2 - How can I boot with an older kernel version? - Ask Ubuntu) and now my output of df -h no longer shows the shares as being temporaily unavailable:

$ df -h
Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              794M  1.8M  793M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   39G   17G   20G  47% /
tmpfs                              3.9G     0  3.9G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
/dev/sda2                          2.0G  252M  1.6G  14% /boot
//192.168.1.11/nextcloud            21T  2.4T   19T  12% /mnt/ncdata
//192.168.1.11/nextcloud_backups    21T  2.4T   19T  12% /mnt/ncbackup
tmpfs                              794M  4.0K  794M   1% /run/user/1000

Running manual backup from AIO interface now to see if it works ok. Will post update once completed.

Manual update completed ok

Performing backup...
Starting the backup...
Creating archive at "/mnt/borgbackup/borg::20240416_144131-nextcloud-aio"
------------------------------------------------------------------------------
Repository: /mnt/borgbackup/borg
Archive name: 20240416_144131-nextcloud-aio
Archive fingerprint: d2d131cb839f008564866a3321e01fccb2321787d4e08b0d23940b2d417a49a6
Time (start): Tue, 2024-04-16 14:41:32
Time (end):   Tue, 2024-04-16 14:43:57
Duration: 2 minutes 24.90 seconds
Number of files: 51451
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               56.14 GB             51.29 GB            286.79 MB
All archives:              708.39 GB            647.16 GB             52.08 GB

                       Unique chunks         Total chunks
Chunk index:                  104896              1065865
------------------------------------------------------------------------------
BORG_PRUNE_OPTS are --keep-within=7d --keep-weekly=4 --keep-monthly=6
Pruning the archives...
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
Deleted data:             -306.82 GB           -280.93 GB           -208.41 MB
All archives:              401.59 GB            366.23 GB             51.87 GB

                       Unique chunks         Total chunks
Chunk index:                  102302               668744
------------------------------------------------------------------------------
Compacting the archives...
Backup finished successfully on 16.04.2024 - 14:44:32 (00 hours 03 minutes 01 seconds).

system · April 24, 2024, 2:43pm

This topic was automatically closed 8 days after the last reply. New replies are no longer allowed.