Btrfs-sync aborts with the following errors

kjathome · August 17, 2020, 4:35pm

Support intro

Sorry to hear you’re facing problems

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can

Nextcloud version:18.0.7
Operating system and version: Linux nextcloudpi 5.4.51-v7l+ #1333 SMP Mon Aug 10 16:51:40 BST 2020 armv7l GNU/Linux
Apache or nginx version: Apache/2.4.38 (Raspbian)
PHP version (eg, 7.1): PHP 7.3.19-1~deb10u1 (cli) (built: Jul 5 2020 06:46:45) ( NTS )

The issue you are facing:

Because btrfs-sync no longer works and doesn’t work again I set up a new device and ran it manually
sudo nohup /usr/local/bin/btrfs-sync -vd -z “/media/myCloudDrive1/ncp-snapshots” “/media/myCloudDrive2/ncp-snapshots”

and redirecting the output to a file with the following errors:

Check source
Check destination
Synchronizing ‘/media/myCloudDrive1/ncp-snapshots/daily_2020-08-13_211702’…
ERROR: creating subvolume daily_2020-08-13_211702 failed: File exists
Delete subvolume (no-commit): ‘/media/myCloudDrive2/ncp-snapshots/daily_2020-08-13_211702’
Retrying ‘/media/myCloudDrive1/ncp-snapshots/daily_2020-08-13_211702’…
Synchronizing ‘/media/myCloudDrive1/ncp-snapshots/daily_2020-08-13_211702’…
ERROR: chown XXXX/files_versions/pdfs/wt/ft/2010/ft_03_2010.pdf.v1446820566 failed: No such file or directory
Retrying ‘/media/myCloudDrive1/ncp-snapshots/daily_2020-08-13_211702’…
Synchronizing ‘/media/myCloudDrive1/ncp-snapshots/daily_2020-08-13_211702’…
ERROR: realpath(/media/myCloudDrive2/ncp-snapshots) failed: No such file or directory
Abort

dmesg | grep -i btrfs results to
[297826.284226] btrfs_dev_stat_print_on_error: 12 callbacks suppressed
[297826.284233] BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
[297826.284956] BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
[297826.285028] BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
[297826.285082] BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0
[297826.285328] BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
[297826.285398] BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
[297826.285508] BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
[297826.286210] BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
[297826.286484] BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0
[297826.288085] BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 10, rd 0, flush 0, corrupt 0, gen 0
[297827.375064] BTRFS: error (device sdc1) in btrfs_commit_transaction:2276: errno=-5 IO failure (Error while writing out transaction)
[297827.375091] BTRFS info (device sdc1): forced readonly
[297827.375100] BTRFS warning (device sdc1): Skipping commit of aborted transaction.
[297827.375108] BTRFS: error (device sdc1) in cleanup_transaction:1827: errno=-5 IO failure
[297827.375124] BTRFS info (device sdc1): delayed_refs has NO entry

I also recovered the Source Device from backup because there were also btrfs errors. Now after the recovery the device seem to be ok:
sudo btrfs check /dev/sda1
Opening filesystem to check…
Checking filesystem on /dev/sda1
UUID: 0b5acce4-4549-490c-8bfe-449290aa4682
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 11403001856 bytes used, no error found
total csum bytes: 10926016
total tree bytes: 211615744
total fs tree bytes: 193036288
total extent tree bytes: 6488064
btree space waste bytes: 34183721
file data blocks allocated: 36782981120
referenced 36782825472

I’ve no idea what to do more to get things working again.
Any help or ideas are appreciated.

TIA
KJ

OliverV · August 18, 2020, 8:17am

This suggests you had some prior unresolved problem?
It looks like mounting issue …
But depends history, if want further help,can you please post output of

sudo ncp-report
ls -lh /media/
df -h

via pastebin com or similar.

Btrfs sync works fine on my systems, have not seen anyone reporting it broken here or on Github.

kjathome · August 18, 2020, 3:20pm

Hi Oliver,

thanks for your reply. The output of the command is saved in corresponding *.log files, gzipped tar which can be found using this link:
https://1drv.ms/u/s!AgD2XsciRN8_20MVP2jtvKo-1AbT?e=NrUmQz
Hope that helps.
KJ

OliverV · August 18, 2020, 4:31pm

I can not open the ncp-report.log, the two others are fine. Try saving it as flat .txt file.

kjathome · August 18, 2020, 4:58pm

Here is ncp-report.txt. There is a huge Nexcloud log line which consists primarily null bytes.

https://1drv.ms/t/s!AgD2XsciRN8_20RNZtE1VXkLG29e?e=ZytKk5

OliverV · August 18, 2020, 5:40pm

Same thing, cannot open the file, sorry, but cannot assist you without.

kjathome · August 18, 2020, 9:36pm

This is really weird…

I just downloaded the file with the link provided, saved it on my PC, opened it with notepad++…
There are 166 lines, total size is 18.765.058 bytes.
It’s also possible to open the file on linux (pi) with less.

So, how do you try to open the file? Which problem you’re facing?

Another attempt: I split the file now in 2 parts, one containing all lines of ncp-report but line 146, the other one a dump (od -cx) of line 146. Both files are in ncp-report-2-parts.tar.gz.
See link https://1drv.ms/u/s!AgD2XsciRN8_20lv203SQha8inwl?e=Q0EAzn

Hope that will work… If not, I’m a little bit at a loss.

OliverV · August 19, 2020, 5:12am

I am using Linux Mint and Xed.

kjathome · August 19, 2020, 8:12am

Are you able to open the 2 files I provided yesterday? That would be great…

kjathome · August 27, 2020, 3:25pm

Just to give a short update: I moved from USB Sticks to USB HDD. The sticks used to work for about more than 10 months. So I’ll see if the new approach will work.