NCP nc-update -nextcloud failed from 17.0.3 to 18.0.1, database error, rollback failed too, plz help

ok yesterday my usb drive was still present when i sshd into my raspberry, but when i now go into my /media/MyCloudDrive
i get the following:
root@raspberrypi:/media/myCloudDrive# ls
ls: reading directory ‘.’: Input/output error

after a reboot of my rasbperry, the usb drive is up and running again, giving my my sub pathes to ncp-backups and nextcloud path
/media/myCloudDrive/ncp-backups
and
/media/myCloudDrive/nextcloud
where
/data
and
/ncdatabase
is present

backups:
i used nc-backup for a periodic backup with data but without compression into
/media/myCloudDrive/ncp-backups
but when I check the last backup in that directory, it is by far to small with 8GB whereas my data folder in
/media/myCloudDrive/nextcloud/data
is hovering somewhere around 150GB

database is placed on my external usb drive which also hosts the data path
/media/myCloudDrive/nextcloud/data
and
/media/myCloudDrive/nextcloud/ncdatabase

after the restart and usb drive present again: systemctl status mariadb.service

Warning: The unit file, source configuration file or drop-ins of mariadb.service changed on disk. Run ‘systemctl daemon-reload’ to reload uni
● mariadb.service - MariaDB 10.3.22 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─ncp-delay-automount.conf
Active: active (running) since Wed 2020-03-18 07:44:47 GMT; 10min ago
Docs: man:mysqld(8)
systemd - MariaDB Knowledge Base
Process: 735 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Process: 760 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 769 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=/usr/bin/galera_recovery; [ $? -eq 0 ] && syste
Process: 931 ExecStartPre=/bin/sleep 20 (code=exited, status=0/SUCCESS)
Process: 1985 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 1989 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
Main PID: 1903 (mysqld)
Status: “Taking your SQL requests now…”
Tasks: 32 (limit: 4915)
Memory: 163.1M
CGroup: /system.slice/mariadb.service
└─1903 /usr/sbin/mysqld

Mar 18 07:44:24 raspberrypi systemd[1]: Starting MariaDB 10.3.22 database server…
Mar 18 07:44:45 raspberrypi mysqld[1903]: 2020-03-18 7:44:45 0 [Note] /usr/sbin/mysqld (mysqld 10.3.22-MariaDB-0+deb10u1) starting as proces
Mar 18 07:44:47 raspberrypi systemd[1]: Started MariaDB 10.3.22 database server.
Mar 18 07:44:47 raspberrypi /etc/mysql/debian-start[1996]: /usr/bin/mysql_upgrade: the ‘–basedir’ option is always ignored
Mar 18 07:44:47 raspberrypi /etc/mysql/debian-start[1996]: Looking for ‘mysql’ as: /usr/bin/mysql
Mar 18 07:44:47 raspberrypi /etc/mysql/debian-start[1996]: Looking for ‘mysqlcheck’ as: /usr/bin/mysqlcheck
Mar 18 07:44:47 raspberrypi /etc/mysql/debian-start[1996]: This installation of MySQL is already upgraded to 10.3.22-MariaDB, use --force if
Mar 18 07:44:47 raspberrypi /etc/mysql/debian-start[2074]: Checking for insecure root accounts.
Mar 18 07:44:47 raspberrypi /etc/mysql/debian-start[2086]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tabl

status does not change with systemctl restart or start
journalctl -xe holds only basic login errors from external spoofs?

journalctl -xe

Mar 18 07:51:20 raspberrypi sshd[4533]: Failed password for root from 123.200.10.42 port 48475 ssh2
Mar 18 07:51:22 raspberrypi sshd[4533]: Received disconnect from 123.200.10.42 port 48475:11: Bye Bye [preauth]
Mar 18 07:51:22 raspberrypi sshd[4533]: Disconnected from authenticating user root 123.200.10.42 port 48475 [preauth]
Mar 18 07:51:41 raspberrypi sshd[4598]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=3
Mar 18 07:51:43 raspberrypi sshd[4598]: Failed password for root from 35.245.95.132 port 45212 ssh2
Mar 18 07:51:43 raspberrypi sshd[4598]: Received disconnect from 35.245.95.132 port 45212:11: Bye Bye [preauth]
Mar 18 07:51:43 raspberrypi sshd[4598]: Disconnected from authenticating user root 35.245.95.132 port 45212 [preauth]
Mar 18 07:52:08 raspberrypi sshd[4647]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=1
Mar 18 07:52:09 raspberrypi sshd[4647]: Failed password for root from 180.167.195.218 port 60967 ssh2
Mar 18 07:52:10 raspberrypi sshd[4647]: Received disconnect from 180.167.195.218 port 60967:11: Bye Bye [preauth]
Mar 18 07:52:10 raspberrypi sshd[4647]: Disconnected from authenticating user root 180.167.195.218 port 60967 [preauth]
Mar 18 07:52:40 raspberrypi sshd[4712]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=7
Mar 18 07:52:42 raspberrypi sshd[4712]: Failed password for root from 77.159.249.91 port 49165 ssh2
Mar 18 07:52:57 raspberrypi sshd[4760]: Invalid user Ronald from 80.211.180.23 port 41272
Mar 18 07:52:57 raspberrypi sshd[4760]: pam_unix(sshd:auth): check pass; user unknown
Mar 18 07:52:57 raspberrypi sshd[4760]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=8
Mar 18 07:52:58 raspberrypi sshd[4774]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=9
Mar 18 07:52:59 raspberrypi sshd[4760]: Failed password for invalid user Ronald from 80.211.180.23 port 41272 ssh2
Mar 18 07:52:59 raspberrypi sshd[4760]: Received disconnect from 80.211.180.23 port 41272:11: Bye Bye [preauth]
Mar 18 07:52:59 raspberrypi sshd[4760]: Disconnected from invalid user Ronald 80.211.180.23 port 41272 [preauth]
Mar 18 07:53:00 raspberrypi sshd[4774]: Failed password for root from 94.191.15.73 port 58956 ssh2
Mar 18 07:53:01 raspberrypi sshd[4774]: Received disconnect from 94.191.15.73 port 58956:11: Bye Bye [preauth]
Mar 18 07:53:01 raspberrypi sshd[4774]: Disconnected from authenticating user root 94.191.15.73 port 58956 [preauth]
Mar 18 07:53:36 raspberrypi sshd[4896]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=1
Mar 18 07:53:38 raspberrypi sshd[4896]: Failed password for root from 189.8.68.56 port 37524 ssh2
Mar 18 07:53:39 raspberrypi sshd[4896]: Received disconnect from 189.8.68.56 port 37524:11: Bye Bye [preauth]
Mar 18 07:53:39 raspberrypi sshd[4896]: Disconnected from authenticating user root 189.8.68.56 port 37524 [preauth]
Mar 18 07:53:56 raspberrypi sshd[4950]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=1
Mar 18 07:53:58 raspberrypi sshd[4950]: Failed password for root from 119.28.176.26 port 38622 ssh2
Mar 18 07:54:15 raspberrypi sshd[5011]: Unable to negotiate with 185.202.1.240 port 19303: no matching cipher found. Their offer:
Mar 18 07:54:15 raspberrypi sshd[5009]: Invalid user surya from 123.200.10.42 port 39470
Mar 18 07:54:15 raspberrypi sshd[5009]: pam_unix(sshd:auth): check pass; user unknown
Mar 18 07:54:15 raspberrypi sshd[5009]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=1
Mar 18 07:54:16 raspberrypi sshd[5009]: Failed password for invalid user surya from 123.200.10.42 port 39470 ssh2
Mar 18 07:54:47 raspberrypi sshd[5076]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=2
Mar 18 07:54:49 raspberrypi sshd[5076]: Failed password for root from 212.19.134.49 port 59768 ssh2
Mar 18 07:54:50 raspberrypi sshd[5076]: Received disconnect from 212.19.134.49 port 59768:11: Bye Bye [preauth]
Mar 18 07:54:50 raspberrypi sshd[5076]: Disconnected from authenticating user root 212.19.134.49 port 59768 [preauth]
Mar 18 07:55:01 raspberrypi CRON[5121]: pam_unix(cron:session): session opened for user root by (uid=0)
Mar 18 07:55:01 raspberrypi CRON[5125]: (root) CMD (/usr/local/etc/duckdns/duck.sh >/dev/null 2>&1)
Mar 18 07:55:15 raspberrypi CRON[5121]: pam_unix(cron:session): session closed for user root
Mar 18 07:55:25 raspberrypi sshd[5173]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=2
Mar 18 07:55:27 raspberrypi sshd[5173]: Failed password for root from 200.151.208.133 port 33384 ssh2
Mar 18 07:55:28 raspberrypi sshd[5173]: Received disconnect from 200.151.208.133 port 33384:11: Bye Bye [preauth]
Mar 18 07:55:28 raspberrypi sshd[5173]: Disconnected from authenticating user root 200.151.208.133 port 33384 [preauth]
Mar 18 07:56:57 raspberrypi sshd[5368]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=1
Mar 18 07:56:59 raspberrypi sshd[5368]: Failed password for root from 189.8.68.56 port 57222 ssh2

Do you have backups?

And did you add a 2nd drive with automount enabled?

I’ve edited my above post with acutal information.

no 2nd drive, just one external usb drive mounted to /media/myCloudDrive

i dont think i have a backup at the moment to what i see now :frowning:
but i can still access the data from console. basicly i can just copy the data path to any other network storage or my workstation to backup whatever i need.

i’ve tried to create a manual backup from ncp web page with nc-backup
this is what happens:

nc-backup

[ nc-backup ]
Nextcloud is in maintenance mode - no apps have been loaded

check free space…
Maintenance mode already enabled
backup database…
backup files…
tar: /media/myCloudDrive/ncp-backups/nextcloud-bkp_20200318_1584519318.tar: Cannot write: Read-only file system
tar: Error is not recoverable: exiting now
error generating backup
{“reqId”:“cBw6hv0nStVgn9eNe6Ej”,“level”:2,“time”:“2020-03-18T08:15:43+00:00”,“remoteAddr”:"",“user”:"–",“app”:“no app in context”,“method”:"",“url”:"–",“message”:“Temporary directory /media/myCloudDrive/nextcloud/data/tmp is not present or writable”,“userAgent”:"–",“version”:“17.0.3.1”}
{“reqId”:“cBw6hv0nStVgn9eNe6Ej”,“level”:3,“time”:“2020-03-18T08:15:43+00:00”,“remoteAddr”:"",“user”:"–",“app”:“PHP”,“method”:"",“url”:"–",“message”:“fopen(/media/myCloudDrive/nextcloud/data/data_dir_writability_test_5e71d8af976be.tmp): failed to open stream: Read-only file system at /var/www/nextcloud/lib/private/legacy/util.php#796”,“userAgent”:"–",“version”:“17.0.3.1”}
Your data directory is not writable
Permissions can usually be fixed by giving the webserver write access to the root directory. See https://docs.nextcloud.com/server/17/go.php?to=admin-dir_permissions.

An unhandled exception has been thrown:
Exception: Environment not properly prepared. in /var/www/nextcloud/lib/private/Console/Application.php:166
Stack trace:
#0 /var/www/nextcloud/console.php(96): OC\Console\Application->loadCommands(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#1 /var/www/nextcloud/occ(11): require_once(’/var/www/nextcl…’)
#2 {main}{“reqId”:“cBw6hv0nStVgn9eNe6Ej”,“level”:3,“time”:“2020-03-18T08:15:43+00:00”,“remoteAddr”:"",“user”:"–",“app”:“PHP”,“method”:"",“url”:"–",“message”:“fopen(/media/myCloudDrive/nextcloud/data/nextcloud.log): failed to open stream: Read-only file system at /var/www/nextcloud/lib/private/Log/File.php#84”,“userAgent”:"–",“version”:“17.0.3.1”}
Abort…
rm: cannot remove ‘/media/myCloudDrive/ncp-backups/nextcloud-bkp_20200318_1584519318.tar’: Read-only file system

it all falls down to file system permissions?
as stated in my intial post, i had some apt-get upgrades on my pi at the beginning of february 2020 that all started the problems - i run a jdownloader docker as well that also makes problems with its path on my usb drive (same usb drive as the nextcloud installation).

how to find the problem with this corrupted filesystem?!

Suggests that the drive is failing or corrupt, I’d try fsck.ext4 or btrfs-check depending fs.

hi Oliver,

btrfs scrub start /dev/sda1

and
btrfs scrub status /dev/sda1

gives me an instant aborted srub on my external ssd
Bildschirmfoto 2020-03-18 um 11.16.44

that does not looks to good right?

however, when i start srub with -Bf

btrfs scrub start -Bf /dev/sda1

it also tells me, that the file system is read-only:
Bildschirmfoto 2020-03-18 um 11.19.42

Can it be the case, that my usb automount does not mount the sda1 device as read/write?

what i have is a backup file in
/var/www/nextcloud-bkp_20200314_1584209598-17.0.3.1.tar.gz
which was created at the beginning of the nc-update procedure 17.0.3 to 18.0.1 - the update procedure was not able to rollback to that backup!
this backup is present on my sd-card where my rasbian is installed on.
so again: rollback to the backuped database and config failed due to the read-only filesystem on the external usb drive where the database and data of nextcloud is present?

mariadb.service behavior:

after a clean restart on my raspberry and the mounted external usb present in bash
i’ve used the following commands:

systemctl status mariadb.service

systemctl status mariadb.service

root@raspberrypi:/home/pi# systemctl status mariadb.service
Warning: The unit file, source configuration file or drop-ins of mariadb.service changed on disk. Run ‘systemctl daemon-reload’ t
● mariadb.service - MariaDB 10.3.22 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─ncp-delay-automount.conf
Active: activating (start-pre) since Wed 2020-03-18 12:48:35 GMT; 3s ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 21742 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Process: 21743 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 21745 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=/usr/bin/galera_recovery; [ $? -eq
Cntrl PID: 21841 (sleep)
Tasks: 1 (limit: 4915)
Memory: 1.1M
CGroup: /system.slice/mariadb.service
└─21841 /bin/sleep 20

Mar 18 12:48:35 raspberrypi systemd[1]: Starting MariaDB 10.3.22 database server…

systemctl restart mariadb.service

systemctl restart mariadb.service

root@raspberrypi:/home/pi# systemctl restart mariadb.service
Warning: The unit file, source configuration file or drop-ins of mariadb.service changed on disk. Run ‘systemctl daemon-reload’ to reload units.
Job for mariadb.service failed because the control process exited with error code.
See “systemctl status mariadb.service” and “journalctl -xe” for details.
root@raspberrypi:/home/pi# systemctl status mariadb.service
Warning: The unit file, source configuration file or drop-ins of mariadb.service changed on disk. Run ‘systemctl daemon-reload’ t
● mariadb.service - MariaDB 10.3.22 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─ncp-delay-automount.conf
Active: activating (start-pre) since Wed 2020-03-18 12:49:15 GMT; 3s ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 22244 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Process: 22245 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 22247 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=/usr/bin/galera_recovery; [ $? -eq
Cntrl PID: 22347 (sleep)
Tasks: 1 (limit: 4915)
Memory: 1.1M
CGroup: /system.slice/mariadb.service
└─22347 /bin/sleep 20

Mar 18 12:49:15 raspberrypi systemd[1]: Starting MariaDB 10.3.22 database server…

systemctl daemon-reload
systemctl status mariadb.service

systemctl status mariadb.service

root@raspberrypi:/home/pi# systemctl status mariadb.service
● mariadb.service - MariaDB 10.3.22 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─ncp-delay-automount.conf
Active: activating (start-pre) since Wed 2020-03-18 12:49:41 GMT; 4s ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Cntrl PID: 22725 (sleep)
Tasks: 1 (limit: 4915)
Memory: 1.3M
CGroup: /system.slice/mariadb.service
└─22725 /bin/sleep 20

Mar 18 12:49:41 raspberrypi systemd[1]: Starting MariaDB 10.3.22 database server…

systemctl restart mariadb.service
systemctl status mariadb.service
systemctl status mariadb.service after restart

root@raspberrypi:/home/pi# systemctl restart mariadb.service
Job for mariadb.service failed because the control process exited with error code.
See “systemctl status mariadb.service” and “journalctl -xe” for details.
root@raspberrypi:/home/pi# systemctl status mariadb.service
● mariadb.service - MariaDB 10.3.22 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─ncp-delay-automount.conf
Active: activating (auto-restart) (Result: exit-code) since Wed 2020-03-18 12:51:05 GMT; 4s ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 24579 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Process: 24580 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 24582 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=/usr/bin/galera_recovery; [ $? -eq
Process: 24691 ExecStartPre=/bin/sleep 20 (code=exited, status=0/SUCCESS)
Process: 24828 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILUR
Main PID: 24828 (code=exited, status=1/FAILURE)
Status: “MariaDB server is down”

I’ve found out that some people have issues with smb server mounts at the same usb drive while using ncp.
that was the case. after i’ve stoped my samba server and stoped it at bootup, the usb device is mounted not read only anymore
now
btrfs scrub start -Bf /dev/sda1
gives me some rrors that were not corrected
Bildschirmfoto 2020-03-19 um 15.58.08

BEFORE i want to try to use btrfs-check to repair the filesystem I want to backup my nextcloud data inside the external usb-drive sda1.
can someone give me a hint how i can rsync the stuff to my nas (smb fileshares or rsync server) with the usage of bash commands?
or even better, can i copy the data folder to a second usb drive i can attach?

what are the steps I should take to backup and start from scratch?
if a backup is not possible i can start from scratch as most of the stuff is basicly just a folder from my workstation that is synced to my nextcould - as well as some folders from my family without any important data.

https://ownyourbits.com/2019/03/03/how-to-recover-a-btrfs-partition/