[NCP] NextCloud and MariaDB down

ncp
nc14

#1

Hello,
I have nextcloutpi running on a raspberry pi for a longer time already.
A few days ago nextcloud and the database were suddenly down, I did not change anything.

NextCloudPi version v0.67.5
NextCloudPi image NextCloudPi_03-18-18
distribution Raspbian GNU/Linux 9 \n \l
automount yes
USB devices sda
datadir /media/USBdrive/ncdata
data in SD no
data filesystem btrfs
data disk usage 463G/1.9T
rootfs usage 1.9G/15G
swapfile /var/swap
dbdir /media/USBdrive/ncdatabase
Nextcloud check error
HTTPD service up
PHP service up
MariaDB service down
Redis service up
Postfix service up
internet check ok
port check 80 open
port check 443 open
IP 192.168.178.15
gateway 192.168.178.1
interface eth0
certificates cloud.mydomain.de
NAT loopback no
uptime 22:53

Also a restart does not change anything.
I have an older backup of the SD card from the time I was using nextcloud 13. When I loaded that on a new SD card, Nextcloud started and MariaDB were up, but the database was not used (“dbdir does not exist”), so nextcloud did not knew any users and I could not log on. Also the lets encrypt script in ncp-web threw an error:
./letsencrypt.sh: line 63: letsencrypt: command not found.

Long story short, any way to get things up again? I am even struggling with finding out why database and nextcloud do not start. Luckily I can still access my files via samba and nfs.

systemctl status mariadb.service shows

mariadb.service - MariaDB 10.1.37 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2018-12-01 08:15:05 UTC; 1 day 6h ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 825 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 574 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=/usr/bin/galera_recovery; [ $? -eq 0 ] && syste
Process: 535 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 524 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 825 (code=exited, status=1/FAILURE)
Status: “MariaDB server is down”

Dec 01 08:15:02 nextcloudpi systemd[1]: Starting MariaDB 10.1.37 database server…
Dec 01 08:15:04 nextcloudpi mysqld[825]: 2018-12-01 8:15:04 1996299680 [Note] /usr/sbin/mysqld (mysqld 10.1.37-MariaDB-0+deb9u1) starting as
Dec 01 08:15:04 nextcloudpi mysqld[825]: 2018-12-01 8:15:04 1996299680 [Warning] Can’t create test file /media/USBdrive/ncdatabase/nextcloud
Dec 01 08:15:04 nextcloudpi mysqld[825]: [102B blob data]
Dec 01 08:15:04 nextcloudpi mysqld[825]: 2018-12-01 8:15:04 1996299680 [ERROR] Aborting
Dec 01 08:15:05 nextcloudpi systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Dec 01 08:15:05 nextcloudpi systemd[1]: Failed to start MariaDB 10.1.37 database server.
Dec 01 08:15:05 nextcloudpi systemd[1]: mariadb.service: Unit entered failed state.
Dec 01 08:15:05 nextcloudpi systemd[1]: mariadb.service: Failed with result ‘exit-code’.

Thanks for any help! :slight_smile:


#2

This looks to me like mariadb expects an external USB disk drive, and can’t find it.


#3

That makes no sense to me, but now I see that also nc-fix-permissions says „data dir not found„.
How can this be? The drive is mounted, I can access it via samba, nfs and ftp, why should it suddenly not be accessible to ncp anymore?


#4

Are you using more than one external drive? If so check out this wiki page.
What is output of:

ls -lh /media/

and

df -h


#5

No, there is just one single usb drive attached and that never changed. I could provide the output of the commands this evening when I am at home again.


#6

Maybe same issue i had:
https://help.nextcloud.com/t/after-last-update-mariadb-doesnt-start-after-reboot/42092/2


#7

Thanks, that sounds like a possible candidate, so it would have tried to access the mounted drive before it was actually mounted?! (although I would then expect that many more people have the issue, unless they did not reboot for a long time).
Can you help me out where the file is that I need to maintain?


#8

When i am home again i will do.


#9

ok, back home now…

try following:

ssh into your pi
following commands with sudo or go root:

systemctl disable mariadb
cp /lib/systemd/system/mariadb.service /etc/systemd/system/mariadb.service 
nano /etc/systemd/system/mariadb.service

paste ExecStartPre=/bin/sleep 20 between this lines:

ExecStartPre=/bin/sh -c "systemctl unset-environment _WSREP_START_POSITION"
***!HERE!***
ExecStartPre=/bin/sh -c "[ ! -e /usr/bin/galera_recovery ] && VAR= || \
 VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] \
 && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1"

systemctl daemon-reload
systemctl enable mariadb.service

reboot

copying the .service file to /etc/systemd/system will override the system default in /lib/systemd/system, so when the default file will get an update it will not been loaded by your system.
Clean solution would be a correct update by the distribution (does it come upstream from armbian?)


#10

Hooray, that works!
Thank you very much, I would not have figured that out. Nextcloud running again, time for a full backup :slight_smile: