[NCP] NextCloud and MariaDB down

ncp
nc14

#1

Hello,
I have nextcloutpi running on a raspberry pi for a longer time already.
A few days ago nextcloud and the database were suddenly down, I did not change anything.

NextCloudPi version v0.67.5
NextCloudPi image NextCloudPi_03-18-18
distribution Raspbian GNU/Linux 9 \n \l
automount yes
USB devices sda
datadir /media/USBdrive/ncdata
data in SD no
data filesystem btrfs
data disk usage 463G/1.9T
rootfs usage 1.9G/15G
swapfile /var/swap
dbdir /media/USBdrive/ncdatabase
Nextcloud check error
HTTPD service up
PHP service up
MariaDB service down
Redis service up
Postfix service up
internet check ok
port check 80 open
port check 443 open
IP 192.168.178.15
gateway 192.168.178.1
interface eth0
certificates cloud.mydomain.de
NAT loopback no
uptime 22:53

Also a restart does not change anything.
I have an older backup of the SD card from the time I was using nextcloud 13. When I loaded that on a new SD card, Nextcloud started and MariaDB were up, but the database was not used (“dbdir does not exist”), so nextcloud did not knew any users and I could not log on. Also the lets encrypt script in ncp-web threw an error:
./letsencrypt.sh: line 63: letsencrypt: command not found.

Long story short, any way to get things up again? I am even struggling with finding out why database and nextcloud do not start. Luckily I can still access my files via samba and nfs.

systemctl status mariadb.service shows

mariadb.service - MariaDB 10.1.37 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2018-12-01 08:15:05 UTC; 1 day 6h ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 825 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 574 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=/usr/bin/galera_recovery; [ $? -eq 0 ] && syste
Process: 535 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 524 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 825 (code=exited, status=1/FAILURE)
Status: “MariaDB server is down”

Dec 01 08:15:02 nextcloudpi systemd[1]: Starting MariaDB 10.1.37 database server…
Dec 01 08:15:04 nextcloudpi mysqld[825]: 2018-12-01 8:15:04 1996299680 [Note] /usr/sbin/mysqld (mysqld 10.1.37-MariaDB-0+deb9u1) starting as
Dec 01 08:15:04 nextcloudpi mysqld[825]: 2018-12-01 8:15:04 1996299680 [Warning] Can’t create test file /media/USBdrive/ncdatabase/nextcloud
Dec 01 08:15:04 nextcloudpi mysqld[825]: [102B blob data]
Dec 01 08:15:04 nextcloudpi mysqld[825]: 2018-12-01 8:15:04 1996299680 [ERROR] Aborting
Dec 01 08:15:05 nextcloudpi systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Dec 01 08:15:05 nextcloudpi systemd[1]: Failed to start MariaDB 10.1.37 database server.
Dec 01 08:15:05 nextcloudpi systemd[1]: mariadb.service: Unit entered failed state.
Dec 01 08:15:05 nextcloudpi systemd[1]: mariadb.service: Failed with result ‘exit-code’.

Thanks for any help! :slight_smile:


#2

This looks to me like mariadb expects an external USB disk drive, and can’t find it.


#3

That makes no sense to me, but now I see that also nc-fix-permissions says „data dir not found„.
How can this be? The drive is mounted, I can access it via samba, nfs and ftp, why should it suddenly not be accessible to ncp anymore?


#4

Are you using more than one external drive? If so check out this wiki page.
What is output of:

ls -lh /media/

and

df -h


#5

No, there is just one single usb drive attached and that never changed. I could provide the output of the commands this evening when I am at home again.


#6

Maybe same issue i had:
https://help.nextcloud.com/t/after-last-update-mariadb-doesnt-start-after-reboot/42092/2


#7

Thanks, that sounds like a possible candidate, so it would have tried to access the mounted drive before it was actually mounted?! (although I would then expect that many more people have the issue, unless they did not reboot for a long time).
Can you help me out where the file is that I need to maintain?


#8

When i am home again i will do.


#9

ok, back home now…

try following:

ssh into your pi
following commands with sudo or go root:

systemctl disable mariadb
cp /lib/systemd/system/mariadb.service /etc/systemd/system/mariadb.service 
nano /etc/systemd/system/mariadb.service

paste ExecStartPre=/bin/sleep 20 between this lines:

ExecStartPre=/bin/sh -c "systemctl unset-environment _WSREP_START_POSITION"
***!HERE!***
ExecStartPre=/bin/sh -c "[ ! -e /usr/bin/galera_recovery ] && VAR= || \
 VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] \
 && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1"

systemctl daemon-reload
systemctl enable mariadb.service

reboot

copying the .service file to /etc/systemd/system will override the system default in /lib/systemd/system, so when the default file will get an update it will not been loaded by your system.
Clean solution would be a correct update by the distribution (does it come upstream from armbian?)


#10

Hooray, that works!
Thank you very much, I would not have figured that out. Nextcloud running again, time for a full backup :slight_smile:


#11

Many thanks. Issue seemed to have surfaced after updating to v0.67.8 for me.


#12

are you using automount?

@pigeldi @jdc7789

can you share the contents of /lib/systemd/system/mariadb.service?


#13

Yes, using automount and DB on USB disk.

Here’s the file after adding the command referred above…

/etc/systemd/system/mariadb.service

This file is free software; you can redistribute it and/or modify it

under the terms of the GNU Lesser General Public License as published by

the Free Software Foundation; either version 2.1 of the License, or

(at your option) any later version.

Thanks to:

Daniel Black

Erkan Yanar

David Strauss

and probably others

[Unit]
Description=MariaDB 10.1.37 database server
Documentation=man:mysqld(8)
Documentation=https://mariadb.com/kb/en/library/systemd/
After=network.target

[Install]
WantedBy=multi-user.target
Alias=mysql.service
Alias=mysqld.service

[Service]

##############################################################################

Core requirements

Type=notify

Setting this to true can break replication and the Type=notify settings

See also bind-address mysqld option.

PrivateNetwork=false

##############################################################################

Package maintainers

User=mysql
Group=mysql

To allow memlock to be used as non-root user if set in configuration

CapabilityBoundingSet=CAP_IPC_LOCK

Prevent writes to /usr, /boot, and /etc

ProtectSystem=full

Doesn’t yet work properly with SELinux enabled

NoNewPrivileges=true

PrivateDevices=true

Prevent accessing /home, /root and /run/user

ProtectHome=true

Execute pre and post scripts as root, otherwise it does it as User=

PermissionsStartOnly=true

ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld

Perform automatic wsrep recovery. When server is started without wsrep,

galera_recovery simply returns an empty string. In any case, however,

the script is not expected to return with a non-zero status.

It is always safe to unset _WSREP_START_POSITION environment variable.

Do not panic if galera_recovery script is not available. (MDEV-10538)

ExecStartPre=/bin/sh -c “systemctl unset-environment _WSREP_START_POSITION”
ExecStartPre=/bin/sh -c “[ ! -e /usr/bin/galera_recovery ] && VAR= ||
VAR=/usr/bin/galera_recovery; [ $? -eq 0 ]
&& systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1”

Needed to create system tables etc.

ExecStartPre=/usr/bin/mysql_install_db -u mysql

Start main service

MYSQLD_OPTS here is for users to set in /etc/systemd/system/mariadb.service.d/MY_SPECIAL.conf

Use the [Service] section and Environment=“MYSQLD_OPTS=…”.

This isn’t a replacement for my.cnf.

_WSREP_NEW_CLUSTER is for the exclusive use of the script galera_new_cluster

ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION
ExecStartPost=/etc/mysql/debian-start

Unset _WSREP_START_POSITION environment variable.

ExecStartPost=/bin/sh -c “systemctl unset-environment _WSREP_START_POSITION”

KillSignal=SIGTERM

Don’t want to see an automated SIGKILL ever

SendSIGKILL=no

Restart crashed server only, on-failure would also restart, for example, when

my.cnf contains unknown option

Restart=on-abort
RestartSec=5s

UMask=007

##############################################################################

USERs can override

by creating a file in /etc/systemd/system/mariadb.service.d/MY_SPECIAL.conf

and adding/setting the following under [Service] will override this file’s

settings.

Useful options not previously available in [mysqld_safe]

Kernels like killing mysqld when out of memory because its big.

Lets temper that preference a little.

OOMScoreAdjust=-600

Explicitly start with high IO priority

BlockIOWeight=1000

If you don’t use the /tmp directory for SELECT … OUTFILE and

LOAD DATA INFILE you can enable PrivateTmp=true for a little more security.

PrivateTmp=false

Options previously available to be set via [mysqld_safe]

that now needs to be set by systemd config files as mysqld_safe

isn’t executed.

Number of files limit. previously [mysqld_safe] open-file-limit

LimitNOFILE=16364

Maximium core size. previously [mysqld_safe] core-file-size

LimitCore=

Nice priority. previously [mysqld_safe] nice

Nice=-5

Timezone. previously [mysqld_safe] timezone

Environment=“TZ=UTC”

Library substitutions. previously [mysqld_safe] malloc-lib with explicit paths

(in LD_LIBRARY_PATH) and library name (in LD_PRELOAD).

Environment=“LD_LIBRARY_PATH=/path1 /path2” "LD_PRELOAD=

Flush caches. previously [mysqld_safe] flush-caches=1

ExecStartPre=sync

ExecStartPre=sysctl -q -w vm.drop_caches=3

numa-interleave=1 equalivant

Change ExecStart=numactl --interleave=all /usr/sbin/mysqld…

crash-script equalivent

FailureAction=


#14

#15

Thanks. This also fixed my problem. NextcloudPi wasn’t running since end of november.
Launching nc-info
Gathering information…
NextCloudPi version v0.67.8
NextCloudPi image NextCloudPi_02-06-18
distribution Raspbian GNU/Linux 9 \n \l
automount yes
USB devices sda
datadir /media/USBdrive/ncdata
data in SD no
data filesystem ext2/ext3
data disk usage 74G/2.7T
rootfs usage 2.0G/15G
swapfile /media/c9af4f9f-f6c6-4304-b1b9-38c32ac2569a/swap
dbdir /media/USBdrive/ncdatabase
Nextcloud check error
HTTPD service up
PHP service up
MariaDB service down
Redis service up
Postfix service up
internet check ok
port check 80 open
port check 443 open


#16

fixed in v1.3.6. This shouldn’t get overriden again