NCP down (MariaDB down)

My NextcloudPi suddenly stopped working.
This happend a few weeks ago and after rebooting it, it worked again for a couple of days. Now again about a week ago it stopped working again. This time a reboot didn’t fix the problem. I managed to login via SSH and generate a report (pasted below). Apparently the database is down and I have no idea how to fix this.
I run NCP on a RaspberryPi 4, that worked fine for a couple of months, before the problem started.

Unfortunately my linux skills are rather limited. Does anyone now how to fix this problem or can help me in further diagnosing?
Thanks a lot!

When I try to reach the NCP via URL, the following message appears:

Internal Server Error
The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.

NextCloudPi diagnostics

NextCloudPi version  v1.35.0
NextCloudPi image    NextCloudPi_12-20-19
distribution         Raspbian GNU/Linux 10 \n \l
automount            yes
USB devices          sda
datadir              /media/myCloudDrive/ncdata
data in SD           no
data filesystem      btrfs
data disk usage      257G/350G
rootfs usage         3.0G/7.1G
swapfile             /var/swap
dbdir                /var/lib/mysql
Nextcloud check      error
HTTPD service        up
PHP service          up
MariaDB service      up
Redis service        up
Postfix service      up
internet check       ok
port check 80        open
port check 443       open
IP                   ***REMOVED SENSITIVE VALUE***
gateway              ***REMOVED SENSITIVE VALUE***
interface            eth0
certificates         ***REMOVED SENSITIVE VALUE***
NAT loopback         no
uptime               30min

Nextcloud configuration

An unhandled exception has been thrown:
Doctrine\DBAL\DBALException: Failed to connect to the database: An exception occurred in driver: SQLSTATE[HY000] [2002] No such file or directory in /var/www/nextcloud/lib/private/DB/Connection.php:72
Stack trace:
#0 /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(1449): OC\DB\Connection->connect()
#1 /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(892): Doctrine\DBAL\Connection->getWrappedConnection()
#2 /var/www/nextcloud/lib/private/DB/Connection.php(202): Doctrine\DBAL\Connection->executeQuery('SELECT * FROM `...', Array, Array, NULL)
#3 /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Query/QueryBuilder.php(206): OC\DB\Connection->executeQuery('SELECT * FROM `...', Array, Array)
#4 /var/www/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php(217): Doctrine\DBAL\Query\QueryBuilder->execute()
#5 /var/www/nextcloud/lib/private/AppConfig.php(345): OC\DB\QueryBuilder\QueryBuilder->execute()
#6 /var/www/nextcloud/lib/private/AppConfig.php(110): OC\AppConfig->loadConfigValues()
#7 /var/www/nextcloud/lib/private/AppConfig.php(301): OC\AppConfig->getApps()
#8 /var/www/nextcloud/lib/private/legacy/OC_App.php(949): OC\AppConfig->getValues(false, 'installed_versi...')
#9 /var/www/nextcloud/lib/private/Server.php(668): OC_App::getAppVersions()
#10 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(155): OC\Server->OC\{closure}(Object(OC\Server))
#11 /var/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(Pimple\Container))
#12 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(122): Pimple\Container->offsetGet('OC\\Memcache\\Fac...')
#13 /var/www/nextcloud/lib/private/ServerContainer.php(156): OC\AppFramework\Utility\SimpleContainer->query('OC\\Memcache\\Fac...', true)
#14 /var/www/nextcloud/lib/private/Server.php(1677): OC\ServerContainer->query('OC\\Memcache\\Fac...')
#15 /var/www/nextcloud/lib/private/Server.php(1017): OC\Server->getMemCacheFactory()
#16 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(155): OC\Server->OC\{closure}(Object(OC\Server))
#17 /var/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(Pimple\Container))
#18 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(122): Pimple\Container->offsetGet('OCP\\Lock\\ILocki...')
#19 /var/www/nextcloud/lib/private/ServerContainer.php(156): OC\AppFramework\Utility\SimpleContainer->query('OCP\\Lock\\ILocki...', true)
#20 /var/www/nextcloud/lib/private/Server.php(1977): OC\ServerContainer->query('OCP\\Lock\\ILocki...')
#21 /var/www/nextcloud/lib/private/Files/View.php(118): OC\Server->getLockingProvider()
#22 /var/www/nextcloud/lib/private/Server.php(395): OC\Files\View->__construct()
#23 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(155): OC\Server->OC\{closure}(Object(OC\Server))
#24 /var/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(Pimple\Container))
#25 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(122): Pimple\Container->offsetGet('OC\\Files\\Node\\H...')
#26 /var/www/nextcloud/lib/private/ServerContainer.php(156): OC\AppFramework\Utility\SimpleContainer->query('OC\\Files\\Node\\H...', true)
#27 /var/www/nextcloud/lib/private/Server.php(1324): OC\ServerContainer->query('OC\\Files\\Node\\H...')
#28 /var/www/nextcloud/lib/base.php(595): OC\Server->boot()
#29 /var/www/nextcloud/lib/base.php(1091): OC::init()
#30 /var/www/nextcloud/console.php(49): require_once('/var/www/nextcl...')
#31 /var/www/nextcloud/occ(11): require_once('/var/www/nextcl...')
#32 {main}
HTTPd logs

[Thu Feb 04 00:00:02.148862 2021] [ssl:warn] [pid 912:tid 3069805072] AH01909: localhost:4443:0 server certificate does NOT include an ID which matches the server name
[Thu Feb 04 00:00:02.154467 2021] [mpm_event:notice] [pid 912:tid 3069805072] AH00489: Apache/2.4.38 (Raspbian) OpenSSL/1.1.1d configured -- resuming normal operations
[Thu Feb 04 00:00:02.154489 2021] [core:notice] [pid 912:tid 3069805072] AH00094: Command line: '/usr/sbin/apache2'
[Thu Feb 04 04:00:00.786797 2021] [mpm_event:notice] [pid 912:tid 3069805072] AH00491: caught SIGTERM, shutting down
[Fri Apr 09 04:30:43.328421 2021] [ssl:warn] [pid 772:tid 3069788688] AH01909: localhost:4443:0 server certificate does NOT include an ID which matches the server name
[Fri Apr 09 04:30:43.364619 2021] [ssl:warn] [pid 958:tid 3069788688] AH01909: localhost:4443:0 server certificate does NOT include an ID which matches the server name
[Fri Apr 09 04:30:43.370275 2021] [mpm_event:notice] [pid 958:tid 3069788688] AH00489: Apache/2.4.38 (Raspbian) OpenSSL/1.1.1d configured -- resuming normal operations
[Fri Apr 09 04:30:43.370365 2021] [core:notice] [pid 958:tid 3069788688] AH00094: Command line: '/usr/sbin/apache2'
[Fri Apr 09 16:50:53.735220 2021] [proxy_fcgi:error] [pid 962:tid 2780902432] [client 192.168.178.53:51863] AH01067: Failed to read FastCGI header
[Fri Apr 09 16:50:53.735274 2021] [proxy_fcgi:error] [pid 962:tid 2780902432] (104)Connection reset by peer: [client 192.168.178.53:51863] AH01075: Error dispatching request to :4443:
[Fri Apr 09 16:50:53.937595 2021] [mpm_event:notice] [pid 958:tid 3069788688] AH00491: caught SIGTERM, shutting down
[Sat May 01 16:29:11.042315 2021] [ssl:warn] [pid 736:tid 3069903376] AH01909: localhost:4443:0 server certificate does NOT include an ID which matches the server name
[Sat May 01 16:29:11.079684 2021] [ssl:warn] [pid 921:tid 3069903376] AH01909: localhost:4443:0 server certificate does NOT include an ID which matches the server name
[Sat May 01 16:29:11.085794 2021] [mpm_event:notice] [pid 921:tid 3069903376] AH00489: Apache/2.4.38 (Raspbian) OpenSSL/1.1.1d configured -- resuming normal operations
[Sat May 01 16:29:11.085876 2021] [core:notice] [pid 921:tid 3069903376] AH00094: Command line: '/usr/sbin/apache2'
[Sat May 01 16:51:45.098947 2021] [mpm_event:notice] [pid 921:tid 3069903376] AH00491: caught SIGTERM, shutting down
[Sat May 01 16:52:11.616112 2021] [ssl:warn] [pid 733:tid 3069587984] AH01909: localhost:4443:0 server certificate does NOT include an ID which matches the server name
[Sat May 01 16:52:11.656469 2021] [ssl:warn] [pid 919:tid 3069587984] AH01909: localhost:4443:0 server certificate does NOT include an ID which matches the server name
[Sat May 01 16:52:11.662801 2021] [mpm_event:notice] [pid 919:tid 3069587984] AH00489: Apache/2.4.38 (Raspbian) OpenSSL/1.1.1d configured -- resuming normal operations
[Sat May 01 16:52:11.662910 2021] [core:notice] [pid 919:tid 3069587984] AH00094: Command line: '/usr/sbin/apache2'

Database logs

2021-05-01 17:22:17 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-05-01 17:22:17 0 [Note] InnoDB: Number of pools: 1
2021-05-01 17:22:17 0 [Note] InnoDB: Using generic crc32 instructions
2021-05-01 17:22:17 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 1, chunk size = 128M
2021-05-01 17:22:18 0 [Note] InnoDB: Completed initialization of buffer pool
2021-05-01 17:22:18 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2021-05-01 17:22:18 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2021-05-01 17:22:18 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-05-01 17:22:18 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-05-01 17:22:18 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2021-05-01 17:22:18 0 [Note] InnoDB: 10.3.27 started; log sequence number 4730829585; transaction id 16168289
2021-05-01 17:22:18 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2021-05-01 17:22:18 0 [Note] InnoDB: Buffer pool(s) load completed at 210501 17:22:18
2021-05-01 17:22:18 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-05-01 17:22:18 0 [Note] Recovering after a crash using tc.log
2021-05-01 17:22:18 0 [ERROR] Bad magic header in tc log
2021-05-01 17:22:18 0 [ERROR] Crash recovery failed. Either correct the problem (if it's, for example, out of memory error) and restart, or delete tc log and start mysqld with --tc-heuristic-recover={commit|rollback}
2021-05-01 17:22:18 0 [ERROR] Can't init tc log
2021-05-01 17:22:18 0 [ERROR] Aborting


Nextcloud logs (shortend)

{"reqId":"YI1-c9X2q0sDmOm-N6POdgAADwA","level":3,"time":"2021-05-01T16:18:59+00:00","remoteAddr":"93.198.52.183","user":"--","app":"index","method":"GET","url":"/index.php/apps/photos/service-worker.js","message":{"Exception":"Doctrine\\DBAL\\DBALException","Message":"Failed to connect to the database: An exception occurred in driver: SQLSTATE[HY000] [2002] No such file or directory","Code":0,"Trace":[{"file":"/var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php","line":1449,"function":"connect","class":"OC\\DB\\Connection","type":"->","args":[]},{"file":"/var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php","line":892,"function":"getWrappedConnection","class":"Doctrine\\DBAL\\Connection","type":"->","args":[]},{"file":"/var/www/nextcloud/lib/private/DB/Connection.php","line":202,"function":"executeQuery","class":"Doctrine\\DBAL\\Connection","type":"->","args":["SELECT * FROM `oc_appconfig`",[],[],null]},{"file":"/var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Query/QueryBuilder.php","line":206,"function":"executeQuery","class":"OC\\DB\\Connection","type":"->","args":["SELECT * FROM `oc_appconfig`",[],[]]},{"file":"/var/www/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php","line":217,"function":"execute","class":"Doctrine\\DBAL\\Query\\QueryBuilder","type":"->","args":[]},{"file":"/var/www/nextcloud/lib/private/AppConfig.php","line":345,"function":"execute","class":"OC\\DB\\QueryBuilder\\QueryBuilder","type":"->","args":[]},{"file":"/var/www/nextcloud/lib/private/AppConfig.php","line":110,"function":"loadConfigValues","class":"OC\\AppConfig","type":"->","args":[]},{"file":"/var/www/nextcloud/lib/private/AppConfig.php","line":301,"function":"getApps","class":"OC\\AppConfig","type":"->","args":[]},{"file":"/var/www/nextcloud/lib/private/legacy/OC_App.php","line":949,"function":"getValues","class":"OC\\AppConfig","type":"->","args":[false,"installed_version"]},{"file":"/var/www/nextcloud/lib/private/Server.php","line":668,"function":"getAppVersions","class":"OC_App","type":"::","args":[]},{"file":"/var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php","line":155,"function":"OC\\{closure}","class":"OC\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php","line":118,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php","line":122,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OC\\Memcache\\Factory"]},{"file":"/var/www/nextcloud/lib/private/ServerContainer.php","line":156,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Memcache\\Factory",true]},{"file":"/var/www/nextcloud/lib/private/Server.php","line":1677,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OC\\Memcache\\Factory"]},{"file":"/var/www/nextcloud/lib/private/Server.php","line":1017,"function":"getMemCacheFactory","class":"OC\\Server","type":"->","args":[]},{"file":"/var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php","line":155,"function":"OC\\{closure}","class":"OC\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php","line":118,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php","line":122,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OCP\\Lock\\ILockingProvider"]},{"file":"/var/www/nextcloud/lib/private/ServerContainer.php","line":156,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OCP\\Lock\\ILockingProvider",true]},{"file":"/var/www/nextcloud/lib/private/Server.php","line":1977,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OCP\\Lock\\ILockingProvider"]},{"file":"/var/www/nextcloud/lib/private/Files/View.php","line":118,"function":"getLockingProvider","class":"OC\\Server","type":"->","args":[]},{"file":"/var/www/nextcloud/lib/private/Server.php","line":395,"function":"__construct","class":"OC\\Files\\View","type":"->","args":[]},{"file":"/var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php","line":155,"function":"OC\\{closure}","class":"OC\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php","line":118,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php","line":122,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OC\\Files\\Node\\HookConnector"]},{"file":"/var/www/nextcloud/lib/private/ServerContainer.php","line":156,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Files\\Node\\HookConnector",true]},{"file":"/var/www/nextcloud/lib/private/Server.php","line":1324,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OC\\Files\\Node\\HookConnector"]},{"file":"/var/www/nextcloud/lib/base.php","line":595,"function":"boot","class":"OC\\Server","type":"->","args":[]},{"file":"/var/www/nextcloud/lib/base.php","line":1091,"function":"init","class":"OC","type":"::","args":[]},{"file":"/var/www/nextcloud/index.php","line":35,"args":["/var/www/nextcloud/lib/base.php"],"function":"require_once"}],"File":"/var/www/nextcloud/lib/private/DB/Connection.php","Line":72,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36","version":"20.0.8.1"}


Regarding the database status I tried pi@nextcloudpi:~ $ sudo service mysql status , what results in:

Warning: The unit file, source configuration file or drop-ins of mariadb.service changed on disk. Run 'systemctl daemon-reload' to reload units.
● mariadb.service - MariaDB 10.3.27 database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─ncp-delay-automount.conf
   Active: activating (start-pre) since Sat 2021-05-01 17:33:29 BST; 9s ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 14790 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
  Process: 14791 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 14793 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSI
Cntrl PID: 14888 (sleep)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/mariadb.service
           └─14888 /bin/sleep 20

May 01 17:33:29 nextcloudpi systemd[1]: Starting MariaDB 10.3.27 database server...

After executing systemctl daemon-reload (and authenticating) and trying again I get:

● mariadb.service - MariaDB 10.3.27 database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset:
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─ncp-delay-automount.conf
   Active: activating (auto-restart) (Result: exit-code) since Sat 2021-05-01 17
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 16702 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/r
  Process: 16703 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_STAR
  Process: 16705 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VA
  Process: 16800 ExecStartPre=/bin/sleep 20 (code=exited, status=0/SUCCESS)
  Process: 16855 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_W
 Main PID: 16855 (code=exited, status=1/FAILURE)
   Status: "MariaDB server is down"

After a reboot, same picture as before using systemctl daemon-reload… Any help welcome!