100% disk, mariaDB won't start again

ℹ️ Support
1

Nextcloud version 12:
Operating system and version Debian 9:
Apache or nginx version nginx 1.10.3:
PHP version 7.0:

First time seeing this error

Steps to replicate it:

  1. Have a great nextcloud 12 instance
  2. Fill your system partition by doing a mistake using a backup command line of large files (nothing to do with nextcloud)
  3. wait

Now I have got a blank page when I try to reach my nextcloud instance using a web browser, I have found
that MariaDB has not started and do not want to start.

root@echoes:/home/mazuyer# systemctl status mariadb.service
● mariadb.service - MariaDB database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2018-03-09 18:15:16 EST; 5min ago
  Process: 1638 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
  Process: 1544 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exite
  Process: 1517 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 1514 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
 Main PID: 1638 (code=exited, status=1/FAILURE)
   Status: "MariaDB server is down"

Mar 09 18:15:13 echoes systemd[1]: Starting MariaDB database server...
Mar 09 18:15:14 echoes mysqld[1638]: 2018-03-09 18:15:14 140534516355648 [Note] /usr/sbin/mysqld (mysqld 10.1.26-MariaDB-0+deb9u1) starting as process 1638 ...
Mar 09 18:15:16 echoes systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Mar 09 18:15:16 echoes systemd[1]: Failed to start MariaDB database server.
Mar 09 18:15:16 echoes systemd[1]: mariadb.service: Unit entered failed state.
Mar 09 18:15:16 echoes systemd[1]: mariadb.service: Failed with result 'exit-code'.

Have you got any idea to start/repair MariaDB ? if possible, I do not want to lose the data that are on my nextcloud database…

Thanks !

2

Quick solved :slightly_smiling_face:

I have look at the MariaDB log:

/var/log/mysql/error.log

2018-03-09 18:13:00 140343736226368 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2018-03-09 18:13:00 140343736226368 [Note] InnoDB: The InnoDB memory heap is disabled
2018-03-09 18:13:00 140343736226368 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-03-09 18:13:00 140343736226368 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2018-03-09 18:13:00 140343736226368 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018-03-09 18:13:00 140343736226368 [Note] InnoDB: Using Linux native AIO
2018-03-09 18:13:00 140343736226368 [Note] InnoDB: Using SSE crc32 instructions
2018-03-09 18:13:00 140343736226368 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2018-03-09 18:13:00 140343736226368 [Note] InnoDB: Completed initialization of buffer pool
2018-03-09 18:13:00 140343736226368 [Note] InnoDB: Highest supported file format is Barracuda.
2018-03-09 18:13:00 140343736226368 [Note] InnoDB: 128 rollback segment(s) are active.
2018-03-09 18:13:00 140343736226368 [Note] InnoDB: Waiting for purge to start
2018-03-09 18:13:00 140343736226368 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 166024022
2018-03-09 18:13:00 140343736226368 [Note] Plugin 'FEEDBACK' is disabled.
2018-03-09 18:13:00 140343076976384 [Note] InnoDB: Dumping buffer pool(s) not yet started
2018-03-09 18:13:00 140343736226368 [Note] Recovering after a crash using tc.log
2018-03-09 18:13:00 140343736226368 [ERROR] Can't init tc log
2018-03-09 18:13:00 140343736226368 [ERROR] Aborting

If you delete /var/lib/mysql/tc.log, it appears to work

Sorry for the inconvenience, I let it here in case someone googling the same problem

1 Like
3

Ran into this issue as well. After filling the disc space 100% could only ssh in. After freeing up space, deleting /var/lib/mysql/tc.log and restarting mysql it worked again. Thanks for the solution. This seems to be a bug in MariaDB?

4

I had the excactly the same problem. My Nextcloud installation was not accessible via web interface any more. The only possibility to connect was via ssh session.

I found out that there was no more disk space left and querying mariadb status with systemctl status mariadb.service showed the mariadb server was not running any more.

After deleting /var/lib/mysql/tc.log and restarting everything worked fine again.

Thanks a lot for your help, folks!!

Thanks AntoineMazuyer for leaving the solution for people like myself who found it through goooogling… :smile:

:slight_smile: Martin

5

This helped me with my owncloud installation. After extending the harddrive my OC was not booting correctly; the service MariaDB seamed to have crashed in the proces of extending the hdd. Deleting this logfile made it possible to restart mariaDB again.

I’m curious why this log prevents MariaDB from starting up? This log was 0 KB in size?