Internal server error after hard reset - mariadb

Hi all.

Somebody got my fuse down so my server faced a hard reset after powerloss. Before, everything was working well, but now I have a problem…:
When I try to acces via http I get this:

Internal Server Error

The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.

First I thought it was apache, but I was wrong. Where httpd is running, mariadb service looks like this:

● mariadb.service - MariaDB 10.3 database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: signal) since Tue 2021-08-24 16:30:32 EDT; 2s ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 1440161 ExecStart=/usr/libexec/mysqld --basedir=/usr $MYSQLD_OPTS $_WSREP_NEW_CLUSTER (code=killed, signal=SEGV)
Process: 1440123 ExecStartPre=/usr/libexec/mysql-prepare-db-dir mariadb.service (code=exited, status=0/SUCCESS)
Process: 1440094 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
Main PID: 1440161 (code=killed, signal=SEGV)
Status: “Starting final batch to recover 6 pages from redo log”
Tasks: 0 (limit: 49172)
Memory: 876.0K
CGroup: /system.slice/mariadb.service

Aug 24 16:30:32 hxx.local systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
Aug 24 16:30:32 xx.local systemd[1]: mariadb.service: Failed with result ‘signal’.
Aug 24 16:30:32 xx.local systemd[1]: Failed to start MariaDB 10.3 database server.

and:

journalctl -xe
#8 0x00007f7e0f61edc3 __clone (libc.so.6)

                                                                Stack trace of thread 1446457:
                                                                #0  0x00007f7e0f61952d syscall (libc.so.6)
                                                                #1  0x00007f7e11374b7e __io_getevents_0_4 (libaio.so.1)
                                                                #2  0x0000556a8ebcfb53 _ZN15LinuxAIOHandler7collectEv (mysql>
                                                                #3  0x0000556a8ebd3520 _ZN15LinuxAIOHandler4pollEPP10fil_nod>
                                                                #4  0x0000556a8ebd495a _Z14os_aio_handlermPP10fil_node_tPPvP>
                                                                #5  0x0000556a8ed1a8a7 _Z12fil_aio_waitm (mysqld)
                                                                #6  0x0000556a8ec521f8 io_handler_thread (mysqld)
                                                                #7  0x00007f7e1179c14a start_thread (libpthread.so.0)
                                                                #8  0x00007f7e0f61edc3 __clone (libc.so.6)
                                                                
                                                                Stack trace of thread 1446462:
                                                                #0  0x00007f7e117a22fc pthread_cond_wait@@GLIBC_2.3.2 (libpt>
                                                                #1  0x0000556a8ebd5b49 _ZN8os_event8wait_lowEl (mysqld)
                                                                #2  0x0000556a8ecd7efb buf_flush_page_cleaner_worker (mysqld)
                                                                #3  0x00007f7e1179c14a start_thread (libpthread.so.0)
                                                                #4  0x00007f7e0f61edc3 __clone (libc.so.6)
                                                                
                                                                Stack trace of thread 1446451:
                                                                #0  0x00007f7e0f61952d syscall (libc.so.6)
                                                                #1  0x00007f7e11374b7e __io_getevents_0_4 (libaio.so.1)
                                                                #2  0x0000556a8ebcfb53 _ZN15LinuxAIOHandler7collectEv (mysql>
                                                                #3  0x0000556a8ebd3520 _ZN15LinuxAIOHandler4pollEPP10fil_nod>
                                                                #4  0x0000556a8ebd495a _Z14os_aio_handlermPP10fil_node_tPPvP>
                                                                #5  0x0000556a8ed1a8a7 _Z12fil_aio_waitm (mysqld)
                                                                #6  0x0000556a8ec521f8 io_handler_thread (mysqld)
                                                                #7  0x00007f7e1179c14a start_thread (libpthread.so.0)
                                                                #8  0x00007f7e0f61edc3 __clone (libc.so.6)
                                                                
                                                                Stack trace of thread 1446460:
                                                                #0  0x00007f7e117a22fc pthread_cond_wait@@GLIBC_2.3.2 (libpt>
                                                                #1  0x0000556a8ebd5b49 _ZN8os_event8wait_lowEl (mysqld)
                                                                #2  0x0000556a8ecd7efb buf_flush_page_cleaner_worker (mysqld)
                                                                #3  0x00007f7e1179c14a start_thread (libpthread.so.0)
                                                                #4  0x00007f7e0f61edc3 __clone (libc.so.6)

– Subject: Process 1446446 (mysqld) dumped core
– Defined-By: systemd
– Support: https://access.redhat.com/support
– Documentation: man:core(5)

– Process 1446446 (mysqld) crashed and dumped core.

– This usually indicates a programming error in the crashing program and
– should be reported to its vendor as a bug.

Now I do really not have any idea what to do.
Some guys wrote to reinstall if there is no content, but I do have content.
Any hint?

My environment:
OS: CentOS 8
Nextcloud 21.0.x
httpd as webserver
(dead)mariadb as db

Thanks for any hints.

nobody here who could help?
I really have no idea, so any hints? Please?

Probably you didn’t get any replies because this really isn’t a nextcloud issue at all. Your mariadb is segfaulting. You might have been in a broken state before the hard reboot as a result of some update that didn’t work out, but since you didn’t restart mariadb after that, it may have still linked against older libraries.

What I would try is a full dnf update and reboot.

You also should consider switching to Rocky Linux (https://rockylinux.org/) which is the new CentOS, since IBM has effectively killed CentOS, and your version 8 is EOL in just 4 months! Use the migration tool: GitHub - rocky-linux/rocky-tools

1 Like

Larry_Boyd thanks for this hint.
I’ve just used this to migrate my vm.
Thanks

1 Like