MariaDB - Corrupt OC filecache - after crash

ℹ️ Support
, ,
1

So I am running nextcloud on Debian as a NAS and yesterday it crashed.
So I fixed the corrupt nodes, but after that MariaDB errored something like files exceeded limit, so after researching I set:
LimitNOFILE=100000
LimitMEMLOCK=100000
inside /etc/systemd/system/mysqld.service.

But after that trying to run systemctl restart mariadb.service, gives:

mariadb.service - MariaDB 10.3.27 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2021-01-05 21:48:57 CET; 33min ago
Docs: man:mysqld(8)
    https://mariadb.com/kb/en/library/systemd/
Process: 5416 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Process: 5417 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 5419 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-e
Process: 5466 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Main PID: 5466 (code=exited, status=1/FAILURE)
    Status: "MariaDB server is down"

And in /var/log/mysql/error.log:

2021-01-05 21:48:56 0 [Note] InnoDB: Using Linux native AIO
2021-01-05 21:48:56 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-01-05 21:48:56 0 [Note] InnoDB: Uses event mutexes
2021-01-05 21:48:56 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-01-05 21:48:56 0 [Note] InnoDB: Number of pools: 1
2021-01-05 21:48:56 0 [Note] InnoDB: Using SSE2 crc32 instructions
2021-01-05 21:48:56 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2021-01-05 21:48:56 0 [Note] InnoDB: Completed initialization of buffer pool
2021-01-05 21:48:56 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See $
2021-01-05 21:48:56 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=18460758099
2021-01-05 21:48:56 0 [Note] InnoDB: Header page consists of zero bytes in datafile: ./nextclouddb/oc_filecache.ibd, Space ID:0, F$
2021-01-05 21:48:56 0 [Note] InnoDB: Page size:1024. Pages to analyze:64
2021-01-05 21:48:56 0 [Note] InnoDB: Page size: 1024. Possible space_id count:0
2021-01-05 21:48:56 0 [Note] InnoDB: Page size:2048. Pages to analyze:64
2021-01-05 21:48:56 0 [Note] InnoDB: Page size: 2048. Possible space_id count:0
2021-01-05 21:48:56 0 [Note] InnoDB: Page size:4096. Pages to analyze:64
2021-01-05 21:48:56 0 [Note] InnoDB: Page size: 4096. Possible space_id count:0
2021-01-05 21:48:56 0 [Note] InnoDB: Page size:8192. Pages to analyze:64
2021-01-05 21:48:56 0 [Note] InnoDB: Page size: 8192. Possible space_id count:0
2021-01-05 21:48:56 0 [Note] InnoDB: Page size:16384. Pages to analyze:64
2021-01-05 21:48:56 0 [Note] InnoDB: Page size: 16384. Possible space_id count:0
2021-01-05 21:48:56 0 [Note] InnoDB: Page size:32768. Pages to analyze:64
2021-01-05 21:48:56 0 [Note] InnoDB: Page size: 32768. Possible space_id count:0
2021-01-05 21:48:56 0 [Note] InnoDB: Page size:65536. Pages to analyze:64
2021-01-05 21:48:56 0 [Note] InnoDB: Page size: 65536. Possible space_id count:0
2021-01-05 21:48:56 0 [ERROR] InnoDB: Datafile './nextclouddb/oc_filecache.ibd' is corrupted. Cannot determine the space ID from t$
2021-01-05 21:48:56 0 [ERROR] InnoDB: Tablespace 144 was not found at ./nextclouddb/oc_filecache.ibd.
2021-01-05 21:48:56 0 [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore this and to permanently lose all changes to the tables$
2021-01-05 21:48:56 0 [ERROR] InnoDB: Plugin initialization aborted with error Tablespace not found
2021-01-05 21:48:56 0 [Note] InnoDB: Starting shutdown...
2021-01-05 21:48:57 0 [ERROR] Plugin 'InnoDB' init function returned error.
2021-01-05 21:48:57 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2021-01-05 21:48:57 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-01-05 21:48:57 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2021-01-05 21:48:57 0 [ERROR] Aborting

The important line is:
[ERROR] InnoDB: Datafile './nextclouddb/oc_filecache.ibd' is corrupted. . Cannot determine the space ID from the first 64 pages.

When trying to run occ:

Doctrine\DBAL\DBALException: Failed to connect to the database: An exception occurred in driver: SQLSTATE[HY000] [2002] No such file or directory in /var/www/html/nextcloud/lib/private/DB/Connection.php:72
Stack trace:
#0 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(1449): OC\DB\Connection->connect()
#1 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(892): Doctrine\DBAL\Connection->getWrappedConnection()

When trying to run mysql:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

I tried researching corrupt oc filecache, but most posts suggest using occ to fix it, but occ does not work for me either.
And I can’t make a backup of the database since I can’t run mysql.

Can anyone help me with this error please?

Addition info:
Most of my files are inside smb shares and are mounted as external storage in nextcloud.
I have around 300gb of used storage in those smb shares.
I had a crontab that remove the cache and swap cache, with:
echo 3 > /proc/sys/vm/drop_caches && swapoff -a && swapon -a

2

You should do a backup before your system breaks. Then you can pick up from this point. You could give up the whole table and start with a new one (be careful if you have sync clients connected), or you find some way to fix the corrupted files.

3

How can I backup, I can’t start the database neither run mysql command.

And what happens if I drop the table, will nextcloud just recreate it? And will this affect my files, will they be deleted if I drop the table?

I do have nextcloud sync clients, but I can backup those folders.

4

Yes, backup these files. Could be that the client deletes the files when it resyncs with the server.

The trick is to do this before. Now it’s too late.

Not sure if it creates the table itself or if you have to create an empty one yourself. With the occ command, you can rescan all the files, this will probably create new “timestamps”, the server version will be considered to be newer and all the clients will download all the files again. If changes were done on the client, these versions could easily be lost, so make a backup.

If you use shares, shared links etc. this is linked to the files in the filecache-table, so all these references will be lost.

Make sure to save everything you have right now that in worst case, you can always go back to the current state and don’t lose even more.

5

Problem was I could not run occ as it could not connect to mariadb as it was corrupted, so ended up reinstalling everything, thanks for the help!