DB - index corrupted

Nextcloud version (eg, 12.0.2): 14.0.3
Operating system and version (eg, Ubuntu 17.04): Ubuntu Server 18.04.1
Apache or nginx version (eg, Apache 2.4.25): Apache/2.4.29
PHP version (eg, 7.1): 7.2.10
DB: MariaDB 10.1.34

The issue you are facing:

The short version is that an index of one (or more, not sure) of the database tables got corrupted. As to how that might have happened, a few days ago I found out that my entire filesystem was read-only. I used fsck to fix the issues and that’s probably how the index became corrupted. Here is the error from Apache2’s error.log:

PHP Fatal error:  Uncaught PDOException: SQLSTATE[HY000]: General error: 1712 Index oc_file_locks is corrupted in /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php:105\nStack trace:\n#0 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php(105): PDOStatement->execute(NULL)\n#1 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(1006): Doctrine\\DBAL\\Driver\\PDOStatement->execute()\n#2 /var/www/html/lib/private/DB/Connection.php(216): Doctrine\\DBAL\\Connection->executeUpdate('UPDATE `oc_file...', Array, Array)\n#3 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Query/QueryBuilder.php(208): OC\\DB\\Connection->executeUpdate('UPDATE `oc_file...', Array, Array)\n#4 /var/www/html/lib/private/DB/QueryBuilder/QueryBuilder.php(214): Doctrine\\DBAL\\Query\\QueryBuilder->execute()\n#5 /var/www/html/lib/private/Lock/DBLockingProvider.php(310): OC\\DB\\QueryBuilder\\QueryBuilder->execute()\n#6 [internal function]: OC\\Lock\\DBLockingProvider->releaseAll()\n#7 {main}\n\nNext Doctrine\\D in /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/AbstractMySQLDriver.php on line 115

The same error comes when I try to do mysqldump:

mysqldump: Error 1712: Index oc_file_locks is corrupted when dumping table `oc_file_locks` at row: 0

What options do I have? Is it possible to recreate the index? If so, how? Or I could restore a mysqldump from a few days ago, before it happened. Will I be able to update the database with the new files?
Another option I see is to replace the current virtual machine with a backup from before 3 weeks (the nextcloud is running inside a VM with a local mariadb). Data folder is on another physical hard disk drive, so it shouldn’t be corrupted. Is there a way to update the database with the new files that the users have uploaded/changed in those 3 weeks?
Any hints are appreciated.

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it: Probably not, the original cause of issue might be a failing HDD

The output of your Nextcloud log in Admin > Logging: I cannot log in, all requests end up on Nextcloud’s “Internal Server Error” page. I have added logging to config.php and the error message is the same as above: index of oc_file_locks is corrupted.

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => 'censored',
  'passwordsalt' => 'censored',
  'secret' => 'censored',
  'trusted_domains' =>
  array (
    0 => 'cloud.domain1',
    1 => 'cloud.domain2'
  ),
  'trusted_proxies' =>
  array (
    0 =>
    array (
      0 => 'reverse proxy IP',
    ),
  ),
  'forwarded_for_headers' =>
  array (
    0 => 'HTTP_X_FORWARDED',
    1 => 'HTTP_FORWARDED_FOR',
  ),
  'datadirectory' => '/mnt/cloud-data/',
  'overwrite.cli.url' => 'https://cloud.domain1',
  'dbtype' => 'mysql',
  'version' => '14.0.3.0',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'censored',
  'dbpassword' => 'censored',
  'installed' => true,
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'mail_domain' => 'cloud.domain1',
  'mail_from_address' => 'nextcloud',
  'mail_smtpmode' => 'sendmail',
  'mail_smtpauthtype' => 'LOGIN',
  'maintenance' => false,
  'theme' => '',
  'loglevel' => 2,
  'updater.secret' => 'censored',
);

The output of your Apache/nginx/system log in /var/log/____:

No new error entries in /var/logs/apache2/errors.log 

You can try to repair the database with mysqlcheck.

E.G. mysqlcheck -u $USER -p --auto-repair --check nextcloud

Thank you for the suggestion. The check found single corrupted table – oc_file_locks, but was unable to repair it:

Repairing tables
nextcloud.oc_file_locks
Error    : Table 'nextcloud.oc_file_locks' doesn't exist in engine
status   : Operation failed

Is it possible to drop the table oc_file_locks and recreate it?

Yes you can recreate it.

But this sounds like a bigger underlying issue.
I Would check the Filesystem and the rights (and selinux context) of the database files.

Do you know how to recreate the table, please? I tried to search for how to do it but got nothing. Thanks!

As mentioned in the original post, I suspect it is my HDD failing. I have already checked the filesystem and this corruption in the database is most likely the result of the fixes. I need to get my nextcloud into working shape so that I can do one more final backup and then I will reinstall my entire system on a new SSD.

I pulled a sql dump from backup from before the corruption occurred and replace the entire database. Re-scanned all files and now things are looking stable.

@rtznprmpftl if you know how to recreate the table, post it please. Perhaps other users will find it in case they don’t have sql backups. Thanks!

1 Like

For the Nextcloud version 14.04 this would be:

mysql> DROP TABLE oc_file_locks;
mysql> CREATE TABLE `oc_file_locks` (
  `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `lock` int(11) NOT NULL DEFAULT '0',
  `key` varchar(64) COLLATE utf8mb4_bin NOT NULL,
  `ttl` int(11) NOT NULL DEFAULT '-1',
  PRIMARY KEY (`id`),
  UNIQUE KEY `lock_key_index` (`key`),
  KEY `lock_ttl_index` (`ttl`)
)

Thanks! Can you perhaps also add where to find this information? I was even browsing through nextcloud server source codes and couldn’t find it :slight_smile:

The easiest way is to install NC.

The Information itself is stored in non RDBMS Specific Migration Files: https://github.com/nextcloud/server/tree/master/core/Migrations therefore quite unreadable

I know this thread is rather old, but it was the first in the search results, therefore I will share my observations on this.

I use Nextcloud 22 and tonight my MariaDB reported [ERROR] InnoDB: Table nextcloud.oc_filecache is corrupted. Please drop the table and recreate.
Also Nextcloud cries with PDOException: SQLSTATE[HY000]: General error: 1712 Index oc_filecache is corrupted in /var/www/nextcloud/3rdparty/doctrine/dbal/src/Driver/PDO/Statement.php:92
Trying REPAIR TABLE oc_filecache didn’t do anything. So I did the following:

  • Put Nextcloud into maintenance mode.
  • Stopped MariaDB with mysqladmin -p shutdown because systemd reported an error and was not able to stop it:
    # systemctl status mysql
      mysql.service - LSB: Start and stop the mysql database server daemon
       Loaded: loaded (/etc/init.d/mysql; generated)
       Active: failed (Result: exit-code) since Wed 2021-12-01 08:54:50 CET; 18s ago
         Docs: man:systemd-sysv-generator(8)
      Process: 20558 ExecStop=/etc/init.d/mysql stop (code=exited, status=1/FAILURE)
        Tasks: 35 (limit: 4915)
       Memory: 412.3M
       CGroup: /system.slice/mysql.service
    
    # mysqladmin -p shutdown
    Enter password: 
    #
    
  • Restarted mariadb on commandline with the --innodb-force-recovery=1 switch (details in this MariaDB KB article)
  • Connected to the mariadb instance on the commandline and recreated the table by copying its content to a new table:
    # mysql -p -D nextcloud
    Enter password: 
    MariaDB [nextcloud]> create table oc_filecache_repaired like oc_filecache;
    Query OK, 0 rows affected (0.418 sec)
    
    MariaDB [nextcloud]> insert oc_filecache_repaired select * from oc_filecache;
    Query OK, 1548596 rows affected (23 min 46.640 sec)
    Records: 1548596  Duplicates: 0  Warnings: 0
    
    MariaDB [nextcloud]> drop table oc_filecache;
    Query OK, 0 rows affected (0.755 sec)
    
    MariaDB [nextcloud]> rename table oc_filecache_repaired to oc_filecache;
    Query OK, 0 rows affected (0.446 sec)
    
    MariaDB [nextcloud]> shutdown;
    Query OK, 0 rows affected (0.000 sec)
    
    MariaDB [nextcloud]> quit
    Bye
    
  • Startup via systemd systemctl start mysql
  • Disable maintenance mode.
  • Check if everything works in Nextcloud.

Maybe this helps someone, who is in the same situation.

Cheers!

3 Likes

Dear @Evil2000, thanks for your post. Could you explain how to completely write the command to restart MariaDB? I’ve tried many things and tried to understand the linked page, but I failed. And I do not know if this is due to a wrong command or due to another issue…

As this is three years ago, I really cannot remember the exact command. Today, I would simply try mysqld --innodb-force-recovery=1
What commands did you try and which output did you get?

Strange, but I haven’t tried this one… But the table seems to be to much damaged…

[...]
2024-08-08 17:01:27 0 [Note] InnoDB: 0000000000000000000000000000000000000000000000000000000000000000
2024-08-08 17:01:27 0 [Note] InnoDB: 0000000000000000000000000000000000000000007000633eb05a3cb3f99a71
2024-08-08 17:01:27 0 [Note] InnoDB: End of page dump
2024-08-08 17:01:27 0 [Note] InnoDB:  You can use CHECK TABLE to scan your table for corruption. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
2024-08-08 17:01:27 0 [ERROR] InnoDB: Failed to read page 3023 from file './nextcloud/oc_authtoken.ibd': Page read from tablespace is corrupted.
2024-08-08 17:01:28 0 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './nextcloud/oc_dav_shares.ibd' page [page id: space=75, page number=4]. You may have to recover from a backup.
2024-08-08 17:01:28 0 [Note] InnoDB: Page dump (16384 bytes):
2024-08-08 17:01:28 0 [Note] InnoDB: 0000000000000004ffffffffffffffff000000003bfd57dc45bf000000000000
2024-08-08 17:01:28 0 [Note] InnoDB: 00000000004b00020260800a016c003702220005000000070000000000382e02
2024-08-08 17:01:28 0 [Note] InnoDB: 000000000000000000e90000004b0000000202720000004b0000000201b20100
2024-08-08 17:01:28 0 [Note] InnoDB: 020178696e66696d756d0008000b000073757072656d756d1008160000001001
2024-08-08 17:01:28 0 [Note] InnoDB: a17072696e636970616c732f75736572732f6a616b7562000000000000000563
2024-08-08 17:01:28 0 [Note] InnoDB: 616c656e6461727779655039726f34357479514432506a000000000000000908
2024-08-08 17:01:28 0 [Note] InnoDB: 170400001800dc7072696e636970616c732f75736572732f736162696e650000
2024-08-08 17:01:28 0 [Note] InnoDB: 00000000000863616c656e646172000000000000000608170400002000377072
2024-08-08 17:01:28 0 [Note] InnoDB: 696e636970616c732f75736572732f736162696e65000000000000000563616c
2024-08-08 17:01:28 0 [Note] InnoDB: 656e6461720000000000000003081704000028ff927072696e636970616c732f
2024-08-08 17:01:28 0 [Note] InnoDB: 75736572732f736162696e65000000000000000763616c656e64617200000000
2024-08-08 17:01:28 0 [Note] InnoDB: 0000000508170420003000000000000000000000000000000000000000000000
2024-08-08 17:01:28 0 [Note] InnoDB: 0000000000000000000000000000000000000000000000000000000817040000
2024-08-08 17:01:28 0 [Note] InnoDB: 38fecd7072696e636970616c732f75736572732f736162696e65000000000000
2024-08-08 17:01:28 0 [Note] InnoDB: 000963616c656e646172000000000000000810081600000040fea67072696e63
2024-08-08 17:01:28 0 [Note] InnoDB: 6970616c732f75736572732f6a616b7562000000000000000463616c656e6461
2024-08-08 17:01:28 0 [Note] InnoDB: 726f7778524e4834665843725344427372000000000000000a10081600000048
2024-08-08 17:01:28 0 [Note] InnoDB: fedc7072696e636970616c732f75736572732f6a616b75620000000000000009
2024-08-08 17:01:28 0 [Note] InnoDB: 63616c656e646172476946466d4e37714a6d4d5350377379000000000000000b
2024-08-08 17:01:28 0 [Note] InnoDB: 0000000000000000000000000000000000000000000000000000000000000000
[...]
2024-08-08 17:01:28 0 [Note] InnoDB: 0000000000000000000000000000000000000000007000633bfd57dc87030c97
2024-08-08 17:01:28 0 [Note] InnoDB: End of page dump
2024-08-08 17:01:28 0 [Note] InnoDB:  You can use CHECK TABLE to scan your table for corruption. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
2024-08-08 17:01:28 0 [ERROR] InnoDB: Failed to read page 4 from file './nextcloud/oc_dav_shares.ibd': Page read from tablespace is corrupted.
2024-08-08 17:01:28 0 [Note] InnoDB: Buffer pool(s) load completed at 240808 17:01:28
240808 17:02:15 [ERROR] mysqld got signal 4 ;
Sorry, we probably made a mistake, and this is a bug.

Your assistance in bug reporting will enable us to fix this for the next release.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

Server version: 10.5.23-MariaDB-0+deb11u1 source revision: 6cfd2ba397b0ca689d8ff1bdb9fc4a4dc516a5eb
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=1
max_threads=153
thread_count=1
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467898 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f4c000c68
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fa86c6868 thread_stack 0x49000
addr2line: 'mysqld': No such file
Printing to addr2line failed
mysqld(my_print_stacktrace+0x30)[0x55703a5a70]
mysqld(handle_fatal_signal+0x458)[0x556fee19dc]
addr2line: 'linux-vdso.so.1': No such file
linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0x7fad0d67a8]
addr2line: 'mysqld': No such file
mysqld(+0x758a60)[0x556fd48a60]
mysqld(_ZN4JOIN15optimize_stage2Ev+0x468)[0x556fd63b38]
mysqld(_ZN4JOIN14optimize_innerEv+0x1308)[0x556fd66ccc]
mysqld(_ZN4JOIN8optimizeEv+0xa4)[0x556fd68518]
mysqld(_Z12mysql_selectP3THDP10TABLE_LISTR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xc0)[0x556fd685f4]
mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x118)[0x556fd68d48]
mysqld(+0x7187ec)[0x556fd087ec]
mysqld(_Z21mysql_execute_commandP3THD+0xc80)[0x556fd111c0]
mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x1f4)[0x556fd14c04]
mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0xb14)[0x556fd16514]
mysqld(_Z10do_commandP3THD+0xec)[0x556fd1808c]
mysqld(_Z24do_handle_one_connectionP7CONNECTb+0x328)[0x556fdef448]
mysqld(handle_one_connection+0x60)[0x556fdef6a4]
mysqld(+0xb02868)[0x55700f2868]
nptl/pthread_create.c:477(start_thread)[0x7facaff648]
aarch64/clone.S:81(thread_start)[0x7fac7bd01c]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f4c012a40): SELECT `appid`, `configkey`, `configvalue`, `type` FROM `oc_appconfig` WHERE `lazy` = 0

Connection ID (thread ID): 3
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off

The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             2598                 2598                 processes 
Max open files            32186                32186                files     
Max locked memory         119223296            119223296            bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       2598                 2598                 signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: core

Kernel version: Linux version 6.1.21-v8+ (dom@buildbot) (aarch64-linux-gnu-gcc-8 (Ubuntu/Linaro 8.4.0-3ubuntu1) 8.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1642 SMP PREEMPT Mon Apr  3 17:24:16 BST 2023

Ungültiger Maschinenbefehl

Thanks for the help, I am prepairing to completely setup my RPi from scratch…