Mysql [ERROR] InnoDB: Space id and page no stored in the page

Support intro

Sorry to hear you’re facing problems :slightly_frowning_face:

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

2021-01-20  9:45:20 0 [ERROR] InnoDB: Space id and page no stored in the page, read in are [page id: space=64, page number=1722], should be [page id: space=64, page number=1050]
2021-01-20 09:45:20 0x80275d700  InnoDB: Assertion failure in file /wrkdirs/usr/ports/databases/mariadb104-server/work/mariadb-10.4.17/storage/innobase/rem/rem0rec.cc line 849
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
210120  9:45:20 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.4.17-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=3
max_threads=153
thread_count=9
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467678 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
0x128296c <my_print_stacktrace+0x3c> at /usr/local/libexec/mysqld
0xc11565 <handle_fatal_signal+0x295> at /usr/local/libexec/mysqld
0x8017bcb70 <_pthread_sigmask+0x530> at /lib/libthr.so.3
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
2021-01-20  9:45:20 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-01-20  9:45:20 0 [Note] InnoDB: Uses event mutexes
2021-01-20  9:45:20 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-01-20  9:45:20 0 [Note] InnoDB: Number of pools: 1
2021-01-20  9:45:20 0 [Note] InnoDB: Using SSE2 crc32 instructions
2021-01-20  9:45:20 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2021-01-20  9:45:20 0 [Note] InnoDB: Completed initialization of buffer pool
2021-01-20  9:45:20 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=6852470443
2021-01-20  9:45:21 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 4 row operations to undo
2021-01-20  9:45:21 0 [Note] InnoDB: Trx id counter is 19788321
2021-01-20  9:45:21 0 [Note] InnoDB: Starting final batch to recover 27 pages from redo log.
2021-01-20  9:45:21 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2021-01-20  9:45:21 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
2021-01-20  9:45:21 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2021-01-20  9:45:21 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-01-20  9:45:21 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-01-20  9:45:21 0 [ERROR] InnoDB: Space id and page no stored in the page, read in are [page id: space=64, page number=1722], should be [page id: space=64, page number=1050]
210120  9:45:21 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.4.17-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467678 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
0x128296c <my_print_stacktrace+0x3c> at /usr/local/libexec/mysqld
0xc11565 <handle_fatal_signal+0x295> at /usr/local/libexec/mysqld
0x8017bcb70 <_pthread_sigmask+0x530> at /lib/libthr.so.3
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can :heart:

Nextcloud version (eg, 20.0.5):
Operating system and version (TrueNAS 12.0-U1):
Apache or nginx version (eg, Apache 2.4):
PHP version (eg, 7.3):

The issue you are facing: mysql keeps crashing, giving the error supplied above. I tried rolling the db dataset back, and then it works for about 15 minutes. Then the error returns.

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:

1.Rollback db dataset to working snapshot
2. Wait 15 minutess
3. visit cloud.mydomain.com and see Internal Server Error.

By then the mysql-server service has stopped.

The output of your Nextcloud log in Admin > Logging:

Not relevant.

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'apps_paths' =>
  array (
    0 =>
    array (
      'path' => '/usr/local/www/nextcloud/apps',
      'url' => '/apps',
      'writable' => true,
    ),
    1 =>
    array (
      'path' => '/usr/local/www/nextcloud/apps-pkg',
      'url' => '/apps-pkg',
      'writable' => false,
    ),
  ),
  'logfile' => '/var/log/nextcloud/nextcloud.log',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'instanceid' => '<OMITTED>',
  'passwordsalt' => '<OMITTED>',
  'secret' => '<OMITTED>',
  'trusted_domains' =>
  array (
    0 => '<OMITTED>',
    1 => '<OMITTED>',
  ),
  'trusted_proxies' =>
  array (
    0 => '<OMITTED>',
  ),
  'datadirectory' => '/mnt/data',
  'dbtype' => 'mysql',
  'version' => '20.0.4.0',
  'overwrite.cli.url' => '<OMITTED>',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost:/tmp/mysql.sock',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud_admin',
  'dbpassword' => '<OMITTED>',
  'installed' => true,
  'redis' =>
  array (
    'host' => '/tmp/redis.sock',
    'port' => 0,
  ),
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'mail_smtpmode' => 'smtp',
  'mail_smtpsecure' => 'tls',
  'mail_sendmailmode' => 'smtp',
  'mail_from_address' => '<OMITTED>',
  'mail_domain' => '<OMITTED>',
  'mail_smtphost' => '<OMITTED>',
  'mail_smtpport' => '<OMITTED>',
  'maintenance' => false,
  'theme' => '',
  'loglevel' => 2,
  'updater.release.channel' => 'stable',
  'mail_smtpauth' => 1,
  'mail_smtpname' => '<OMITTED>',
  'mail_smtppassword' => '<OMITTED>',
  'app_install_overwrite' =>
  array (
    0 => 'gallery',
    1 => 'maps',
  ),
);

The output of your Apache/nginx/system log in /var/log/____:

Nothing in /var/log/messages from the same time.

Worth noting: Yesterday, I put my Nextcloud server behind a reverse proxy (nginx). It worked fine after the, and rolling back db, it works still. I don´t know if this could have any impact?

Suddenly it started doing more stuff. This seems like a good hint:

´´´
2021-01-20 11:26:41 389 [ERROR] InnoDB: Corruption of an index tree: table nextcloud.oc_activity index activity_filter, father ptr page no 386, child page no 1050

´´´

However, I do not know how to correct it. Has anyone seen this before?

more info:
I tried running the command mysqlcheck nextcloud -u root -p. I got OK on the first table, then I got this:

´´´
mysqlcheck: Got error: 2013: Lost connection to MySQL server during query when executing 'CHECK TABLE … ’
´´´

Looking in the log, it seems to fail on tha oc_activity table.

Running mysqlcheck nextcloud oc_activity -u root -p gives the same error. How can I correct this? Has anybody done this?

Ok, I solved it. Found that you can use mysqlcheck for repairs:

´´´
mysqlcheck -o nextcloud oc_activity -u root -p
´´´

Got message: Table does not support optimize, doing recreate + analyze instead.

After that, the table checked out, and the db is intact again.