Debian Updates - Database Corruption?

Support intro

Sorry to hear you’re facing problems :slightly_frowning_face:

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can :heart:

Nextcloud version (eg, 12.0.2): 12.0.7.1
Operating system and version (eg, Ubuntu 17.04): Debian 9.4
Apache or nginx version (eg, Apache 2.4.25): Apache 2.4.25
PHP version (eg, 7.1): 7.0.27

The issue you are facing:
After weekly apt-get update and upgrade, noticed NextCloud no longer accessible “Internal Server Error”, apache2 is running. When I view nextcloud.log, I get the generic:
General error: 11 database disk image is malformed

Doing some Googling, figured I’d do a restore by stopping Apache, then running:
sqlite3 owncloud.db < nextcloud-sqlbkp_Tue.bak
Then move file into data directory, restart, same error

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:

  1. Start apache

The output of your Nextcloud log in Admin > Logging:

{"reqId":"QxaO5LyO4Hs8SRxoDvnc","level":4,"time":"2018-06-07T23:06:13+00:00","remoteAddr":"my_ip","user":"--","app":"webdav","method":"PROPFIND","url":"\/nextcloud\/remote.php\/dav\/files\/southhendo\/","message":"Exception: {\"Exception\":\"Sabre\\\\DAV\\\\Exception\\\\ServiceUnavailable\",\"Message\":\"Doctrine\\\\DBAL\\\\Exception\\\\DriverException: An exception occurred while executing 'SELECT \\\"id\\\", \\\"uid\\\", \\\"login_name\\\", \\\"password\\\", \\\"name\\\", \\\"type\\\", \\\"remember\\\", \\\"token\\\", \\\"last_activity\\\", \\\"last_check\\\", \\\"scope\\\" FROM \\\"oc_authtoken\\\" WHERE \\\"token\\\" = ?' with params [\\\"hidden"]:\\n\\nSQLSTATE[HY000]: General error: 11 database disk image is malformed\",\"Code\":0,\"Trace\":\"#0 \\\/var\\\/www\\\/html\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Auth\\\/Plugin.php(201): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\Auth->check(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#1 \\\/var\\\/www\\\/html\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Auth\\\/Plugin.php(150): Sabre\\\\DAV\\\\Auth\\\\Plugin->check(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#2 [internal function]: Sabre\\\\DAV\\\\Auth\\\\Plugin->beforeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#3 \\\/var\\\/www\\\/html\\\/nextcloud\\\/3rdparty\\\/sabre\\\/event\\\/lib\\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\n#4 \\\/var\\\/www\\\/html\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(466): Sabre\\\\Event\\\\EventEmitter->emit('beforeMethod', Array)\\n#5 \\\/var\\\/www\\\/html\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(254): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#6 \\\/var\\\/www\\\/html\\\/nextcloud\\\/apps\\\/dav\\\/lib\\\/Server.php(258): Sabre\\\\DAV\\\\Server->exec()\\n#7 \\\/var\\\/www\\\/html\\\/nextcloud\\\/apps\\\/dav\\\/appinfo\\\/v2\\\/remote.php(33): OCA\\\\DAV\\\\Server->exec()\\n#8 \\\/var\\\/www\\\/html\\\/nextcloud\\\/remote.php(164): require_once('\\\/var\\\/www\\\/html\\\/n...')\\n#9 {main}\",\"File\":\"\\\/var\\\/www\\\/html\\\/nextcloud\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/Auth.php\",\"Line\":162}","userAgent":"Mozilla\/5.0 (Macintosh) mirall\/2.3.3 (build 84) (Nextcloud)","version":"12.0.7.1"}

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => 'instanceid',
  'passwordsalt' => 'password',
  'secret' => 'secret',
  'trusted_domains' =>
  array (
    0 => 'my_ip',
    1 => 'my.web.addr,
  ),
  'datadirectory' => '/data/nextcloud/data',
  'overwrite.cli.url' => 'http://my_ip/nextcloud',
  'dbtype' => 'sqlite3',
  'version' => '12.0.7.1',
  'logtimezone' => 'UTC',
  'installed' => true,
  'mail_from_address' => 'myemail',
  'mail_smtpmode' => 'smtp',
  'mail_domain' => 'gmail.com',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpauth' => 1,
  'mail_smtpsecure' => 'tls',
  'mail_smtphost' => 'smtp.gmail.com',
  'mail_smtpport' => '587',
  'mail_smtpname' => 'myemail@gmail.com',
  'mail_smtppassword' => 'mypassword',
  'maintenance' => false,
  'theme' => '',
  'loglevel' => 2,
  'memcache.local' => '\\OC\\Memcache\\APCu',
);

The output of your Apache/nginx/system log in /var/log/____:

Last few lines...
[Thu Jun 07 15:44:52.703574 2018] [:error] [pid 2336] [client 71.197.180.252:56625] PHP Fatal error:  Uncaught Doctrine\\DBAL\\DBALException: Failed to connect to the database: An exception occurred while executing 'PRAGMA journal_mode = WAL':\n\nSQLSTATE[HY000]: General error: 8 attempt to write a readonly database in /var/www/html/nextcloud/lib/private/DB/Connection.php:61\nStack trace:\n#0 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(992): OC\\DB\\Connection->connect()\n#1 /var/www/html/nextcloud/lib/private/DB/Connection.php(213): Doctrine\\DBAL\\Connection->executeUpdate('PRAGMA read_unc...', Array, Array)\n#2 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(623): OC\\DB\\Connection->executeUpdate('PRAGMA read_unc...')\n#3 /var/www/html/nextcloud/lib/private/DB/Connection.php(148): Doctrine\\DBAL\\Connection->setTransactionIsolation(2)\n#4 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DriverManager.php(172): OC\\DB\\Connection->__construct(Array, Object(Doctrine\\DBAL\\Driver\\PDOSqlite\\Driver), Object(Doctrine\\DBAL\\Configuration), Object(Doct in /var/www/html/nextcloud/lib/private/DB/Connection.php on line 61
[Thu Jun 07 15:46:29.587516 2018] [mpm_prefork:notice] [pid 2321] AH00169: caught SIGTERM, shutting down
[Thu Jun 07 15:46:46.456224 2018] [ssl:warn] [pid 686] AH01906: 127.0.1.1:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Thu Jun 07 15:46:46.457383 2018] [ssl:warn] [pid 686] AH01909: 127.0.1.1:443:0 server certificate does NOT include an ID which matches the server name
[Thu Jun 07 15:46:48.093116 2018] [ssl:warn] [pid 1113] AH01906: 127.0.1.1:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Thu Jun 07 15:46:48.093296 2018] [ssl:warn] [pid 1113] AH01909: 127.0.1.1:443:0 server certificate does NOT include an ID which matches the server name
[Thu Jun 07 15:46:48.117599 2018] [mpm_prefork:notice] [pid 1113] AH00163: Apache/2.4.25 (Raspbian) OpenSSL/1.0.2l configured -- resuming normal operations
[Thu Jun 07 15:46:48.117792 2018] [core:notice] [pid 1113] AH00094: Command line: '/usr/sbin/apache2'

Problem solved. I had a bi-directional rsync from this server to it’s partner-pair, and once I disabled the rsync back from the other server, stopped apache, restored the database from a backup, and started apache, problem solved.