Sudden Internal Server Error

Support intro

Sorry to hear you’re facing problems :slightly_frowning_face:

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can :heart:

Nextcloud version (eg, 18.0.2): 19.0.4
Operating system and version (eg, Ubuntu 20.04): FreeBSD 11.4 jail under TrueNAS Core 12.0-RELEASE
Apache or nginx version (eg, Apache 2.4.25): Apache 2.4
PHP version (eg, 7.1): 7.3.12

The issue you are facing:
Internal Server Error. Login page shows ok, but the error appears after logging in

. It worked fine yesterday, and I have not upgraded or changed config in any way since then.

Is this the first time you’ve seen this error? (Y/N): Yes

Steps to replicate it:

  1. Go to my nextcloud webpage
  2. Enter login credentials
  3. Click login

The output of your Nextcloud log in Admin > Logging:

I cannot login to the Nextcloud interface.
In httpd-error.log, after starting Apache (starts ok), I have this.

`[Thu Nov 19 11:02:42.365519 2020] [mpm_prefork:notice] [pid 4019] AH00169: caught SIGTERM, shutting down
[Thu Nov 19 11:02:54.467786 2020] [mpm_prefork:notice] [pid 24464] AH00163: Apache/2.4.37 (FreeBSD) OpenSSL/1.0.2u-freebsd configured -- resuming normal operations
[Thu Nov 19 11:02:54.468043 2020] [core:notice] [pid 24464] AH00094: Command line: '/usr/local/sbin/httpd -D NOHTTPACCEPT'
[Thu Nov 19 11:07:43.189477 2020] [mpm_prefork:notice] [pid 24464] AH00169: caught SIGTERM, shutting down
[Thu Nov 19 11:10:47.074122 2020] [mpm_prefork:notice] [pid 3380] AH00163: Apache/2.4.37 (FreeBSD) OpenSSL/1.0.2u-freebsd configured -- resuming normal operations
[Thu Nov 19 11:10:47.092292 2020] [core:notice] [pid 3380] AH00094: Command line: '/usr/local/sbin/httpd -D NOHTTPACCEPT'
[Thu Nov 19 12:01:06.731138 2020] [mpm_prefork:notice] [pid 3380] AH00169: caught SIGTERM, shutting down
[Thu Nov 19 12:01:18.484718 2020] [mpm_prefork:notice] [pid 12001] AH00163: Apache/2.4.37 (FreeBSD) OpenSSL/1.0.2u-freebsd configured -- resuming normal operations
[Thu Nov 19 12:01:18.485036 2020] [core:notice] [pid 12001] AH00094: Command line: '/usr/local/sbin/httpd -D NOHTTPACCEPT'
`

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

        <?php
$CONFIG = array (
  'apps_paths' =>
  array (
    0 =>
    array (
      'path' => '/usr/local/www/nextcloud/apps',
      'url' => '/apps',
      'writable' => true,
    ),
    1 =>
    array (
      'path' => '/usr/local/www/nextcloud/apps-pkg',
      'url' => '/apps-pkg',
      'writable' => false,
    ),
  ),
  'logfile' => '/var/log/nextcloud/nextcloud.log',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'instanceid' => 'DELETED by MOD',
  'passwordsalt' => 'DELETED by MOD',
  'secret' => 'DELETED by MOD [JK]',
  'trusted_domains' =>
  array (
    0 => '192.168.1.84',
    1 => 'sub.domain.tld',
  ),
  'trusted_proxies' =>
  array (
    0 => '192.168.1.84',
  ),
  'datadirectory' => '/mnt/data',
  'dbtype' => 'mysql',
  'version' => '19.0.4.2',
  'overwrite.cli.url' => 'http://192.168.1.84',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost:/tmp/mysql.sock',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => '********',
  'dbpassword' => '********',
  'installed' => true,
  'redis' =>
  array (
    'host' => '/tmp/redis.sock',
    'port' => 0,
  ),
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'mail_smtpmode' => 'smtp',
  'mail_smtpsecure' => 'tls',
  'mail_sendmailmode' => 'smtp',
  'mail_from_address' => 'user',
  'mail_domain' => 'domain.com',
  'mail_smtphost' => 'smtp.domain.com',
  'mail_smtpport' => '587',
  'maintenance' => false,
  'theme' => '',
  'loglevel' => 2,
  'updater.release.channel' => 'stable',
  'mail_smtpauth' => 1,
  'mail_smtpname' => 'user@domain.com',
  'mail_smtppassword' => '********',
);

The output of your Apache/nginx/system log in /var/log/____:

My /var/log/apache directory is empty.

I found this is the nextcloud log:

{"reqId":"sEfJvuJJkAdNa6WTgTjs","level":3,"time":"2020-11-20T01:30:01+00:00","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":{"Exception":"Error","Message":"PDOStatement::execute(): MySQL server has gone away at /usr/local/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php#117","Code":0,"Trace":[{"function$
{"reqId":"sEfJvuJJkAdNa6WTgTjs","level":3,"time":"2020-11-20T01:30:01+00:00","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":{"Exception":"Error","Message":"PDOStatement::execute(): Error reading result set's header at /usr/local/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php#117","Code":0,"Trace":[{"f$
{"reqId":"sEfJvuJJkAdNa6WTgTjs","level":3,"time":"2020-11-20T01:30:01+00:00","remoteAddr":"","user":"--","app":"cron","method":"","url":"--","message":{"Exception":"Doctrine\\DBAL\\Exception\\DriverException","Message":"An exception occurred while executing 'UPDATE `oc_appconfig` SET `configvalue` = ? WHERE (`appid` = ?) AND (`configkey` = ?) AND (`configval$

So it seems there is a database error. I´ve checked that I can log into my mysql server as my nextcloud mysql user and view the database content. Also checked that the mysql server starts correctly, and that mysql.sock shows up under /tmp.

I would really like some help with this. Anyone with any experience of this?

ummm. I’m afraid I can’t help you here… it looks as if you httpdeamon won’t accept incoming requests for some reason (referring to your apache.log)
Maybe it’s because some problem with your MySQL-server.

i’m not sure if this would work.

maybe you’d need to add a port here as well (usually it’s 3302 I think - pls re-check that)

very uncommon to rewrite to an internal adress, usually you’d do the rewrite to your URL (secure)

but I saw that you accidently posted some important data about your instance which better should be kept secret. I adjusted that for you. Though you better should regard your instance as being compromised. Even if it’s not up, yet.

It has been working for a long time, and I haven´t changed any config, so I believe all of the above should work. Thanks for the help with what I missed to remove from what I posted. Will change all passwords if I ever get this upp and running again.

Just noticed that the mysql-server isn´starting any more.
Found this in the mysql log:

Server version: 10.2.19-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=3
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467175 K  bytes of 
memory
2020-11-20 13:59:22 34422743040 [Note] InnoDB: 5.7.24 started; log sequence number 
5931001780
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x81bc162c8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fffdd9e6f08 thread_stack 0x49000
2020-11-20 13:59:22 34739088128 [Note] InnoDB: Loading buffer pool(s) from 
/var/db/mysql/ib_buffer_pool
0xc6277e <my_print_stacktrace+0x2e> at /usr/local/libexec/mysqld
0x7925c4 <handle_fatal_signal+0x294> at /usr/local/libexec/mysqld
0x803184c80 <pthread_sigmask+0x530> at /lib/libthr.so.3

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): is an invalid pointer
Connection ID (thread ID): 1
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid$

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

We think the query pointer is invalid, but we will try to print it anyway.
Query:

2020-11-20 14:13:46 34422743040 [Note] InnoDB: Mutexes and rw_locks use GCC atomic 
builtins
2020-11-20 14:13:46 34422743040 [Note] InnoDB: Uses event mutexes
2020-11-20 14:13:46 34422743040 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-11-20 14:13:46 34422743040 [Note] InnoDB: Number of pools: 1
2020-11-20 14:13:46 34422743040 [Note] InnoDB: Using SSE2 crc32 instructions
2020-11-20 14:13:46 34422743040 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2020-11-20 14:13:46 34422743040 [Note] InnoDB: Completed initialization of buffer pool
2020-11-20 14:13:47 34422743040 [Note] InnoDB: Highest supported file format is Barracuda.
2020-11-20 14:13:47 34422743040 [Note] InnoDB: Starting crash recovery from checkpoint LSN=5930993524
2020-11-20 14:13:50 34422743040 [Note] InnoDB: Starting final batch to recover 10 pages from redo log.
2020-11-20 14:13:52 34422743040 [Note] InnoDB: 128 out of 128 rollback segments are active.
2020-11-20 14:13:52 34422743040 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2020-11-20 14:13:52 34422743040 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-11-20 14:13:52 34422743040 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-11-20 14:13:52 34422743040 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-11-20 14:13:52 0x8169c5700  InnoDB: Assertion failure in file /wrkdirs/usr/ports/databases/mariadb102-server/work/mariadb-10.2.19/storage/innobase/include/fut0lst.ic line 85
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/
InnoDB: about forcing recovery.
201120 14:13:52 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Can anyone make sense of that, and point me in any direction?

have you looked here already?

Thanks. I had a read, but that gets me no further, I´m afraid.
I do have daily snapshots of the jail. Any point / risk in rolling back a week? It would not affect user files stored on the server.

ummm. I don’t really know. Usually it shouldn’t effect your new files… but it depends upon the way of how the snapshot was taken. make it was a good idea to copy the files to a safe place just to be sure.

Ok,

Another solution would be if I could reinstall / reconfigure mysql in a way that doesn´t mean I have to install the whole jail with web servers, certificates and Nextcloud all over. Is that possible? Can the database be backed up and recreated somehow? Or could Nextcloud populate the data it needs into a fresh database?
Any thoughts on that? What would be lost in the process?
I would really like not having to do things from scratch…

Solved it! I rolled all my Nextcloud snapshots back to the same time. After that it works again.

1 Like