Suddenly, Server Internal Error (500)

Hi guys, my Nextcloud instance was running fine for two years, but suddenly I receive this Server Internal Error with a black screen. I do not know, what causes it, but my web server and mysql seem to be fine, because other web sites are still running/showing (on the same machine).

Unfortunately, logs do not help at all:

sudo cat /var/www/nextcloud/data/nextcloud.log →
nothing (empty)

sudo cat /var/log/apache2/error.log | grep nextcloud →
nothing

sudo cat /var/log/apache2/access.log | grep nextcloud →
X.X.X.X - - [26/May/2023:12:17:57 +0200] “GET /favicon.ico HTTP/1.1” 500 477 “http://nextcloud.X.com/” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36 Edg/113.0.1774.50”

I really cannot narrow it down to anything. :\ Maybe someone knows the best way to troubleshoot/solve this? Thank you in advance.

Once again with the support template:

system information

There is no “settings” directory in /var/www/nextcloud.

Nextcloud version: replace me
Operating system and version: Debian Bullseye
Apache or nginx version: Apache/2.4.56
PHP version (eg, 8.1): PHP 7.4.33
Database (sqlite or MariaDB or Postgres) mysql Ver 14.14 Distrib 5.7.42
Docker (compose) no, native
Snap
Is this the first time you’ve seen this error? yes

The issue you are facing:

  • Steps to replicate it:
  1. Open up the web site → Server Internal Error
  • The output of your Nextcloud log in Admin > Logging:
not accessible
  • The output of your config.php file in /path/to/nextcloud**
    (use https://.../settings/admin/support which auto-removes identifying information!):
No "settings directory" in /var/www/nextcloud, so manually from /var/www/nextcloud/config/config.php:

sudo cat /var/www/nextcloud/config/config.php
<?php
$CONFIG = array (
  'instanceid' => 'X',
  'passwordsalt' => 'XXX',
  'secret' => 'X',
  'trusted_domains' => 
  array (
    0 => 'nextcloud.X.COM',
  ),
  'datadirectory' => '/var/www/nextcloud/data',
  'dbtype' => 'mysql',
  'version' => '21.0.1.1',
  'overwrite.cli.url' => 'http://nextcloud.X.COM',
  'dbname' => 'DATABASE',
  'dbhost' => 'ADDRESS:PORT',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'USER',
  'dbpassword' => 'PASSWORD',
  'installed' => true,
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'maintenance' => false,
  'theme' => '',
  'loglevel' => 2,
  'mail_smtpmode' => 'smtp',
  'mail_smtpsecure' => 'ssl',
  'mail_sendmailmode' => 'smtp',
  'mail_from_address' => 'nextcloud',
  'mail_domain' => 'X.COM',
  'mail_smtphost' => 'X.X.X',
  'mail_smtpport' => '465',
  'mail_smtpauth' => 1,
  'mail_smtpname' => 'nextcloud@X.COM',
  'mail_smtppassword' => 'X.X.X',
  'mail_smtpauthtype' => 'LOGIN',
  'updater.secret' => 'XXX',
);
  • for Docker/Podman list environment variables ( run “docker inspect {container name}”)**
no Docker, native
  • The output of your Apache/nginx/system log in /var/log/____:
  • For Docker/Podman post STDOUT of the container (run docker logs {container name})
Access.log:
X.X.X.X - - [27/May/2023:13:05:17 +0200] "GET /favicon.ico HTTP/1.1" 500 477 "http://nextcloud.X.COM/index.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36 Edg/113.0.1774.57"

Error.log:
...
[Sat May 27 03:12:52.128188 2023] [autoindex:error] [pid 90087] [client 139.59.77.147:43004] AH01276: Cannot serve directory /var/www/: No matching DirectoryIndex (index.html,index.cgi,index.pl,index.php,index.xhtml,index.htm) found, and server-generated directory index forbidden by Options directive
[Sat May 27 03:42:17.944613 2023] [autoindex:error] [pid 90084] [client 103.149.192.93:54799] AH01276: Cannot serve directory /var/www/: No matching DirectoryIndex (index.html,index.cgi,index.pl,index.php,index.xhtml,index.htm) found, and server-generated directory index forbidden by Options directive
[Sat May 27 03:55:07.852519 2023] [autoindex:error] [pid 90192] [client 35.195.93.98:51326] AH01276: Cannot serve directory /var/www/: No matching DirectoryIndex (index.html,index.cgi,index.pl,index.php,index.xhtml,index.htm) found, and server-generated directory index forbidden by Options directive
[Sat May 27 04:29:19.816271 2023] [autoindex:error] [pid 90085] [client 184.105.139.67:25560] AH01276: Cannot serve directory /var/www/: No matching DirectoryIndex (index.html,index.cgi,index.pl,index.php,index.xhtml,index.htm) found, and server-generated directory index forbidden by Options directive
[Sat May 27 05:57:12.272568 2023] [autoindex:error] [pid 90084] [client 179.43.177.244:54738] AH01276: Cannot serve directory /var/www/: No matching DirectoryIndex (index.html,index.cgi,index.pl,index.php,index.xhtml,index.htm) found, and server-generated directory index forbidden by Options directive
[Sat May 27 07:07:53.052920 2023] [php7:error] [pid 90083] [client 114.119.151.206:26991] PHP Fatal error:  Uncaught Error: Call to undefined function mysql_num_rows() in /var/www/hlstatsx/pages/dailyawardinfo.php:145\nStack trace:\n#0 /var/www/hlstatsx/hlstats.php(224): include()\n#1 {main}\n  thrown in /var/www/hlstatsx/pages/dailyawardinfo.php on line 145, referer: http://stats.X.COM/hlstats.php?mode=awards&game=csgo
[Sat May 27 07:13:30.963099 2023] [php7:error] [pid 90192] [client 114.119.132.10:34183] PHP Fatal error:  Uncaught Error: Call to undefined function mysql_num_rows() in /var/www/hlstatsx/pages/dailyawardinfo.php:145\nStack trace:\n#0 /var/www/hlstatsx/hlstats.php(224): include()\n#1 {main}\n  thrown in /var/www/hlstatsx/pages/dailyawardinfo.php on line 145, referer: http://stats.X.COM/hlstats.php?mode=awards&game=csgo
[Sat May 27 07:21:30.194711 2023] [authz_core:error] [pid 90086] [client 104.140.148.122:59284] AH01630: client denied by server configuration: /var/www/server-status
[Sat May 27 07:23:28.299336 2023] [php7:error] [pid 90087] [client 114.119.148.94:50185] PHP Fatal error:  Uncaught Error: Call to undefined function mysql_num_rows() in /var/www/hlstatsx/pages/dailyawardinfo.php:145\nStack trace:\n#0 /var/www/hlstatsx/hlstats.php(224): include()\n#1 {main}\n  thrown in /var/www/hlstatsx/pages/dailyawardinfo.php on line 145, referer: http://stats.X.COM/hlstats.php?mode=awards&game=csgo
[Sat May 27 07:30:39.448492 2023] [autoindex:error] [pid 90084] [client 103.110.32.156:54234] AH01276: Cannot serve directory /var/www/: No matching DirectoryIndex (index.html,index.cgi,index.pl,index.php,index.xhtml,index.htm) found, and server-generated directory index forbidden by Options directive
[Sat May 27 07:40:41.934189 2023] [autoindex:error] [pid 90086] [client 128.14.133.58:44318] AH01276: Cannot serve directory /var/www/: No matching DirectoryIndex (index.html,index.cgi,index.pl,index.php,index.xhtml,index.htm) found, and server-generated directory index forbidden by Options directive
[Sat May 27 08:04:52.833035 2023] [php7:error] [pid 90087] [client 54.153.52.134:46680] script '/var/www/wp-config.php' not found or unable to stat
[Sat May 27 08:07:39.223088 2023] [php7:error] [pid 90083] [client 54.206.161.18:54784] script '/var/www/wp-config.php' not found or unable to stat
[Sat May 27 09:24:07.680581 2023] [autoindex:error] [pid 90083] [client 107.150.121.179:54476] AH01276: Cannot serve directory /var/www/: No matching DirectoryIndex (index.html,index.cgi,index.pl,index.php,index.xhtml,index.htm) found, and server-generated directory index forbidden by Options directive
[Sat May 27 09:31:08.610098 2023] [autoindex:error] [pid 90087] [client 178.79.191.180:2407] AH01276: Cannot serve directory /var/www/: No matching DirectoryIndex (index.html,index.cgi,index.pl,index.php,index.xhtml,index.htm) found, and server-generated directory index forbidden by Options directive
[Sat May 27 11:06:29.917039 2023] [autoindex:error] [pid 90083] [client 31.56.77.43:35561] AH01276: Cannot serve directory /var/www/: No matching DirectoryIndex (index.html,index.cgi,index.pl,index.php,index.xhtml,index.htm) found, and server-generated directory index forbidden by Options directive

Nothing related to Nextcloud as far as I can see, just random bots inquiries on non-existing web sites. All existing web sites on the same machine work fine.
  • errors in nextcloud.log in /var/www/ or as admin user in top right menu, filtering for errors.
Nothing in /var/www/nextcloud/data/nextcloud.log.

hi @vega welcome to the forum :handshake:

you are missing the required support template. Please fill this form out and edit into your post.

This will give us the technical info and logs needed to help you in case the machine in question in spitting out errors! Since you’ve provided zero technical details it is impossible to help you as-is. Thanks.

General troubleshooting hints are described in the docs
https://docs.nextcloud.com/server/stable/admin_manual/issues/index.html

are you looking for the right file? does it exist and is this the file configured in the application?

maybe you grep for the wrong term?

I updated the initial post with the support template information.

I also did a check on the mysql database, but it does not look to be corrupted.

did you run an upgrade before the error occurred? Nextcloud 26 doesn’t support PHP7.4 anymore (which is end of life since 7.8 months now), in this case you must upgrade to more modern PHP version.

Thank you, I updated PHP to 8.2 and restarted the Apache2, but still no change on the outcome. Other web sites are still working though.

I’m sorry there is no value in trying some wild guesses. Depending on whether you have NC26 or not PHP 8.2 update might even be wrong (no support for PHP8.2 until NC26). as you didn’t provide you NC version nobody can tell you which PHP is required.

I recommend you should systematically check the system one piece after another until you find the root cause.

Oh, wow, it seems I still have an old version: Running Nextcloud 18.0.4.2 says the Nextcloud scanner (I did not know, how to check the version without being able to run it).

Can I update the broken Nextcloud in any way?

security scanner doesn’t show current version until you re-trigger the scan. you better analyze config.php and version.php

if this is true PHP 8.2 is not supported. maybe no even 8.1…