Sudden fail on Nextcloud server - Internal Server Error

ℹ️ Support
1
Support intro

Sorry to hear you’re facing problems :slightly_frowning_face:

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can :heart:

Nextcloud version (eg, 18.0.2): 18.0.3.0
Operating system and version (eg, Ubuntu 20.04): 20.04.1 LTS
Apache or nginx version (eg, Apache 2.4.25): Apache/2.4.41 (Ubuntu)
PHP version (eg, 7.1): PHP 7.4.3 (cli)

The issue you are facing:

I am suddenly not able to connect to the server from Nextcloud clients and I get this message on the server’s webpage:

Internal Server Error

The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:

  1. Go to https://10.0.0.7/nextcloud/
  2. Error message appears, see above

The output of your Nextcloud log in Admin > Logging:

None. The page is not available.

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'log_type' => 'file',
  'logfile' => 'nextcloud.log',
  'loglevel' => 0,
  'logdateformat' => 'F d, Y H:i:s',
  'instanceid' => '******',
  'passwordsalt' => '******',
  'secret' => '*****',
  'trusted_domains' => 
  array (
    0 => 'localhost',
    1 => '10.0.0.7',
  ),
  'datadirectory' => '/mnt/md0/data/nextcloud/',
  'dbtype' => 'mysql',
  'version' => '18.0.3.0',
  'overwrite.cli.url' => 'https://10.0.0.7/nextcloud',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'nextcloud007',
  'dbpassword' => '************',
  'installed' => true,
  'maintenance' => false,
  'mysql.utf8mb4' => true,
);

Note: I added the logging now, that is;
  'log_type' => 'file',
  'logfile' => 'nextcloud.log',
  'loglevel' => 0,

The output of your Apache/nginx/system log in /var/log/____:

Apache log:

/var/log/apache2/error.log
[Fri Jan 15 00:00:48.215380 2021] [ssl:warn] [pid 986] AH01909: 10.0.0.7:443:0 server certificate does NOT include an ID which matches the server name
[Fri Jan 15 00:00:48.215550 2021] [mpm_prefork:notice] [pid 986] AH00163: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1f configured -- resuming normal operations
[Fri Jan 15 00:00:48.215558 2021] [core:notice] [pid 986] AH00094: Command line: '/usr/sbin/apache2'
[Fri Jan 15 16:52:29.655060 2021] [mpm_prefork:notice] [pid 986] AH00169: caught SIGTERM, shutting down
[Fri Jan 15 16:53:27.371673 2021] [ssl:warn] [pid 807] AH01909: 10.0.0.7:443:0 server certificate does NOT include an ID which matches the server name
[Fri Jan 15 16:53:28.307035 2021] [ssl:warn] [pid 989] AH01909: 10.0.0.7:443:0 server certificate does NOT include an ID which matches the server name
[Fri Jan 15 16:53:28.326817 2021] [mpm_prefork:notice] [pid 989] AH00163: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1f configured -- resuming normal operations
[Fri Jan 15 16:53:28.326892 2021] [core:notice] [pid 989] AH00094: Command line: '/usr/sbin/apache2'
[Fri Jan 15 17:23:29.988559 2021] [mpm_prefork:notice] [pid 989] AH00169: caught SIGTERM, shutting down
[Fri Jan 15 17:24:12.491764 2021] [ssl:warn] [pid 826] AH01909: 10.0.0.7:443:0 server certificate does NOT include an ID which matches the server name
[Fri Jan 15 17:24:13.370991 2021] [ssl:warn] [pid 988] AH01909: 10.0.0.7:443:0 server certificate does NOT include an ID which matches the server name
[Fri Jan 15 17:24:13.400512 2021] [mpm_prefork:notice] [pid 988] AH00163: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1f configured -- resuming normal operations
[Fri Jan 15 17:24:13.400561 2021] [core:notice] [pid 988] AH00094: Command line: '/usr/sbin/apache2'

/var/log/apache2/access.log
10.0.0.3 - - [15/Jan/2021:09:37:16 +0000] "GET /nextcloud/ocs/v2.php/core/navigation/apps?absolute=true&format=json HTTP/1.1" 200 2318 "-" "Mozilla/5.0 (Linux) mirall/2.6.2-1build1 (Nextcloud)"
10.0.0.3 - {username} [15/Jan/2021:09:37:16 +0000] "GET /nextcloud/status.php HTTP/1.1" 500 2300 "-" "Mozilla/5.0 (Linux) mirall/2.6.2-1build1 (Nextcloud)"
10.0.0.3 - {username} [15/Jan/2021:09:37:48 +0000] "GET /nextcloud/status.php HTTP/1.1" 500 2300 "-" "Mozilla/5.0 (Linux) mirall/2.6.2-1build1 (Nextcloud)"
...

System log (exerpt - can't see anything of relevance...):

Jan 15 17:25:19 ubuntu-server systemd[1]: Starting PackageKit Daemon...
Jan 15 17:25:19 ubuntu-server PackageKit: daemon start
Jan 15 17:25:19 ubuntu-server dbus-daemon[741]: [system] Successfully activated service 'org.freedesktop.PackageKit'
Jan 15 17:25:19 ubuntu-server systemd[1]: Started PackageKit Daemon.
Jan 15 17:26:35 ubuntu-server systemd[1]: proc-sys-fs-binfmt_misc.automount: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 2081 (find)
Jan 15 17:26:35 ubuntu-server systemd[1]: Mounting Arbitrary Executable File Formats File System...
Jan 15 17:26:35 ubuntu-server systemd[1]: Mounted Arbitrary Executable File Formats File System.
Jan 15 17:30:24 ubuntu-server PackageKit: daemon quit
Jan 15 17:30:24 ubuntu-server systemd[1]: packagekit.service: Succeeded.
Jan 15 17:39:01 ubuntu-server CRON[2797]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jan 15 17:39:13 ubuntu-server systemd[1]: Starting Clean php session files...
Jan 15 17:39:13 ubuntu-server systemd[1]: Starting Cleanup of Temporary Directories...
Jan 15 17:39:13 ubuntu-server systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Jan 15 17:39:13 ubuntu-server systemd[1]: Finished Cleanup of Temporary Directories.
Jan 15 17:39:13 ubuntu-server systemd[1]: phpsessionclean.service: Succeeded.
Jan 15 17:39:13 ubuntu-server systemd[1]: Finished Clean php session files.
Jan 15 17:50:02 ubuntu-server CRON[2896]: (daemon) CMD (test -x /usr/bin/debsecan && /usr/bin/debsecan --cron)
Jan 15 17:50:02 ubuntu-server systemd-resolved[720]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Jan 15 18:09:01 ubuntu-server CRON[3109]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jan 15 18:09:14 ubuntu-server systemd[1]: Starting Clean php session files...
Jan 15 18:09:14 ubuntu-server systemd[1]: phpsessionclean.service: Succeeded.
Jan 15 18:09:14 ubuntu-server systemd[1]: Finished Clean php session files.
Jan 15 18:17:01 ubuntu-server CRON[3194]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jan 15 18:20:44 ubuntu-server systemd[1]: Reloading.
Jan 15 18:20:45 ubuntu-server systemd[1]: /lib/systemd/system/dbus.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/dbus/system_bus_socket → /run/dbus/system_bus_socket; please update the unit file accordingly.
Jan 15 18:20:45 ubuntu-server systemd[1]: /lib/systemd/system/docker.socket:6: ListenStream= references a path below legacy directory /var/run/, updating /var/run/docker.sock → /run/docker.sock; please update the unit file accordingly.
Jan 15 18:20:45 ubuntu-server systemd[1]: Reloading.
Jan 15 18:20:45 ubuntu-server systemd[1]: /lib/systemd/system/dbus.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/dbus/system_bus_socket → /run/dbus/system_bus_socket; please update the unit file accordingly.
Jan 15 18:20:45 ubuntu-server systemd[1]: /lib/systemd/system/docker.socket:6: ListenStream= references a path below legacy directory /var/run/, updating /var/run/docker.sock → /run/docker.sock; please update the unit file accordingly.
Jan 15 18:20:45 ubuntu-server systemd[1]: Reloading.
Jan 15 18:20:46 ubuntu-server systemd[1]: /lib/systemd/system/dbus.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/dbus/system_bus_socket → /run/dbus/system_bus_socket; please update the unit file accordingly.
Jan 15 18:20:46 ubuntu-server systemd[1]: /lib/systemd/system/docker.socket:6: ListenStream= references a path below legacy directory /var/run/, updating /var/run/docker.sock → /run/docker.sock; please update the unit file accordingly.
Jan 15 18:20:46 ubuntu-server systemd[1]: Reloading.
Jan 15 18:20:46 ubuntu-server systemd[1]: /lib/systemd/system/dbus.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/dbus/system_bus_socket → /run/dbus/system_bus_socket; please update the unit file accordingly.
Jan 15 18:20:46 ubuntu-server systemd[1]: /lib/systemd/system/docker.socket:6: ListenStream= references a path below legacy directory /var/run/, updating /var/run/docker.sock → /run/docker.sock; please update the unit file accordingly.
Jan 15 18:20:46 ubuntu-server systemd[1]: Starting Refresh fwupd metadata and update motd...
Jan 15 18:20:46 ubuntu-server systemd[1]: Started Atop advanced performance monitor.
Jan 15 18:20:46 ubuntu-server systemd[1]: Reloading.
Jan 15 18:20:47 ubuntu-server systemd[1]: /lib/systemd/system/dbus.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/dbus/system_bus_socket → /run/dbus/system_bus_socket; please update the unit file accordingly.
Jan 15 18:20:47 ubuntu-server systemd[1]: /lib/systemd/system/docker.socket:6: ListenStream= references a path below legacy directory /var/run/, updating /var/run/docker.sock → /run/docker.sock; please update the unit file accordingly.
Jan 15 18:20:47 ubuntu-server systemd[1]: Starting Daily apt download activities...
Jan 15 18:20:47 ubuntu-server dbus-daemon[741]: [system] Activating via systemd: service name='org.freedesktop.fwupd' unit='fwupd.service' requested by ':1.18' (uid=62803 pid=3469 comm="/usr/bin/fwupdmgr refresh --no-metadata-check " label="unconfined")
Jan 15 18:20:47 ubuntu-server systemd[1]: Starting Firmware update daemon...
Jan 15 18:20:47 ubuntu-server systemd[1]: Reloading.
Jan 15 18:20:47 ubuntu-server systemd[1]: /lib/systemd/system/dbus.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/dbus/system_bus_socket → /run/dbus/system_bus_socket; please update the unit file accordingly.
Jan 15 18:20:47 ubuntu-server systemd[1]: /lib/systemd/system/docker.socket:6: ListenStream= references a path below legacy directory /var/run/, updating /var/run/docker.sock → /run/docker.sock; please update the unit file accordingly.
Jan 15 18:20:47 ubuntu-server dbus-daemon[741]: [system] Successfully activated service 'org.freedesktop.fwupd'
Jan 15 18:20:47 ubuntu-server systemd[1]: Starting Atop process accounting daemon...
Jan 15 18:20:47 ubuntu-server atopacctd[3585]: Version: 2.4.0 - 2019/01/12 20:37:57  <gerlof.langeveld@atoptool.nl>
Jan 15 18:20:47 ubuntu-server kernel: [ 3408.261891] Process accounting resumed
Jan 15 18:20:47 ubuntu-server atopacctd[3585]: accounting to /run/pacct_source
Jan 15 18:20:47 ubuntu-server systemd[1]: Started Atop process accounting daemon.
Jan 15 18:20:47 ubuntu-server systemd[1]: Started Firmware update daemon.
Jan 15 18:20:48 ubuntu-server systemd[1]: apt-daily.service: Succeeded.
Jan 15 18:20:48 ubuntu-server systemd[1]: Finished Daily apt download activities.
Jan 15 18:20:48 ubuntu-server fwupdmgr[3469]: Fetching metadata https://cdn.fwupd.org/downloads/firmware.xml.gz
Jan 15 18:20:48 ubuntu-server fwupdmgr[3469]: Fetching signature https://cdn.fwupd.org/downloads/firmware.xml.gz.asc
Jan 15 18:20:49 ubuntu-server fwupdmgr[3469]: Successfully downloaded new metadata: 0 local devices supported
Jan 15 18:20:49 ubuntu-server systemd[1]: fwupd-refresh.service: Succeeded.
Jan 15 18:20:49 ubuntu-server systemd[1]: Finished Refresh fwupd metadata and update motd.
Jan 15 18:20:49 ubuntu-server kernel: [ 3410.006692] perf: interrupt took too long (3176 > 3162), lowering kernel.perf_event_max_sample_rate to 62750
Jan 15 18:20:58 ubuntu-server systemd[1]: Reloading.
Jan 15 18:20:58 ubuntu-server systemd[1]: /lib/systemd/system/dbus.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/dbus/system_bus_socket → /run/dbus/system_bus_socket; please update the unit file accordingly.
Jan 15 18:20:58 ubuntu-server systemd[1]: /lib/systemd/system/docker.socket:6: ListenStream= references a path below legacy directory /var/run/, updating /var/run/docker.sock → /run/docker.sock; please update the unit file accordingly.
Jan 15 18:20:59 ubuntu-server dbus-daemon[741]: [system] Activating via systemd: service name='org.freedesktop.PackageKit' unit='packagekit.service' requested by ':1.20' (uid=0 pid=6172 comm="/usr/bin/gdbus call --system --dest org.freedeskto" label="unconfined")
Jan 15 18:20:59 ubuntu-server systemd[1]: Starting PackageKit Daemon...
Jan 15 18:20:59 ubuntu-server PackageKit: daemon start
Jan 15 18:20:59 ubuntu-server dbus-daemon[741]: [system] Successfully activated service 'org.freedesktop.PackageKit'
Jan 15 18:20:59 ubuntu-server systemd[1]: Started PackageKit Daemon.
Jan 15 18:24:12 ubuntu-server freshclam[736]: Fri Jan 15 18:24:12 2021 -> Received signal: wake up
Jan 15 18:24:12 ubuntu-server freshclam[736]: Fri Jan 15 18:24:12 2021 -> ClamAV update process started at Fri Jan 15 18:24:12 2021
Jan 15 18:24:12 ubuntu-server freshclam[736]: Fri Jan 15 18:24:12 2021 -> daily.cld database is up to date (version: 26050, sigs: 4211903, f-level: 63, builder: raynman)
Jan 15 18:24:12 ubuntu-server freshclam[736]: Fri Jan 15 18:24:12 2021 -> main.cld database is up to date (version: 59, sigs: 4564902, f-level: 60, builder: sigmgr)
Jan 15 18:24:12 ubuntu-server freshclam[736]: Fri Jan 15 18:24:12 2021 -> bytecode.cld database is up to date (version: 331, sigs: 94, f-level: 63, builder: anvilleg)
Jan 15 18:26:04 ubuntu-server PackageKit: daemon quit
Jan 15 18:26:04 ubuntu-server systemd[1]: packagekit.service: Succeeded.
Jan 15 18:39:01 ubuntu-server CRON[6378]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jan 15 18:39:14 ubuntu-server systemd[1]: Starting Clean php session files...
Jan 15 18:39:14 ubuntu-server systemd[1]: phpsessionclean.service: Succeeded.
Jan 15 18:39:14 ubuntu-server systemd[1]: Finished Clean php session files.
Jan 15 18:50:01 ubuntu-server CRON[7133]: (daemon) CMD (test -x /usr/bin/debsecan && /usr/bin/debsecan --cron)
Jan 15 18:50:01 ubuntu-server systemd-resolved[720]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Jan 15 18:54:11 ubuntu-server smartd[767]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 71 to 72
Jan 15 18:54:26 ubuntu-server kernel: [ 5426.469994] perf: interrupt took too long (3972 > 3970), lowering kernel.perf_event_max_sample_rate to 50250

Edit 2020-01-18:

There are only old messages in the nextcloud.log. They are all related to filenames that are too long:

{"reqId":"ltp0nHmiuog6ICN5kZVL","level":4,"time":"2020-10-18T18:07:35+00:00","remoteAddr":"10.0.0.3","user":"{username}","app":"webdav","method":"PUT","url":"/nextcloud/remote.php/dav/files/{username}/Music-MP3/Music/Various-Anna_Bolena_[Callas,_Gavazzeni_-_Milano]/25.Maria_Meneghini_Callas,_Gabriella_Carturan,_Gianni_Raimondi,_Plinio_Clabassi,_Coro_del_Teatro_alla_Scala_di_Milano,_Orchestra_del_Teatro_alla_Scala_di_Milano,_dir.;_Gianandrea_Gavazzeni-Anna_Bolena_2.Akt_-_3._Szene_-_Cielo,_a_miei_lunghi_spaSimi.mp3","message":{"Exception":"OCA\\DAV\\Connector\\Sabre\\Exception\\InvalidPath","Message":"File name is too long","Code":0,"Trace":[{"file":"/var/www/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Tree.php","line":76,"function":"getChild","class":"OCA\\DAV\\Connector\\Sabre\\Directory","type":"->","args":["25.Maria_Meneghini_Callas,_Gabriella_Carturan,_Gianni_Raimondi,_Plinio_Clabassi,_Coro_del_Teatro_alla_Scala_di_Milano,_Orchestra_del_Teatro_alla_Scala_di_Milano,_dir.;_Gianandrea_Gavazzeni-Anna_Bolena_2.Akt_-_3._Szene_-_Cielo,_a_miei_lunghi_spaSimi.mp3"]},{"file":"/var/www/html/nextcloud/apps/dav/lib/Connector/Sabre/LockPlugin.php","line":59,"function":"getNodeForPath","class":"Sabre\\DAV\\Tree","type":"->","args":["files/{username}/Music-MP3/Music/Various-Anna_Bolena_[Callas,_Gavazzeni_-_Milano]/25.Maria_Meneghini_Callas,_Gabriella_Carturan,_Gianni_Raimondi,_Plinio_Clabassi,_Coro_del_Teatro_alla_Scala_di_Milano,_Orchestra_del_Teatro_alla_Scala_di_Milano,_dir.;_Gianandrea_Gavazzeni-Anna_Bolena_2.Akt_-_3._Szene_-_Cielo,_a_miei_lunghi_spaSimi.mp3"]},{"function":"getLock","class":"OCA\\DAV\\Connector\\Sabre\\LockPlugin","type":"->","args":[{"absoluteUrl":"https://10.0.0.7/nextcloud/remote.php/dav/files/{username}/Music-MP3/Music/Various-Anna_Bolena_[Callas,_Gavazzeni_-_Milano]/
2

Could be something with php and if you disabled the logging, it could explain why you can’t see anything in the logs.

3

All,

Having the same issue on my linxserver.io docker running in unraid.

And there seems to be a problem no matter what the platform. Saw a post from someone running NC on a Pi…

I don’t think it is a php problem because I looked at the version (php -v) in Unraid and got this response:

PHP 7.4.14 (cli) (built: Jan 7 2021 11:46:25) ( NTS )
Copyright © The PHP Group
Zend Engine v3.4.0, Copyright © Zend Technologies
with Zend OPcache v7.4.14, Copyright ©, by Zend Technologies

The PHP version was updated Jan7-2021 and it was working just before the update…

I think that its one of those NC updates that goes bad. It seems every 2nd or 3rd update - the update
:poop: the bed.

Cheers
Toolmanz

4

Sorry, I didn’t mean it’s a problem of php itself, rather than the disabled logging prevents you from getting error messages which are needed to identify the problem.

It works for others though. So without more information it’s pure speculation what might be wrong.

5

I haven’t disabled logging. I have done the opposite; setting loglevel to DEBUG, which is the maximum detail - Logging — Nextcloud latest Administration Manual latest documentation

I just did that recently though, so previous errors may go unreported, as it was on the default level before.

6

On my server I now got php 7.4.3 running. Do you happen to know if there any knows issues with running 7.4.3 together with NC 18.0.3.0? Thanks.

Note that I haven’t updated the Nextcloud version, but I have updated the platform it is running on (the Linux kernel, php, etc.). Perhaps an update to the latest NC could solve this?

7

In the Apache-log, I find this:

[Mon Jan 18 00:00:34.347083 2021] [ssl:warn] [pid 988] AH01909: 10.0.0.7:443:0 server certificate does NOT include an ID which matches the server name

That shouldn’t cause an internal server error, should it? worst case, it would just show a warning about the SSL certificate in the browser, right?

I also see - from the Apache access log - that there is one successful operation (200) followed by two unsuccessful ones (500). Does that tell us anything? Thanks.

8

Gave up on the fix and installed Nextcloud anew. Guess the learning point is to keep NC updated…

9

If it is related to a known problem, it could have fixed it. If it is some configuration problem, it will perhaps come back.

Yes, the error 500 are the problematic ones. If you know what was called, it could give an indication. And then better find out, how to increase logging that we get more information about it. Well, we must check that next time it happens (hopefully not the case).

10

Thanks, @tflidd. Will increase logging as the first order of business.

11

Next time, a look in the /var/log/apache2/error.log maybe a good idea. Perhaps the /var/log/syslog and the /var/log/php7.x-fpm.log gives some hints, too (replace the x with your php version).

12

I looked in the syslog and the Apache error log, see above. I completely forgot about the php-log, though. Thanks for the reminder.