Very presistent Internal Server Error

Nextcloud version (eg, 12.0.2): 16.0.6.1
Operating system and version (eg, Ubuntu 17.04): Debian 9.11
Apache or nginx version (eg, Apache 2.4.25): Apache/2.4.25
PHP version (eg, 7.1): v7.3.12

The issue you are facing:

I get the Internal server error page from index.php of nextcloud when i try to access it

The Actual Strory:

So i wanted to update to 17. then i learned theres a new intermediate update to 16 so need to do this first run it got an internal server error did restart didnt helped rolled back the vm to where it was before the update still had the error restartet worked again.

somedays later i tried it again from cli it also gave the internal server error but it went all the way through on restart it said my root was corrupted and i should run fsck i couldn’t really find out how to make it work - so i went back to the working snapshot again.

btw. data is on an nfs share and not included in the snapshot of the vm

then im still stuck with the internal server error. all the clients (linux, andrtoid) seem to work just fine though.

Things i did in the last days to get rid of the error:

  • run fsck on ( technically a copy) of the vm - no errors
  • run a scrub (fsck for btrfs) on the data volume - no errors (still runnung (bit that is running regularly anyway)
  • renew letsencrypt certs
  • chowned the whole data folder again
  • disabled all the apps (possible) (enabled again)
  • checked all this https://forum.owncloud.org/viewtopic.php?t=34343
  • updated the vm and host ( apt) ( there was a php update(vm))
  • surely some other things i cant remember…

really don’t know what this stat errors wanna tell me.
oh also i always get/got redirected to common nextcloud paths - but only always got presented the internal server error
also i have an index.html on the document root which always runs perfectly.

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:

  1. Open the Website

The output of your Nextcloud log in Admin > Logging:

{"reqId":"53RzjWZFb4b08hvZeyMz","level":3,"time":"2020-01-09T09:27:56+00:00","remoteAddr":"213.196.140.253","user":"--","app":"PHP","method":"GET","url":"\/owncloud\/login","message":"stat(): stat failed for \/mnt\/xxx-ubuntu-nas\/owncloud\/data\/appdata_ocXXXXXXXX\/js\/core\/merged-template-prepend.js.deps at \/var\/www\/html\/o$
{"reqId":"53RzjWZFb4b08hvZeyMz","level":3,"time":"2020-01-09T09:27:56+00:00","remoteAddr":"213.196.140.253","user":"--","app":"PHP","method":"GET","url":"\/owncloud\/login","message":"filesize(): stat failed for \/mnt\/xxx-ubuntu-nas\/owncloud\/data\/appdata_ocXXXXXXXX\/js\/core\/merged-template-prepend.js.deps at \/var\/www\/htm$
{"reqId":"53RzjWZFb4b08hvZeyMz","level":3,"time":"2020-01-09T09:27:56+00:00","remoteAddr":"213.196.140.253","user":"--","app":"PHP","method":"GET","url":"\/owncloud\/login","message":"file_put_contents(\/mnt\/xxx-ubuntu-nas\/owncloud\/data\/appdata_ocXXXXXXXX\/js\/core\/merged-template-prepend.js): failed to open stream: Datei od$
{"reqId":"53RzjWZFb4b08hvZeyMz","level":3,"time":"2020-01-09T09:27:56+00:00","remoteAddr":"213.196.140.253","user":"--","app":"core","method":"GET","url":"\/owncloud\/login","message":{"Exception":"OCP\\Files\\GenericFileException","Message":"file_put_contents failed","Code":0,"Trace":[{"file":"\/var\/www\/html\/owncloud\/lib\/private$
{"reqId":"53RzjWZFb4b08hvZeyMz","level":3,"time":"2020-01-09T09:27:57+00:00","remoteAddr":"213.196.140.253","user":"--","app":"core","method":"GET","url":"\/owncloud\/login","message":{"Exception":"OCP\\Files\\GenericFileException","Message":"file_put_contents failed","Code":0,"Trace":[{"file":"\/var\/www\/html\/owncloud\/lib\/private$
{"reqId":"EA3eK0UTO9JVz

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => 'ocXXXXXXXX',
  'passwordsalt' => 'XXXX',
  'secret' => 'XXXX',
  'trusted_domains' => 
  array (
    0 => 'XXXX',
    1 => 'XXXX',
    2 => 'XXXX',
  ),
  'datadirectory' => 'XXXX',
  'overwrite.cli.url' => 'XXXX',
  'htaccess.RewriteBase' => '/owncloud',
  'dbtype' => 'mysql',
  'version' => '16.0.6.1',
  'dbname' => 'XXXX',
  'dbhost' => 'localhost',
  'dbtableprefix' => 'XXXX_',
  'dbuser' => 'XXXX',
  'dbpassword' => 'XXXX',
  'logtimezone' => 'UTC',
  'installed' => true,
  'appstore.experimental.enabled' => true,
  'mail_smtpmode' => 'smtp',
  'mail_smtpsecure' => 'tls',
  'mail_from_address' => 'XXXX',
  'mail_domain' => 'XXXX',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpauth' => 1,
  'mail_smtphost' => 'XXXX',
  'mail_smtpport' => 'XXXX',
  'mail_smtpname' => 'XXXX',
  'mail_smtppassword' => 'XXXX',
  'loglevel' => 1,
  'updatechecker' => false,
  'maintenance' => false,
  'theme' => '',
  'updater.release.channel' => 'production',
);

The output of your Apache/nginx/system log in /var/log/____ BEFORE i turned log level to debug today :

[Thu Jan 09 06:25:33.130829 2020] [ssl:warn] [pid 10327] AH01909: 127.0.1.1:443:0 server certificate does NOT include an ID which matches the server name
[Thu Jan 09 06:25:33.131005 2020] [http2:warn] [pid 10327] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected$
[Thu Jan 09 06:25:33.170829 2020] [mpm_prefork:notice] [pid 10327] AH00163: Apache/2.4.25 (Debian) OpenSSL/1.0.2u configured -- resuming normal operations
[Thu Jan 09 06:25:33.170859 2020] [core:notice] [pid 10327] AH00094: Command line: '/usr/sbin/apache2'
stat: cannot stat '/mnt/XXX-ubuntu-nas/owncloud/data/appdata_ocXXXXXXXX/avatar/XXX/avatar.384.png': No such file or directory
stat: cannot stat '/mnt/XXX-ubuntu-nas/owncloud/data/appdata_ocXXXXXXXX/avatar/xsxs/avatar.384.png': No such file or directory
stat: cannot stat '/mnt/XXX-ubuntu-nas/owncloud/data/appdata_ocXXXXXXXX/avatar/sdf/avatar.384.png': No such file or directory
stat: cannot stat '/mnt/XXX-ubuntu-nas/owncloud/data/appdata_ocXXXXXXXX/avatar/asdfsdf/avatar.384.png': No such file or directory
stat: cannot stat '/mnt/XXX-ubuntu-nas/owncloud/data/appdata_ocXXXXXXXX/avatar/sadfsd/avatar.384.png': No such file or directory
stat: cannot stat '/mnt/XXX-ubuntu-nas/owncloud/data/appdata_ocXXXXXXXX/avatar/asdfsdf/avatar.384.png': No such file or directory
stat: cannot stat '/mnt/XXX-ubuntu-nas/owncloud/data/appdata_ocXXXXXXXX/avatar/asfdsfs/avatar.384.png': No such file or directory
stat: cannot stat '/mnt/XXX-ubuntu-nas/owncloud/data/appdata_ocXXXXXXXX/preview/224612/256-256-crop.png': No such file or directory
stat: cannot stat '/mnt/XXX-ubuntu-nas/owncloud/data/appdata_ocXXXXXXXX/preview/222755/256-256-crop.png': No such file or directory
stat: cannot stat '/mnt/XXX-ubuntu-nas/owncloud/data/appdata_ocXXXXXXXX/preview/222704/256-256-crop.png': No such file or directory
stat: cannot stat '/mnt/XXX-ubuntu-nas/owncloud/data/appdata_ocXXXXXXXX/avatar/XXX/avatar.384.png': No such file or directory
stat: cannot stat '/mnt/XXX-ubuntu-nas/owncloud/data/appdata_ocXXXXXXXX/preview/227747/256-256-crop.png': No such file or directory
stat: cannot stat '/mnt/XXX-ubuntu-nas/owncloud/data/appdata_ocXXXXXXXX/preview/95683/256-256-crop.png': No such file or directory
stat: cannot stat '/mnt/XXX-ubuntu-nas/owncloud/data/appdata_ocXXXXXXXX/preview/224612/256-256-crop.png': No such file or directory
stat: cannot stat '/mnt/XXX-ubuntu-nas/owncloud/data/appdata_ocXXXXXXXX/preview/222755/256-256-crop.png': No such file or directory
stat: cannot stat '/mnt/XXX-ubuntu-nas/owncloud/data/appdata_ocXXXXXXXX/preview/222704/256-256-crop.png': No such file or directory
stat: cannot stat '/mnt/XXX-ubuntu-nas/owncloud/data/appdata_ocXXXXXXXX/preview/222692/256-256-crop.png': No such file or directory
stat: cannot stat '/mnt/XXX-ubuntu-nas/owncloud/data/appdata_ocXXXXXXXX/preview/222484/256-256-crop.png': No such file or directory
stat: cannot stat '/mnt/XXX-ubuntu-nas/owncloud/data/appdata_ocXXXXXXXX/preview/218654/256-256-crop.png': No such file or directory
stat: cannot stat '/mnt/XXX-ubuntu-nas/owncloud/data/appdata_ocXXXXXXXX/preview/218527/256-256-crop.png': No such file or directory
stat: cannot stat '/mnt/XXX-ubuntu-nas/owncloud/data/appdata_ocXXXXXXXX/preview/217655/256-256-crop.png': No such file or directory

The output of your Apache/nginx/system log in /var/log/____ AFTER i turned log level to debug today :


[Thu Jan 09 10:41:41.478291 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of <RequireAny>: granted
[Thu Jan 09 10:41:41.698889 2020] [ssl:debug] [pid 14868] ssl_engine_kernel.c(366): [client 213.196.140.253:49996] AH02034: Subsequent (No.2) HTTPS request received for child 3 (server 127.0.1.1:443)
[Thu Jan 09 10:41:41.701821 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of Require all granted: granted
[Thu Jan 09 10:41:41.701954 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of <RequireAny>: granted
[Thu Jan 09 10:41:41.702986 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of Require all granted: granted
[Thu Jan 09 10:41:41.703010 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of <RequireAny>: granted
[Thu Jan 09 10:41:41.831153 2020] [ssl:debug] [pid 14868] ssl_engine_kernel.c(366): [client 213.196.140.253:49996] AH02034: Subsequent (No.3) HTTPS request received for child 3 (server 127.0.1.1:443)
[Thu Jan 09 10:41:41.833852 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of Require all granted: granted
[Thu Jan 09 10:41:41.833967 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of <RequireAny>: granted
[Thu Jan 09 10:41:41.835076 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of Require all granted: granted
[Thu Jan 09 10:41:41.835172 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of <RequireAny>: granted
[Thu Jan 09 10:41:42.006411 2020] [ssl:debug] [pid 14868] ssl_engine_kernel.c(366): [client 213.196.140.253:49996] AH02034: Subsequent (No.4) HTTPS request received for child 3 (server 127.0.1.1:443)
[Thu Jan 09 10:41:42.006855 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of Require all granted: granted
[Thu Jan 09 10:41:42.006875 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of <RequireAny>: granted
[Thu Jan 09 10:41:42.007449 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of Require all granted: granted
[Thu Jan 09 10:41:42.007476 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of <RequireAny>: granted
[Thu Jan 09 10:41:42.146360 2020] [ssl:debug] [pid 14868] ssl_engine_kernel.c(366): [client 213.196.140.253:49996] AH02034: Subsequent (No.5) HTTPS request received for child 3 (server 127.0.1.1:443)
[Thu Jan 09 10:41:42.147274 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of Require all granted: granted
[Thu Jan 09 10:41:42.147330 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of <RequireAny>: granted
[Thu Jan 09 10:41:42.147817 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of Require all granted: granted
[Thu Jan 09 10:41:42.147864 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of <RequireAny>: granted
[Thu Jan 09 10:41:42.356704 2020] [ssl:debug] [pid 14868] ssl_engine_kernel.c(366): [client 213.196.140.253:49996] AH02034: Subsequent (No.6) HTTPS request received for child 3 (server 127.0.1.1:443)
[Thu Jan 09 10:41:42.358860 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of Require all granted: granted
[Thu Jan 09 10:41:42.358966 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of <RequireAny>: granted
[Thu Jan 09 10:41:42.360078 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of Require all granted: granted
[Thu Jan 09 10:41:42.360468 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of <RequireAny>: granted
[Thu Jan 09 10:41:42.735399 2020] [ssl:debug] [pid 14868] ssl_engine_kernel.c(366): [client 213.196.140.253:49996] AH02034: Subsequent (No.7) HTTPS request received for child 3 (server 127.0.1.1:443)
[Thu Jan 09 10:41:42.737189 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of Require all granted: granted
[Thu Jan 09 10:41:42.737267 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of <RequireAny>: granted
[Thu Jan 09 10:41:42.738035 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of Require all granted: granted
[Thu Jan 09 10:41:42.738100 2020] [authz_core:debug] [pid 14868] mod_authz_core.c(809): [client 213.196.140.253:49996] AH01626: authorization result of <RequireAny>: granted
[Thu Jan 09 10:41:47.884012 2020] [ssl:debug] [pid 14868] ssl_engine_io.c(1044): [client 213.196.140.253:49996] AH02001: Connection closed to child 3 with standard shutdown (server 127.0.1.1:443)

So, i went for a manual install of the latest nextcloud 17. accordging to the manual upgrade manual.

while backing up the database i saw these two errors and fixed them accordingly:


running the occ upgrade command i got a lot of

Checked for update of app "cospend" in appstore 
stat: der Aufruf von stat für '/mnt/xxxowncloud/data/appdata_xxxxxxxc/appstore/apps.json' ist nicht möglich: Datei oder Verzeichnis nicht gefunden

which makes sense because i got that for the apps that i didn’t copy over from the the old nexctloud.

so the new situation is. in nextcloud config it says its updated to latest 17. but i still get the same internal server error pag on the browser - and also still all the clients and also dav continues to work…

when i check the code for the files app dough e.g (app:check) i get errors

Analysing /var/www/html/owncloud/apps/files/ajax/list.php
 5 errors
    line   30: OC_JSON - Static method of deprecated class must not be called
    line   82: OC_JSON - Static method of deprecated class must not be called
    line   85: OC_JSON - Static method of deprecated class must not be called
    line   93: OC_JSON - Static method of deprecated class must not be called
    line  101: OC_JSON - Static method of deprecated class must not be called
Analysing /var/www/html/owncloud/apps/files/ajax/download.php
 3 errors
    line   32: OCP\User - Static method of deprecated class must not be called
    line   32: OCP\User::checkLoggedIn - Method of deprecated class must not be called
    line   64: OC_Files - Static method of private class must not be called
Analysing /var/www/html/owncloud/apps/files/lib/Helper.php
 1 errors
    line   49: OC_Helper - Static method of private class must not be called
Analysing /var/www/html/owncloud/apps/files/lib/Controller/ViewController.php
 4 errors
    line  108: OC_App - Static method of private class must not be called
    line  130: OC_Helper - Static method of private class must not be called
    line  233: OC_Helper - Static method of private class must not be called
    line  237: OC_Helper - Static method of private class must not be called
Analysing /var/www/html/owncloud/apps/files/lib/BackgroundJob/ScanFiles.php
 1 errors
    line   92: OC_Util - Static method of private class must not be called
Analysing /var/www/html/owncloud/apps/files/recentlist.php
 2 errors
    line   25: OCP\User - Static method of deprecated class must not be called
    line   25: OCP\User::checkLoggedIn - Method of deprecated class must not be called
Analysing /var/www/html/owncloud/apps/files/appinfo/app.php
 1 errors
    line   36: OC_Helper - Static method of private class must not be called
Analysing /var/www/html/owncloud/apps/files/templates/appnavigation.php
 1 errors
    line   86: == - is discouraged
App is not compliant

that may break everythin but its also weird since its the fresh nc17 from the zip.

also if i turn maintenance mode on - i still get nothing else than the server error…

i think - the only things i didnt’ replace by now is

  • the data directory
  • the database
  • the config file

there should be the problem
(or maybae apache config but that would be super weird (other stuff is loading as well)

love when you just keep being logic about things and in the end you actualy solve the problem.

So first, i did some more investigations about the fact, that i get this ** internatl server error but then my error log does not show me any errors… - ppl in the internet were saying this can be caused by broken php scripts and or php modules - so i checked over all my modules again updated, checked them and set them up right version. - didnt help.

i also set up another instance of nextcloud to see if the server is actually broken.
as this was working perfectly fine (and much faster than my actual instance) i came back to

i think - the only things i didnt’ replace by now is
the data directory
the database
the config file

so i
Went into my config.php and started to Mess up things
when i broke the path to the data directory - i suddenly got a proper answer from the webui. (it loaded!!) and told me wrong directory. i could even log in and everything worked when i made a new valid data directory. (apart from there was no data of course)
So then i thought it must be the data directory then

i checked all permissions (again) and had a lok at what was there. (still files from the owncloud age for example) and started deleting stuff until i got a response from the webui.

in the end it was the appdata_instanceid folder. i tried quiet a while to reconstruct that first from my new nextcloud instance and then from my backups (since i have plenty of them) - but nothing acutally worked fully… ( kinda bad sign)

then i learned about that What is the appdata_ocpom4nckwru directory? - #9 by Andymoene and that Screwed up data/appdata_instanceid - #3 by immersenetworks. and that finally got my instance up and runnung.

only 2 months of webui downtime…

So in the end and after ~20+ hours of troubleshooting i think here is what happened (also based on what i saw in the backup iterations):

I started the Update (to latest 16) in the webui and that destroyed my appdata folder. (actually there was only the preview folder left in it) - then neither the commandline updater nor the nextcloud fix/repair commands fixed that. - very bad

So i highly recommend to change stuff that this cannot happen. - eighter the appdata folder should get checked/fixed by the repair commands or it should not be able to cause such a severe problem aka should give out an appropriate error in webui…

1 Like

opened issue: https://github.com/nextcloud/server/issues/19579