Trouble after upgrade: Internal Server Error

[Smitje]

After upgrading my nextcloud doesn’t work.
I may have made a right mess of it, or it may be something simple.
When I go to the url I get an “internal server error” with a message suggesting to view the logs. There are so many log files around the system I don’t know where to start.

Nextcloud version : 11 initially now 14.0.4.2
Operating system and version : Raspbian GNU/Linux 8 (jessie) on raspberry pi 3
Apache or nginx version : Apache 2.4.10
PHP version :7.0.30

After upgrading I get:
‘’’
Internal Server Error

The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.
‘’’

Is this the first time you’ve seen this error? : N

Steps to replicate it:

1. I was still at NC 11 so it really needed upgrading. I tried to upgrade via the admin login / account, which looked like it went ok but then I got the “Internal server error” when I tried to log in.
2. Thinking I would re-try I ran occ upgrade from the command line. But that brought me to NC 13 without changing the problem. (this may have been a big mistake.)
3. Thinking I was in trouble anyway I ran the command again ending up at version 14.something. and did the recommended db updates, which looked like they finished ok.
4. I tried disabling all non essential plugins through occ, because I found various posts blaming this or the other plugin for internal server errors but to no avail.
5. there are so many .log files I dont really no where to start. the section pasted below from nextcloud.log in my data directory seems to be repeating over and over again.
6. I tried to do a fresh manual upgrade today, replacing /var/www/nextcloud with a freshly downloaded one and putting back the config.php (included below)
   but I still get the same error message on a white screen.

One funny thing is I found a .php file that looks like it creates the error text. So I changed the text in the file, but I don’t see the change in my browser.

Any suggestions are apreciated, no problem if more logs are needed, but please include a hint to where I can find them.

Cheers,
Smitje

The last section of Nextcloud .log in my data folder (not located in www/nextcloud/):

{"reqId":"7mtNw8z5VzVbmNBxpfre","level":3,"time":"2018-12-05T22:33:27+00:00","remoteAddr":"192.168.178.1","user":"--","app":"remote","method":"GET","url":"\/nextcloud\/status.php","message":{"Exception":"OC\\HintException","Message":"Memcache \\OC\\Memcache\\APC not available for local cache","Code":0,"Trace":[{"file":"\/var\/www\/nextcloud\/lib\/private\/Server.php","line":502,"function":"__construct","class":"OC\\Memcache\\Factory","type":"->","args":["50a3b60b9eab8f5e7729cd966ad2afc7",{"__class__":"OC\\Log"},"\\OC\\Memcache\\APC","\\OC\\Memcache\\APC",null]},{"file":"\/var\/www\/nextcloud\/3rdparty\/pimple\/pimple\/src\/Pimple\/Container.php","line":118,"function":"OC\\{closure}","class":"OC\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/nextcloud\/lib\/private\/AppFramework\/Utility\/SimpleContainer.php","line":117,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OC\\Memcache\\Factory"]},{"file":"\/var\/www\/nextcloud\/lib\/private\/ServerContainer.php","line":132,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\Memcache\\Factory"]},{"file":"\/var\/www\/nextcloud\/lib\/private\/AppFramework\/Utility\/SimpleContainer.php","line":165,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OC\\Memcache\\Factory"]},{"file":"\/var\/www\/nextcloud\/3rdparty\/pimple\/pimple\/src\/Pimple\/Container.php","line":114,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/nextcloud\/lib\/private\/AppFramework\/Utility\/SimpleContainer.php","line":117,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["MemCacheFactory"]},{"file":"\/var\/www\/nextcloud\/lib\/private\/ServerContainer.php","line":132,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["MemCacheFactory"]},{"file":"\/var\/www\/nextcloud\/lib\/private\/Server.php","line":1514,"function":"query","class":"OC\\ServerContainer","type":"->","args":["MemCacheFactory"]},{"file":"\/var\/www\/nextcloud\/lib\/private\/Server.php","line":686,"function":"getMemCacheFactory","class":"OC\\Server","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/pimple\/pimple\/src\/Pimple\/Container.php","line":118,"function":"OC\\{closure}","class":"OC\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/nextcloud\/lib\/private\/AppFramework\/Utility\/SimpleContainer.php","line":117,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["OC\\App\\AppManager"]},{"file":"\/var\/www\/nextcloud\/lib\/private\/ServerContainer.php","line":132,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["OC\\App\\AppManager"]},{"file":"\/var\/www\/nextcloud\/lib\/private\/AppFramework\/Utility\/SimpleContainer.php","line":165,"function":"query","class":"OC\\ServerContainer","type":"->","args":["OC\\App\\AppManager"]},{"file":"\/var\/www\/nextcloud\/3rdparty\/pimple\/pimple\/src\/Pimple\/Container.php","line":114,"function":"OC\\AppFramework\\Utility\\{closure}","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/nextcloud\/lib\/private\/AppFramework\/Utility\/SimpleContainer.php","line":117,"function":"offsetGet","class":"Pimple\\Container","type":"->","args":["AppManager"]},{"file":"\/var\/www\/nextcloud\/lib\/private\/ServerContainer.php","line":132,"function":"query","class":"OC\\AppFramework\\Utility\\SimpleContainer","type":"->","args":["AppManager"]},{"file":"\/var\/www\/nextcloud\/lib\/private\/Server.php","line":1704,"function":"query","class":"OC\\ServerContainer","type":"->","args":["AppManager"]},{"file":"\/var\/www\/nextcloud\/lib\/private\/legacy\/app.php","line":342,"function":"getAppManager","class":"OC\\Server","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/lib\/private\/legacy\/app.php","line":113,"function":"getEnabledApps","class":"OC_App","type":"::","args":[]},{"file":"\/var\/www\/nextcloud\/lib\/base.php","line":654,"function":"loadApps","class":"OC_App","type":"::","args":[["session"]]},{"file":"\/var\/www\/nextcloud\/lib\/base.php","line":1068,"function":"init","class":"OC","type":"::","args":[]},{"file":"\/var\/www\/nextcloud\/status.php","line":36,"args":["\/var\/www\/nextcloud\/lib\/base.php"],"function":"require_once"}],"File":"\/var\/www\/nextcloud\/lib\/private\/Memcache\/Factory.php","Line":99,"Hint":"Is the matching PHP module installed and enabled?","CustomMessage":"--"},"userAgent":"Mozilla\/5.0 (Linux) mirall\/2.5.0git (Nextcloud)","version":"14.0.4.2"}
{"reqId":"7mtNw8z5VzVbmNBxpfre","level":3,"time":"2018-12-05T22:33:27+00:00","remoteAddr":"192.168.178.1","user":"--","app":"PHP","method":"GET","url":"\/nextcloud\/status.php","message":"You are using a fallback implementation of the intl extension. Installing the native one is highly recommended instead. at \/var\/www\/nextcloud\/3rdparty\/patchwork\/utf8\/src\/Patchwork\/Utf8\/Bootup\/intl.php#18","userAgent":"Mozilla\/5.0 (Linux) mirall\/2.5.0git (Nextcloud)","version":"14.0.4.2"}

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

$CONFIG = array (
  'instanceid' => '##',
  'passwordsalt' => '##',
  'secret' => '##',
  'trusted_domains' => 
  array (
    0 => 'myurl',
  ),
  'datadirectory' => '/CloudData',
  'overwrite.cli.url' => 'https://myurl/nextcloud',
  'dbtype' => 'mysql',
  'version' => '14.0.4.2',
  'dbname' => '##',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => '##',
  'dbpassword' => '##',
  'logtimezone' => 'UTC',
  'installed' => true,
  'memcache.local' => '\\OC\\Memcache\\APC',
  'maintenance' => false,
  'theme' => '',
  'loglevel' => 2,
);

The output of your Apache/nginx/system log in /var/log/____:

PASTE HERE

[AmonBune]

Try this to try to repair your installation and database:
sudo -u www-data php occ maintenance:repair

Also execute the following to check for any anomalies:
sudo -u www-data php occ integrity:check-core

[Smitje]

Hi AmonBune,

Thanks for your tips.
I am afraid I need a bit more help.

The first command runs fine, and appears to fix some calendar things. (output below)
The second one asks for more arguments, and doesn’t appear to do anything. I need some help here, I think I can find “username” “hostname” and “database” but what should “type” be?. (output belower)
The third could be the most interesting results in:

  - INVALID_HASH:
    - index.php:
      - expected: ###Some Long Number I replaced
      - current:  ###Some Different Long Number I replaced 

Result of " sudo -u www-data php occ maintenance:repair":

/var/www/nextcloud $ sudo -u www-data php occ maintenance:repair - Repair MySQL collation
- All tables already have the correct collation → nothing to do

• Repair mime types
• Clean tags and favorites
  • 0 tags of deleted users have been removed.
  • 0 tags for delete files have been removed.
  • 0 tag entries for deleted tags have been removed.
  • 0 tags with no entries have been removed.
• Repair invalid shares
• Remove shares of a users root folder
• Move .step file of updater to backup location
• Fix potential broken mount points
  • No mounts updated
• Repair invalid paths in file cache
• Add log rotate job
• Clear frontend caches
  • Image cache cleared
  • SCSS cache cleared
  • JS cache cleared
• Add preview background cleanup job
• Queue a one-time job to cleanup old backups of the updater
• Repair pending cron jobs
  • No need to repair pending cron jobs.
• Fix component of birthday calendars
  • 1 birthday calendars updated.
• Fix broken values of calendar objects
  0 [>---------------------------]
• Registering building of calendar search index as background job
  • Repair step already executed
• Fix the share type of guest shares when migrating from ownCloud
• Copy the share password into the dedicated column

Result of " sudo -u www-data php occ db:convert-type":

Not enough arguments (missing: “type, username, hostname, database”).

db:convert-type [–port PORT] [–password PASSWORD] [–clear-schema] [–all-apps] [–chunk-size CHUNK-SIZE] [–]

Cheers,
Smitje

[Schmu]

Hi @Smitje,

Don’t run that command:
sudo -u www-data php occ db:convert-type

It actually converts the database from SQLite to MySQL for example. So this is not applicable here.

For details:
https://docs.nextcloud.com/server/15/admin_manual/configuration_database/db_conversion.html

[AmonBune]

@Smitje Sorry, my bad. db:convert-type isn’t needed, as @Schmu already said.

But regarding your index.php…
I’m not sure what happened to it but try to download your Nextcloud version from https://nextcloud.com/install/#instructions-server and replace your index.php with the one from the archive. You should at least fix something then. Remember to change the ownership of the new file to www-data after doing that:
sudo chown www-data:www-data index.php

Or if you want to do it over the whole NC folder:
sudo chown -R www-data:www-data .

[Smitje]

Hi AminBune and Schmu,
Thanks for your replies,

I tried replacing the index.php (and edited the text to verify it is what is shown, it isn’t)
So I checked /etc/apache2/apache2.conf and sitesavailable
and they do point to /var/www/nextcloud.

also I emptied my firefox cache. But I still get the “internal server error” without my edit to the index.php text.

Refreshing the entire /var/www/nextcloud dir with a fresh one (downloaded last week so version 14) and replacing config.php did not help either.

I’m lost all ideas are welcome.
Cheers,
Smitje

[Schmu]

Can you post a screenshot here of the “Internal server error”?
Could you please also post your “sites-enabled” file contents here?

[Smitje]

Hi Schmu,

Cant paste the screenshot but here is the pasted text:

Internal Server Error

The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.

It is black text on a white background, Firefox also shows the URL in the address-bar: https://myurl/nextcloud/

sites-enabled contains:

lrwxrwxrwx 1 root root 35 Mar 8 2017 000-default.conf → …/sites-available/000-default.conf
lrwxrwxrwx 1 root root 52 Mar 11 2017 000-default-le-ssl.conf → /etc/apache2/sites-available/000-default-le-ssl.conf
lrwxrwxrwx 1 root root 35 Mar 9 2017 default-ssl.conf → …/sites-available/default-ssl.conf
lrwxrwxrwx 1 root root 43 Mar 9 2017 nextcloud.conf → /etc/apache2/sites-available/nextcloud.conf

this is sites-enabled/nextcloud.conf:

Alias /nextcloud “/var/www/nextcloud/”

<Directory /var/www/nextcloud/>
Options +FollowSymlinks
AllowOverride All

Dav off

SetEnv HOME /var/www/nextcloud
SetEnv HTTP_HOME /var/www/nextcloud

and for good measure here is the last part of my index.php with the edited text:

                // no way to log it properly - but to avoid a white page of death we try harder and ignore this one here
            }

            //show the user a detailed error page
            OC_Template::printExceptionErrorPage($ex, 500);
    }

} catch (\OC\User\LoginException $ex) {
OC_Template::printErrorPage($ex->getMessage(), $ex->getMessage(), 403);
} catch (Exception $ex) {
\OC::$server->getLogger()->logException($ex, array(‘app’ => ‘index’));

    //show the user a detailed error page
    OC_Template::printExceptionErrorPage($ex, 500);

} catch (Error $ex) {
try {
\OC::$server->getLogger()->logException($ex, array(‘app’ => ‘index’));
} catch (Error $e) {
http_response_code(500);
header(‘Content-Type: text/plain; charset=utf-8’);
print(“Internal Server Error\n\n”);
print(“The server encountered an internal error and was unable to complete your request.\n”);
print("Please contact the server administrator if this error reappears multiple times, please include the technical deta$
print(“More details can be found in the webserver. log IS THAT TRUE?.\n”);

            throw $e;
    }
   OC_Template::printExceptionErrorPage($ex, 500);

}

Cheers,
Smitje