Blank page shortly after upgrade to 16.0.1.1, but may be unrelated ... where to start debugging?

ℹ️ Support
1

Nextcloud version: 15.0.2.0 upgraded to 16.0.1.1
Operating system and version: Ubuntu 18.04
Apache or nginx version: Apache 2.4.29
PHP version: PHP 7.3.6

The issue you are facing:

I ran the web-based auto-updater last night at 20:00 (Vienna time). It completed w/o error. Initially, the updated server instance showed 3 of my apps were disabled (Calendar, Talk, Tasks), with a red-flagged option to enable them despite not having been tested on this version of NextCloud. I also saw a warning message noting that updated status info was still being processed. Came back 3-5 minutes later, and both the status info was complete, and the red flags were gone (just a regular “enable” button for each) … so I enabled them.

At that point, I also did Linux system upgrades (apt update && apt upgrade) covering 3-4 months of backlogged patches.

Everything was working fine when I went to bed last night.

Nine hours later, this morning at 6:25, blank page for login and all client syncing died.

I have 4 “file not found” errors in the Nextcloud log, a few minutes after the upgrade completed, and nothing after that. I have normal, functional syncing log entries every minute or two in the php log until 6:25 this morning, then all 500 errors.

Oh, also, I received a Nextcloud client sync notification, last night at 22:00 (2 hrs after the upgrade), that “Some of your link shares have been removed.” I have no idea what that means.

Is this the first time you’ve seen this error?:
– No (?) I hit the “blank page” once, after a previous upgrade, but this seems different.

Steps to replicate it:

IDK

The output of your Nextcloud log in Admin > Logging:

Not available via NC … from the terminal, nextcloud.log, here’s the last “updater” entry and the first 'file not found entry (the other 3 are similar)

{"reqId":"iZkpPdNpN8rbmcKtVkeg","level":1,"time":"2019-07-01T19:56:22+00:00","remoteAddr":"80.98.37.78","user":"--","app":"updater","method":"GET","url":"\/core\/ajax\/update.php?requesttoken=imaVbKlfRh5irZviThbkTU3i4FK6MyWfoWoHsrIQ%2FwQ%3D%3A%2FCnwLZ5rJHMb59WaLHLVHya62GLxSl%2FL8xg0w9E%2Ftkg%3D","message":"\\OC\\Updater::resetLogLevel: Reset log level to Warning(2)","userAgent":"Mozilla\/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko\/20100101 Firefox\/67.0","version":"16.0.1.1"}

{"reqId":"qhwjgs3KlChwimRNOyoa","level":3,"time":"2019-07-01T19:56:39+00:00","remoteAddr":"80.98.37.78","user":"nextcloud_admin","app":"index","method":"GET","url":"\/index.php\/apps\/recommendations\/api\/recommendations","message":{"Exception":"OCP\\Files\\NotFoundException","Message":"\/nextcloud_admin\/files\/SharedPics","Code":0,"Trace":[{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/Files\/Node\/Folder.php","line":134,"function":"get","class":"OC\\Files\\Node\\Root","type":"->","args":["\/nextcloud_admin\/files\/SharedPics"]},{"file":"\/var\/www\/html\/nextcloud\/apps\/recommendations\/lib\/Service\/RecentlySharedFilesSource.php","line":122,"function":"get","class":"OC\\Files\\Node\\Folder","type":"->","args":["\/SharedPics"]},{"function":"OCA\\Recommendations\\Service\\{closure}","class":"OCA\\Recommendations\\Service\\RecentlySharedFilesSource","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/html\/nextcloud\/apps\/recommendations\/lib\/Service\/RecentlySharedFilesSource.php","line":127,"function":"array_map","args":[{"__class__":"Closure"},["*** sensitive parameter replaced ***",{"__class__":"OC\\Share20\\Share"},{"__class__":"OC\\Share20\\Share"},{"__class__":"OC\\Share20\\Share"},{"__class__":"OC\\Share20\\Share"},{"__class__":"OC\\Share20\\Share"}]]},{"file":"\/var\/www\/html\/nextcloud\/apps\/recommendations\/lib\/Service\/RecommendationService.php","line":91,"function":"getMostRecentRecommendation","class":"OCA\\Recommendations\\Service\\RecentlySharedFilesSource","type":"->","args":[{"__class__":"OC\\User\\User"},6]},{"function":"OCA\\Recommendations\\Service\\{closure}","class":"OCA\\Recommendations\\Service\\RecommendationService","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/html\/nextcloud\/apps\/recommendations\/lib\/Service\/RecommendationService.php","line":92,"function":"array_reduce","args":[[{"__class__":"OCA\\Recommendations\\Service\\RecentlyCommentedFilesSource"},{"__class__":"OCA\\Recommendations\\Service\\RecentlyEditedFilesSource"},"*** sensitive parameter replaced ***"],{"__class__":"Closure"},[]]},{"file":"\/var\/www\/html\/nextcloud\/apps\/recommendations\/lib\/Controller\/RecommendationController.php","line":63,"function":"getRecommendations","class":"OCA\\Recommendations\\Service\\RecommendationService","type":"->","args":[{"__class__":"OC\\User\\User"}]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/AppFramework\/Http\/Dispatcher.php","line":166,"function":"index","class":"OCA\\Recommendations\\Controller\\RecommendationController","type":"->","args":[]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/AppFramework\/Http\/Dispatcher.php","line":99,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OCA\\Recommendations\\Controller\\RecommendationController"},"index"]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/AppFramework\/App.php","line":126,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OCA\\Recommendations\\Controller\\RecommendationController"},"index"]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/AppFramework\/Routing\/RouteActionHandler.php","line":47,"function":"main","class":"OC\\AppFramework\\App","type":"::","args":["OCA\\Recommendations\\Controller\\RecommendationController","index",{"__class__":"OC\\AppFramework\\DependencyInjection\\DIContainer"},{"_route":"recommendations.recommendation.index"}]},{"function":"__invoke","class":"OC\\AppFramework\\Routing\\RouteActionHandler","type":"->","args":[{"_route":"recommendations.recommendation.index"}]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/Route\/Router.php","line":297,"function":"call_user_func","args":[{"__class__":"OC\\AppFramework\\Routing\\RouteActionHandler"},{"_route":"recommendations.recommendation.index"}]},{"file":"\/var\/www\/html\/nextcloud\/lib\/base.php","line":975,"function":"match","class":"OC\\Route\\Router","type":"->","args":["\/apps\/recommendations\/api\/recommendations"]},{"file":"\/var\/www\/html\/nextcloud\/index.php","line":42,"function":"handleRequest","class":"OC","type":"::","args":[]}],"File":"\/var\/www\/html\/nextcloud\/lib\/private\/Files\/Node\/Root.php","Line":202,"CustomMessage":"--"},"userAgent":"Mozilla\/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko\/20100101 Firefox\/67.0","version":"16.0.1.1"}

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => 'xxxxxx',
  'passwordsalt' => 'xxxxxx',
  'secret' => 'xxxxxx',
  'trusted_domains' => 
  array (
    0 => '99.99.99.99,
    1 => 'domain',
    2 => 'www.domain',
  ),
  'datadirectory' => '/var/www/path-to/nextcloud_data',
  'dbtype' => 'mysql',
  'version' => '16.0.1.1',
  'overwrite.cli.url' => 'http://99.99.99.99/nextcloud',
  'htaccess.RewriteBase' => '/',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'nextcloud_user',
  'dbpassword' => 'xxxxxx',
  'installed' => true,
  'maintenance' => false,
  'mail_smtpmode' => 'smtp',
  'mail_smtpsecure' => 'ssl',
  'mail_sendmailmode' => 'smtp',
  'mail_from_address' => 'nextcloud_admin',
  'mail_domain' => 'domain',
  'mail_smtpauthtype' => 'PLAIN',
  'mail_smtphost' => 'mail.messagingengine.com',
  'mail_smtpport' => '465',
  'mail_smtpauth' => 1,
  'mail_smtpname' => 'my_email',
  'mail_smtppassword' => 'xxxxxx',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'updater.secret' => 'xxxxxx',
  'theme' => '',
  'loglevel' => 2,
);

The output of your Apache/nginx/system log in /var/log/____:

Here is the tail of the log file right up to the moment NextCloud stopped working … what’s with the “Starting Clean php session” every 30 minutes? That pattern started after last night’s upgrade.

Jul  2 04:39:06 vultr systemd[1]: Starting Clean php session files...
Jul  2 04:39:06 vultr systemd[1]: Started Clean php session files.
Jul  2 05:09:00 vultr systemd[1]: Starting Clean php session files...
Jul  2 05:09:00 vultr systemd[1]: Started Clean php session files.
Jul  2 05:09:01 vultr CRON[29461]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jul  2 05:17:01 vultr CRON[29924]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul  2 05:39:00 vultr systemd[1]: Starting Clean php session files...
Jul  2 05:39:00 vultr systemd[1]: Started Clean php session files.
Jul  2 05:39:01 vultr CRON[31493]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jul  2 06:09:01 vultr CRON[1347]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jul  2 06:09:06 vultr systemd[1]: Starting Clean php session files...
Jul  2 06:09:06 vultr systemd[1]: Started Clean php session files.
Jul  2 06:17:01 vultr CRON[1980]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul  2 06:25:01 vultr CRON[2439]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Jul  2 06:25:02 vultr cracklib: no dictionary update necessary.
Jul  2 06:25:03 vultr systemd[1]: Reloading The Apache HTTP Server.
Jul  2 06:25:03 vultr systemd[1]: Reloaded The Apache HTTP Server.

And here’s the start of the final system log, started immediately after NC died.

Jul  2 06:25:03 vultr rsyslogd:  [origin software="rsyslogd" swVersion="8.32.0" x-pid="501" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Jul  2 06:25:10 vultr popularity-contest: unable to submit report to https://popcon.ubuntu.com/popcon-submit.cgi.
Jul  2 06:25:10 vultr popularity-contest: unable to submit report.
Jul  2 06:32:06 vultr systemd[1]: Starting Daily apt upgrade and clean activities...
Jul  2 06:32:06 vultr systemd[1]: Started Daily apt upgrade and clean activities.
Jul  2 06:39:01 vultr CRON[2863]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jul  2 06:39:06 vultr systemd[1]: Starting Clean php session files...
Jul  2 06:39:06 vultr systemd[1]: Started Clean php session files.
Jul  2 07:09:01 vultr CRON[3175]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jul  2 07:09:02 vultr systemd[1]: Starting Clean php session files...
Jul  2 07:09:02 vultr systemd[1]: Started Clean php session files.
Jul  2 07:15:04 vultr systemd[1]: Created slice User Slice of root.
Jul  2 07:15:04 vultr systemd[1]: Starting User Manager for UID 0...
Jul  2 07:15:04 vultr systemd[1]: Started Session 7464 of user root.
Jul  2 07:15:04 vultr systemd[3333]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Jul  2 07:15:04 vultr systemd[3333]: Listening on GnuPG network certificate management daemon.
Jul  2 07:15:04 vultr systemd[3333]: Listening on GnuPG cryptographic agent and passphrase cache.
Jul  2 07:15:04 vultr systemd[3333]: Reached target Timers.
Jul  2 07:15:04 vultr systemd[3333]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers).
Jul  2 07:15:04 vultr systemd[3333]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Jul  2 07:15:04 vultr systemd[3333]: Reached target Paths.
Jul  2 07:15:04 vultr systemd[3333]: Reached target Sockets.
Jul  2 07:15:04 vultr systemd[3333]: Reached target Basic System.
Jul  2 07:15:04 vultr systemd[1]: Started User Manager for UID 0.
Jul  2 07:15:04 vultr systemd[3333]: Reached target Default.
Jul  2 07:15:04 vultr systemd[3333]: Startup finished in 27ms.
Jul  2 07:15:18 vultr systemd[1]: Stopping Availability of block devices...
Jul  2 07:15:18 vultr systemd[1]: Stopped target Timers.
Jul  2 07:15:18 vultr systemd[1]: Stopped Daily apt upgrade and clean activities.
Jul  2 07:15:18 vultr systemd[1]: Stopped Daily apt download activities.
Jul  2 07:15:18 vultr systemd[1]: Stopped Clean PHP session files every 30 mins.
Jul  2 07:15:18 vultr systemd[1]: Stopped Daily Cleanup of Temporary Directories.
Jul  2 07:15:18 vultr systemd[1]: Stopped Message of the Day.
Jul  2 07:15:18 vultr systemd[1]: Stopped Discard unused blocks once a week.
Jul  2 07:15:18 vultr systemd[1]: Stopped Run certbot twice daily.
Jul  2 07:15:18 vultr systemd[1]: Stopped target Graphical Interface.
Jul  2 07:15:18 vultr systemd[1]: Stopping Accounts Service...

Thanks to anyone actually willing to dig through this with me.

2

Lets start with easy part.

  1. Check if you nextcloud installation is really there (it was the case when it was simply deleted like here Update to 13 fails - core/shipped.json is not available and you have to do some manual job via ssh to bring it back).

  2. Check if php is up and running. Check php Logs (seems already done).

  3. Check that your webserver is up and running. Check Web server logs also when you try to connect, e.g. for apache2 under /var/log/apache2/access.log and error.log via command tail -f /var/log/apache2/access.log and then tail -f /var/log/apache2/error.log

  4. You are using higher PHP version, check if it really works with your installation, e.g. here was mentioned some problems: Upgrade php 7.0 to 7.3 and here https://github.com/nextcloud/server/issues/11868

  5. Check if your DB is up and running.

3

Hello gas85,

Yep, all of that checks out. I’m not 100% sure about the PHP version – I mean, I know I have 7.3 on there, but I’m not quite sure how to tell if NextCloud still knows to use that version, or if it somehow reset itself back to trying to use an older version.

Probably fine, though … in the apache2/error.log, I have this going on –

[Tue Jul 02 15:19:46.923349 2019] [php7:error] [pid 5913] [client 99.122.66.238:63292] PHP Fatal error: Uncaught Doctrine\DBAL\DBALException: Failed to connect to the database: An exception occurred in driver: could not find driver in /var/www/html/nextcloud/lib/private/DB/Connection.php:64\nStack trace:\n#0 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(448): OC\DB\Connection->connect()\n#1 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(410): Doctrine\DBAL\Connection->getDatabasePlatformVersion()\n#2 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(354): Doctrine\DBAL\Connection->detectDatabasePlatform()\n#3 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(710): Doctrine\DBAL\Connection->getDatabasePlatform()\n#4 /var/www/html/nextcloud/lib/private/DB/Connection.php(151): Doctrine\DBAL\Connection->setTransactionIsolation(2)\n#5 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DriverManager.php(181): OC\DB\Connection->__construct(Array, Object(Doctrine\DBAL\Driver\PDOMySql\Dri in /var/www/html/nextcloud/lib/private/DB/Connection.php on line 64

It keeps re-throwing this error message every 10 seconds or so.

MySQL is definitely working, and my NextCloud db is still there – I’ve been nosing around in it, looking for issues – but, apparently, NextCloud has lost the ability to connect to the db. Although, I do not understand what the specific issue is. I opened the first file mentioned, “Connection.php”, it’s just throwing an exception because it couldn’t connect to the db. Do I need to drill through every file in the stacktrace? Or does this have an obvious solution?

Thanx,

4

if you just want to test the db connection you’ll find here example php code.

https://www.w3schools.com/php/php_mysql_connect.asp
or

are you sure the php73-mysql module is installed?

5

Addendum … looked back thru the apache2 error.log history.

I got 15-20 of these error messages, right during or immediately after the upgrade, all within 30-40 seconds, and then nothing …

[Mon Jul 01 19:49:05.773098 2019] [php7:error] [pid 2283] [client 99.122.66.238:55555] PHP Fatal error: Uncaught Doctrine\DBAL\DBALException: Failed to connect to the database: An exception occured in driver: SQLSTATE[HY000] [2002] No such file or directory in /var/www/html/nextcloud/lib/private/DB/Connection.php:64\nStack trace:\n#0 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(429): OC\DB\Connection->connect()\n#1 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(389): Doctrine\DBAL\Connection->getDatabasePlatformVersion()\n#2 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(328): Doctrine\DBAL\Connection->detectDatabasePlatform()\n#3 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(623): Doctrine\DBAL\Connection->getDatabasePlatform()\n#4 /var/www/html/nextcloud/lib/private/DB/Connection.php(151): Doctrine\DBAL\Connection->setTransactionIsolation(2)\n#5 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DriverManager.php(172): OC\DB\Connection->__construct(Array, Object(Doctrin in /var/www/html/nextcloud/lib/private/DB/Connection.php on line 64

… until this morning, when apache2 apparently decided to restart itself (I didn’t tell it to), which is when the trouble became apparent to me.

[Tue Jul 02 06:25:03.684158 2019] [mpm_prefork:notice] [pid 2258] AH00171: Graceful restart requested, doing restart

Trying the db connection test, per Reiner_Nippes, now.

6

Addendum … yeah, php script to connect to MySQL works.

sudo apt install php73-mysql returns “E: Unable to locate package php73-mysql”

7

Check ping to local host (ping localhost), if you did not reject it e.g. with iptables.
Also try to restart DB, php and then Apache2 and better via stop and start commands, not via restart.

Check if you are using php7.3 it could be that currently installed and being used is php7.3-fpm in this case you have to run command like sudo service php7.3-fpm restart instead of php7.3. Took it from here:

Check your installed modules by e.g.

php -m

to get the list of installed php modules , you will probably find it helpful to get the list of the currently installed php packages in Ubuntu (source):

sudo dpkg --get-selections | grep -v deinstall | grep php

You can then install the needed modules by selecting from the available Ubuntu php packages , which you can view by running:

sudo apt-cache search php | grep "^php7"
8

All right. I finally figured this out. It is the same ‘gotcha’ that bit me in the backside 6 months ago, so I am documenting the full cycle of the issue here – sure, for everyone else, but mainly for myself when I screw this up again in another 6 months.

TL;DR:

My NextCloud instance was using the wrong version of php. Make abso-damn-lutely certain you are checking your NextCloud php version … then, once you finally realize that, yes, it’s using the wrong version, again, then do this–

sudo a2dismod php_wrong_version
sudo a2enmod php_right_version

Long version …

So, in my case, this time … I think the issue was that I first upgraded NextCloud (15.0.2.0 --> 16.0.1.1), and then minutes later, I did the full Linux system-wide package upgrade (“sudo apt update && sudo apt upgrade”). That covered 3-4 (-6?) months of backlogged Linux patches. No, I did not meticulously read through every one of the 2-page list of updates being applied.

Shame on me.

But I’m betting that, included in that bucket of updates was a nice shiny php7.2 -> php7.3 upgrade … and my entire system – linux, apache, php – all showed that I was running php7.3. Except NextCloud kept right on trying to work with php7.2.

And once NC dies, it is really tricky to figure out which version of php it is using. All the tech websites on Earth say “well, it just uses whatever apache gives it” … well, I’m sorry, but all the tech website on Earth are wrong. (I might be overstating that, a bit).

But every config file (NextCloud, apache, php), every system log, access log, error log, yada-yada … even every active check against apache (phpinfo(), php -v, etc) … unless it is tested from inside the NC site … will give you incorrect info.

*** How to determine what version of php your NextCloud instance is using, after it’s broken. ***

  1. Go to your NextCloud root folder. Where that is, I guess, varies depending on a lot of stuff. For me, it is located in /var/www/html/nextcloud … and that’s not universal, but I believe it’s a pretty common location for it. It should have stuff in it, like core/, data/, occ, robots.txt, themes/
  2. Create a new file, let’s call it phpinfo.php (like “touch phpinfo.php”). The name doesn’t matter, but the .php does.
  3. In that file, add this … (not including the quotes) “<?php phpinfo(); ?>” … save it.
  4. From your browser, go to http://myNextCloudDomain.com/phpinfo.php … that should give you the correct php version that your NextCloud instance is using (along with holy crap a whole lot of other info).

*** Then, once you finally realize that, yes, NextCloud is still/again using the wrong version of php ***

sudo a2dismod php_current-wrong_version
sudo a2enmod php_right_version

Seriously, though … thanks again to the people that tried to help me out with this.

The end.

1 Like
9

I think I have the same issue, but not sure given my limited unix experience and skill. I did have version 13 running on my hosted web server. I used the updater today to upgrade 13 - 14 -> 15 without any observed issues.

Then I tried to update v15 -> v16 there was a non specific error on the webpage and I think it is what has resulted in a blank screen with web access to my NextCloud install.

I was running php7.1 (at least per my host webserver settings), and have upgraded that to 7.3 but nothing is working. I tried the ‘a2dismod’ command (when connected to server via SSH), but I get a command not found.

Am I trying to run the commands the wrong way? Other beginner level tips?

10

after some server review and some greater familiarity with NextCloud, it seems that the updater hit a max file limit on the server and stopped. After clearing backups in the data folder it works.