Nextcloud version: 15.0.2.0 upgraded to 16.0.1.1
Operating system and version: Ubuntu 18.04
Apache or nginx version: Apache 2.4.29
PHP version: PHP 7.3.6
The issue you are facing:
I ran the web-based auto-updater last night at 20:00 (Vienna time). It completed w/o error. Initially, the updated server instance showed 3 of my apps were disabled (Calendar, Talk, Tasks), with a red-flagged option to enable them despite not having been tested on this version of NextCloud. I also saw a warning message noting that updated status info was still being processed. Came back 3-5 minutes later, and both the status info was complete, and the red flags were gone (just a regular “enable” button for each) … so I enabled them.
At that point, I also did Linux system upgrades (apt update && apt upgrade) covering 3-4 months of backlogged patches.
Everything was working fine when I went to bed last night.
Nine hours later, this morning at 6:25, blank page for login and all client syncing died.
I have 4 “file not found” errors in the Nextcloud log, a few minutes after the upgrade completed, and nothing after that. I have normal, functional syncing log entries every minute or two in the php log until 6:25 this morning, then all 500 errors.
Oh, also, I received a Nextcloud client sync notification, last night at 22:00 (2 hrs after the upgrade), that “Some of your link shares have been removed.” I have no idea what that means.
Is this the first time you’ve seen this error?:
– No (?) I hit the “blank page” once, after a previous upgrade, but this seems different.
Steps to replicate it:
IDK
The output of your Nextcloud log in Admin > Logging:
Not available via NC … from the terminal, nextcloud.log, here’s the last “updater” entry and the first 'file not found entry (the other 3 are similar)
{"reqId":"iZkpPdNpN8rbmcKtVkeg","level":1,"time":"2019-07-01T19:56:22+00:00","remoteAddr":"80.98.37.78","user":"--","app":"updater","method":"GET","url":"\/core\/ajax\/update.php?requesttoken=imaVbKlfRh5irZviThbkTU3i4FK6MyWfoWoHsrIQ%2FwQ%3D%3A%2FCnwLZ5rJHMb59WaLHLVHya62GLxSl%2FL8xg0w9E%2Ftkg%3D","message":"\\OC\\Updater::resetLogLevel: Reset log level to Warning(2)","userAgent":"Mozilla\/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko\/20100101 Firefox\/67.0","version":"16.0.1.1"}
{"reqId":"qhwjgs3KlChwimRNOyoa","level":3,"time":"2019-07-01T19:56:39+00:00","remoteAddr":"80.98.37.78","user":"nextcloud_admin","app":"index","method":"GET","url":"\/index.php\/apps\/recommendations\/api\/recommendations","message":{"Exception":"OCP\\Files\\NotFoundException","Message":"\/nextcloud_admin\/files\/SharedPics","Code":0,"Trace":[{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/Files\/Node\/Folder.php","line":134,"function":"get","class":"OC\\Files\\Node\\Root","type":"->","args":["\/nextcloud_admin\/files\/SharedPics"]},{"file":"\/var\/www\/html\/nextcloud\/apps\/recommendations\/lib\/Service\/RecentlySharedFilesSource.php","line":122,"function":"get","class":"OC\\Files\\Node\\Folder","type":"->","args":["\/SharedPics"]},{"function":"OCA\\Recommendations\\Service\\{closure}","class":"OCA\\Recommendations\\Service\\RecentlySharedFilesSource","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/html\/nextcloud\/apps\/recommendations\/lib\/Service\/RecentlySharedFilesSource.php","line":127,"function":"array_map","args":[{"__class__":"Closure"},["*** sensitive parameter replaced ***",{"__class__":"OC\\Share20\\Share"},{"__class__":"OC\\Share20\\Share"},{"__class__":"OC\\Share20\\Share"},{"__class__":"OC\\Share20\\Share"},{"__class__":"OC\\Share20\\Share"}]]},{"file":"\/var\/www\/html\/nextcloud\/apps\/recommendations\/lib\/Service\/RecommendationService.php","line":91,"function":"getMostRecentRecommendation","class":"OCA\\Recommendations\\Service\\RecentlySharedFilesSource","type":"->","args":[{"__class__":"OC\\User\\User"},6]},{"function":"OCA\\Recommendations\\Service\\{closure}","class":"OCA\\Recommendations\\Service\\RecommendationService","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/html\/nextcloud\/apps\/recommendations\/lib\/Service\/RecommendationService.php","line":92,"function":"array_reduce","args":[[{"__class__":"OCA\\Recommendations\\Service\\RecentlyCommentedFilesSource"},{"__class__":"OCA\\Recommendations\\Service\\RecentlyEditedFilesSource"},"*** sensitive parameter replaced ***"],{"__class__":"Closure"},[]]},{"file":"\/var\/www\/html\/nextcloud\/apps\/recommendations\/lib\/Controller\/RecommendationController.php","line":63,"function":"getRecommendations","class":"OCA\\Recommendations\\Service\\RecommendationService","type":"->","args":[{"__class__":"OC\\User\\User"}]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/AppFramework\/Http\/Dispatcher.php","line":166,"function":"index","class":"OCA\\Recommendations\\Controller\\RecommendationController","type":"->","args":[]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/AppFramework\/Http\/Dispatcher.php","line":99,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OCA\\Recommendations\\Controller\\RecommendationController"},"index"]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/AppFramework\/App.php","line":126,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OCA\\Recommendations\\Controller\\RecommendationController"},"index"]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/AppFramework\/Routing\/RouteActionHandler.php","line":47,"function":"main","class":"OC\\AppFramework\\App","type":"::","args":["OCA\\Recommendations\\Controller\\RecommendationController","index",{"__class__":"OC\\AppFramework\\DependencyInjection\\DIContainer"},{"_route":"recommendations.recommendation.index"}]},{"function":"__invoke","class":"OC\\AppFramework\\Routing\\RouteActionHandler","type":"->","args":[{"_route":"recommendations.recommendation.index"}]},{"file":"\/var\/www\/html\/nextcloud\/lib\/private\/Route\/Router.php","line":297,"function":"call_user_func","args":[{"__class__":"OC\\AppFramework\\Routing\\RouteActionHandler"},{"_route":"recommendations.recommendation.index"}]},{"file":"\/var\/www\/html\/nextcloud\/lib\/base.php","line":975,"function":"match","class":"OC\\Route\\Router","type":"->","args":["\/apps\/recommendations\/api\/recommendations"]},{"file":"\/var\/www\/html\/nextcloud\/index.php","line":42,"function":"handleRequest","class":"OC","type":"::","args":[]}],"File":"\/var\/www\/html\/nextcloud\/lib\/private\/Files\/Node\/Root.php","Line":202,"CustomMessage":"--"},"userAgent":"Mozilla\/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko\/20100101 Firefox\/67.0","version":"16.0.1.1"}
The output of your config.php file in /path/to/nextcloud
(make sure you remove any identifiable information!):
<?php
$CONFIG = array (
'instanceid' => 'xxxxxx',
'passwordsalt' => 'xxxxxx',
'secret' => 'xxxxxx',
'trusted_domains' =>
array (
0 => '99.99.99.99,
1 => 'domain',
2 => 'www.domain',
),
'datadirectory' => '/var/www/path-to/nextcloud_data',
'dbtype' => 'mysql',
'version' => '16.0.1.1',
'overwrite.cli.url' => 'http://99.99.99.99/nextcloud',
'htaccess.RewriteBase' => '/',
'dbname' => 'nextcloud',
'dbhost' => 'localhost',
'dbport' => '',
'dbtableprefix' => 'oc_',
'dbuser' => 'nextcloud_user',
'dbpassword' => 'xxxxxx',
'installed' => true,
'maintenance' => false,
'mail_smtpmode' => 'smtp',
'mail_smtpsecure' => 'ssl',
'mail_sendmailmode' => 'smtp',
'mail_from_address' => 'nextcloud_admin',
'mail_domain' => 'domain',
'mail_smtpauthtype' => 'PLAIN',
'mail_smtphost' => 'mail.messagingengine.com',
'mail_smtpport' => '465',
'mail_smtpauth' => 1,
'mail_smtpname' => 'my_email',
'mail_smtppassword' => 'xxxxxx',
'memcache.local' => '\\OC\\Memcache\\APCu',
'updater.secret' => 'xxxxxx',
'theme' => '',
'loglevel' => 2,
);
The output of your Apache/nginx/system log in /var/log/____
:
Here is the tail of the log file right up to the moment NextCloud stopped working … what’s with the “Starting Clean php session” every 30 minutes? That pattern started after last night’s upgrade.
Jul 2 04:39:06 vultr systemd[1]: Starting Clean php session files...
Jul 2 04:39:06 vultr systemd[1]: Started Clean php session files.
Jul 2 05:09:00 vultr systemd[1]: Starting Clean php session files...
Jul 2 05:09:00 vultr systemd[1]: Started Clean php session files.
Jul 2 05:09:01 vultr CRON[29461]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jul 2 05:17:01 vultr CRON[29924]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 2 05:39:00 vultr systemd[1]: Starting Clean php session files...
Jul 2 05:39:00 vultr systemd[1]: Started Clean php session files.
Jul 2 05:39:01 vultr CRON[31493]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jul 2 06:09:01 vultr CRON[1347]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jul 2 06:09:06 vultr systemd[1]: Starting Clean php session files...
Jul 2 06:09:06 vultr systemd[1]: Started Clean php session files.
Jul 2 06:17:01 vultr CRON[1980]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 2 06:25:01 vultr CRON[2439]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Jul 2 06:25:02 vultr cracklib: no dictionary update necessary.
Jul 2 06:25:03 vultr systemd[1]: Reloading The Apache HTTP Server.
Jul 2 06:25:03 vultr systemd[1]: Reloaded The Apache HTTP Server.
And here’s the start of the final system log, started immediately after NC died.
Jul 2 06:25:03 vultr rsyslogd: [origin software="rsyslogd" swVersion="8.32.0" x-pid="501" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Jul 2 06:25:10 vultr popularity-contest: unable to submit report to https://popcon.ubuntu.com/popcon-submit.cgi.
Jul 2 06:25:10 vultr popularity-contest: unable to submit report.
Jul 2 06:32:06 vultr systemd[1]: Starting Daily apt upgrade and clean activities...
Jul 2 06:32:06 vultr systemd[1]: Started Daily apt upgrade and clean activities.
Jul 2 06:39:01 vultr CRON[2863]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jul 2 06:39:06 vultr systemd[1]: Starting Clean php session files...
Jul 2 06:39:06 vultr systemd[1]: Started Clean php session files.
Jul 2 07:09:01 vultr CRON[3175]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jul 2 07:09:02 vultr systemd[1]: Starting Clean php session files...
Jul 2 07:09:02 vultr systemd[1]: Started Clean php session files.
Jul 2 07:15:04 vultr systemd[1]: Created slice User Slice of root.
Jul 2 07:15:04 vultr systemd[1]: Starting User Manager for UID 0...
Jul 2 07:15:04 vultr systemd[1]: Started Session 7464 of user root.
Jul 2 07:15:04 vultr systemd[3333]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Jul 2 07:15:04 vultr systemd[3333]: Listening on GnuPG network certificate management daemon.
Jul 2 07:15:04 vultr systemd[3333]: Listening on GnuPG cryptographic agent and passphrase cache.
Jul 2 07:15:04 vultr systemd[3333]: Reached target Timers.
Jul 2 07:15:04 vultr systemd[3333]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers).
Jul 2 07:15:04 vultr systemd[3333]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Jul 2 07:15:04 vultr systemd[3333]: Reached target Paths.
Jul 2 07:15:04 vultr systemd[3333]: Reached target Sockets.
Jul 2 07:15:04 vultr systemd[3333]: Reached target Basic System.
Jul 2 07:15:04 vultr systemd[1]: Started User Manager for UID 0.
Jul 2 07:15:04 vultr systemd[3333]: Reached target Default.
Jul 2 07:15:04 vultr systemd[3333]: Startup finished in 27ms.
Jul 2 07:15:18 vultr systemd[1]: Stopping Availability of block devices...
Jul 2 07:15:18 vultr systemd[1]: Stopped target Timers.
Jul 2 07:15:18 vultr systemd[1]: Stopped Daily apt upgrade and clean activities.
Jul 2 07:15:18 vultr systemd[1]: Stopped Daily apt download activities.
Jul 2 07:15:18 vultr systemd[1]: Stopped Clean PHP session files every 30 mins.
Jul 2 07:15:18 vultr systemd[1]: Stopped Daily Cleanup of Temporary Directories.
Jul 2 07:15:18 vultr systemd[1]: Stopped Message of the Day.
Jul 2 07:15:18 vultr systemd[1]: Stopped Discard unused blocks once a week.
Jul 2 07:15:18 vultr systemd[1]: Stopped Run certbot twice daily.
Jul 2 07:15:18 vultr systemd[1]: Stopped target Graphical Interface.
Jul 2 07:15:18 vultr systemd[1]: Stopping Accounts Service...
Thanks to anyone actually willing to dig through this with me.