Support intro
Sorry to hear you’re facing problems 
help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.
In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:
example
Or for longer, use three backticks above and below the code snippet:
longer
example
here
Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can 
Nextcloud version (eg, 20.0.5): 21.0.3
Operating system and version (eg, Ubuntu 20.04): Ubuntu 20.04.2 LTS (Focal Fossa)
Apache or nginx version (eg, Apache 2.4.25): lighttpd/1.4.55
PHP version (eg, 7.4): PHP 7.4.3
The issue you are facing:
Last background job execution ran 23 days ago. Something seems wrong. [Check the background job settings](https://nephele.thumbs.place/index.php/settings/admin#backgroundjobs)
Is this the first time you’ve seen this error? (Y/N): Sort of, but it’s been persistent for a long while clearly and been coming and going methinks. Not 100% sure as I don’t check the overview or logs routinely only when something prompts me to.
Steps to replicate it:
- Just surf to: /index.php/settings/admin/overview and voila I see this message.
- Not a reproducible problem mind you, an issue on a specific server to resolve. Only key issue is it’s been running a year or more without this issue. Which may have started with an upgrade as I did upgrades as soon as possible as a rule to keep up with security.
The output of your Nextcloud log in Admin > Logging:
This is a tad verbose, and none of it relates I think, but indicates another issue I need tor solve clearly (those fatal errors complaining about a database column that does not exist - though possible if scheduled maintenance isn’t working that may relate). I have taken lines back a bit from the last message I could go on and on, but there’s plenty of meat in this already to try and understand.
The content of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):
Not sure what is “identifiable” information so I take a liberal redacting hand to it:
$ cat /var/www/html/nextcloud/config/config.php
<?php
$CONFIG = array (
'instanceid' => '<redacted: id>',
'passwordsalt' => '<redacted: salt>',
'secret' => '<redacted: secret>',
'trusted_domains' =>
array (
0 => '<redacted: domain1>',
1 => '<redacted: domain2>',
2 => '<redacted: domain3>',
3 => '<redacted: domain4>',
4 => '192.168.0.14',
5 => '127.0.0.1',
),
'datadirectory' => '/data/nextcloud',
'dbtype' => 'pgsql',
'version' => '21.0.3.1',
'dbname' => 'nextclouddb',
'dbhost' => 'localhost:5432',
'dbport' => '',
'dbtableprefix' => 'oc_',
'dbuser' => 'nextcloud',
'dbpassword' => 'h7jS5naxCARFgv',
'installed' => true,
'logtimezone' => 'Australia/Hobart',
'loglevel' => '2',
'trusted_proxies' =>
array (
0 => '192.168.0.1',
),
'forwarded-for-headers' =>
array (
0 => 'HTTP_X_FORWARDED_FOR',
),
'memcache.local' => '\\OC\\Memcache\\APCu',
'app_install_overwrite' =>
array (
0 => 'joplin',
),
'maintenance' => false,
'overwriteprotocol' => 'https',
'overwrite.cli.url' => '<redacted: URL>',
'theme' => '',
'updater.release.channel' => 'stable',
'mail_from_address' => '<redacted: user>',
'mail_smtpmode' => 'smtp',
'mail_sendmailmode' => 'smtp',
'mail_domain' => '<redacted: domain>',
'mail_smtpauthtype' => 'LOGIN',
'mail_smtpauth' => 1,
'mail_smtphost' => '<redacted: mailhost>',
'mail_smtpport' => '465',
'mail_smtpsecure' => 'ssl',
'mail_smtpname' => '<redacted: username>',
'mail_smtppassword' => '<redacted: password>',
'htaccess.RewriteBase' => '/',
'default_phone_region' => 'AU',
'updater.secret' => '<redacted: secret>',
The output of your Apache/nginx/system log in /var/log/____:
Oh how I love these Apache/nginx assumptions. Real performant FOSS emplys lighttpd (as a complete aside nginx and lighttpd perform on par across all benchmarks and kick Apache’s ass in performance, and resource footprints, but lighttpd is FOSS and nginx isn’t).
A tail on the error log reveals no issues bar an attempted connection for localhost port 9980. I keep a rolling access log as well, but it is noisy as the server is mildly busy and working (though slow as a dog this last week - I use uptime robot to watch servers and this last week this server has been timing out a lot - though I cannot catch it at all stressed with glances, CPU, memory, traffic, swapping etc all totally fine, and box seems performant at the CLI)
$ tail -15 /data/log/lighttpd/error.log
2021-07-17 02:16:55: (server.c.1970) server stopped by UID = 0 PID = 1
2021-07-17 02:16:55: (server.c.1488) server started (lighttpd/1.4.55)
2021-07-17 02:16:55: (server.c.1970) server stopped by UID = 0 PID = 1
2021-07-17 02:16:56: (server.c.1488) server started (lighttpd/1.4.55)
2021-07-29 20:50:36: (gw_backend.c.236) establishing connection failed: Connection refused socket: tcp:127.0.0.1:9980
2021-07-29 20:50:36: (gw_backend.c.939) all handlers for /hosting/capabilities? on are down.
2021-07-29 20:50:38: (gw_backend.c.315) gw-server re-enabled: tcp:127.0.0.1:9980 127.0.0.1 9980
2021-07-29 20:50:46: (server.c.1970) server stopped by UID = 0 PID = 1
2021-07-29 20:50:47: (server.c.1488) server started (lighttpd/1.4.55)
2021-07-29 22:52:46: (server.c.1970) server stopped by UID = 0 PID = 1
2021-07-29 22:53:53: (server.c.1488) server started (lighttpd/1.4.55)
2021-07-31 07:58:09: (mod_openssl.c.1762) SSL: -1 5 104 Connection reset by peer
2021-07-31 08:02:43: (server.c.936) [note] graceful shutdown started
2021-07-31 08:03:41: (server.c.1970) server stopped by UID = 0 PID = 55730
2021-07-31 08:03:41: (server.c.1488) server started (lighttpd/1.4.55)
The real issue I have is that I can’t diagnose this easily any further. For example:
$ sudo -u www-data ps ux
[sudo] password for cirrus:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
www-data 50672 5.2 0.5 375476 90200 ? S 05:32 16:29 php-fpm: pool www
www-data 52051 5.2 0.5 301372 89648 ? S 06:24 13:51 php-fpm: pool www
www-data 53194 5.4 0.5 375616 91688 ? S 06:44 13:14 php-fpm: pool www
www-data 54484 5.4 0.5 375452 91432 ? S 07:34 10:25 php-fpm: pool www
www-data 56604 0.0 0.0 14516 11992 ? Ss 08:03 0:05 /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
www-data 56610 5.0 0.5 375380 90540 ? S 08:03 8:16 php-fpm: pool www
www-data 61800 0.0 0.0 6164 3128 pts/0 R+ 10:46 0:00 ps ux
then:
$ sudo -u www-data crontab -u www-data -l
# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').
#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h dom mon dow command
*/5 * * * * php -f /var/www/html/nextcloud/cron.php
then:
$ ls -l /var/www/html/nextcloud/cron.php
-rw-r--r-- 1 www-data www-data 5122 Jul 7 14:05 /var/www/html/nextcloud/cron.php
then:
$ sudo -u www-data php -f /var/www/html/nextcloud/cron.php
Yes the total lack of any output here isn’t useful. So a check to see what’s in it:
$ cat /var/www/html/nextcloud/cron.php
<?php
/**
* @copyright Copyright (c) 2016, ownCloud, Inc.
*
* @author Artem Sidorenko <artem@posteo.de>
* @author Christopher Schäpers <kondou@ts.unde.re>
* @author Christoph Wurst <christoph@winzerhof-wurst.at>
* @author Daniel Kesselberg <mail@danielkesselberg.de>
* @author hoellen <dev@hoellen.eu>
* @author J0WI <J0WI@users.noreply.github.com>
* @author Jakob Sack <mail@jakobsack.de>
* @author Joas Schilling <coding@schilljs.com>
* @author Jörn Friedrich Dreyer <jfd@butonic.de>
* @author Ko- <k.stoffelen@cs.ru.nl>
* @author Michael Kuhn <michael@ikkoku.de>
* @author Morris Jobke <hey@morrisjobke.de>
* @author Oliver Kohl D.Sc. <oliver@kohl.bz>
* @author Robin Appelman <robin@icewind.nl>
* @author Roeland Jago Douma <roeland@famdouma.nl>
* @author Steffen Lindner <mail@steffen-lindner.de>
* @author Thomas Müller <thomas.mueller@tmit.eu>
* @author Vincent Petry <vincent@nextcloud.com>
*
* @license AGPL-3.0
*
* This code is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License, version 3,
* as published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License, version 3,
* along with this program. If not, see <http://www.gnu.org/licenses/>
*
*/
require_once __DIR__ . '/lib/versioncheck.php';
try {
require_once __DIR__ . '/lib/base.php';
if (\OCP\Util::needUpgrade()) {
\OC::$server->getLogger()->debug('Update required, skipping cron', ['app' => 'cron']);
exit;
}
if ((bool) \OC::$server->getSystemConfig()->getValue('maintenance', false)) {
\OC::$server->getLogger()->debug('We are in maintenance mode, skipping cron', ['app' => 'cron']);
exit;
}
// load all apps to get all api routes properly setup
OC_App::loadApps();
\OC::$server->getSession()->close();
// initialize a dummy memory session
$session = new \OC\Session\Memory('');
$cryptoWrapper = \OC::$server->getSessionCryptoWrapper();
$session = $cryptoWrapper->wrapSession($session);
\OC::$server->setSession($session);
$logger = \OC::$server->getLogger();
$config = \OC::$server->getConfig();
// Don't do anything if Nextcloud has not been installed
if (!$config->getSystemValue('installed', false)) {
exit(0);
}
\OC::$server->getTempManager()->cleanOld();
// Exit if background jobs are disabled!
$appMode = $config->getAppValue('core', 'backgroundjobs_mode', 'ajax');
if ($appMode === 'none') {
if (OC::$CLI) {
echo 'Background Jobs are disabled!' . PHP_EOL;
} else {
OC_JSON::error(['data' => ['message' => 'Background jobs disabled!']]);
}
exit(1);
}
if (OC::$CLI) {
// set to run indefinitely if needed
if (strpos(@ini_get('disable_functions'), 'set_time_limit') === false) {
@set_time_limit(0);
}
// the cron job must be executed with the right user
if (!function_exists('posix_getuid')) {
echo "The posix extensions are required - see https://www.php.net/manual/en/book.posix.php" . PHP_EOL;
exit(1);
}
$user = posix_getuid();
$configUser = fileowner(OC::$configDir . 'config.php');
if ($user !== $configUser) {
echo "Console has to be executed with the user that owns the file config/config.php" . PHP_EOL;
echo "Current user id: " . $user . PHP_EOL;
echo "Owner id of config.php: " . $configUser . PHP_EOL;
exit(1);
}
// We call Nextcloud from the CLI (aka cron)
if ($appMode !== 'cron') {
$config->setAppValue('core', 'backgroundjobs_mode', 'cron');
}
// Work
$jobList = \OC::$server->getJobList();
// We only ask for jobs for 14 minutes, because after 5 minutes the next
// system cron task should spawn and we want to have at most three
// cron jobs running in parallel.
$endTime = time() + 14 * 60;
$executedJobs = [];
while ($job = $jobList->getNext()) {
if (isset($executedJobs[$job->getId()])) {
$jobList->unlockJob($job);
break;
}
$job->execute($jobList, $logger);
// clean up after unclean jobs
\OC_Util::tearDownFS();
$jobList->setLastJob($job);
$executedJobs[$job->getId()] = true;
unset($job);
if (time() > $endTime) {
break;
}
}
} else {
// We call cron.php from some website
if ($appMode === 'cron') {
// Cron is cron :-P
OC_JSON::error(['data' => ['message' => 'Backgroundjobs are using system cron!']]);
} else {
// Work and success :-)
$jobList = \OC::$server->getJobList();
$job = $jobList->getNext();
if ($job != null) {
$job->execute($jobList, $logger);
$jobList->setLastJob($job);
}
OC_JSON::success();
}
}
// Log the successful cron execution
$config->setAppValue('core', 'lastcron', time());
exit();
And now I’m in deeper than immediate skills go, and I’d need to look into instrumenting this to trace its run and make sure for exampl this $config->setAppValue('core', 'lastcron', time()); runs etc. (as that seems to be how this script tells the rest of the NextCloud that it ran (dispelling that warning on the overview page).
