Cron.php no output and does not run background jobs. No debug logs either

Nextcloud version: 20.0.6
Operating system and version: Debian buster docker container
Apache or nginx version: nginx/1.19.6
PHP version: php-fpm 7.4.14

The issue you are facing:

Lately, maybe for the last month or so, cron jobs have been unstable and often show Last background job execution ran 3 hours ago. Something seems wrong. on the overview page. What’s very strange is that manually running php -f /var/www/html/cron.php as the www-data user in the container produces no output at all, and no debug logs (with occ log:manage --level=debug or with occ log:manage --level=info.

I also use preview generator which takes a very long time to run sometimes, but I have disabled that for 24 hours or so and even after server restarts I’m still having the same problem.
EDIT: This is in fact caused by preview:pre-generate somehow. See my follow up comment.

Once and a while, after taking down all containers and starting them again, I am able to get a few successful runs. After a while though cron.php stops working again and I start to see the error on the overview page again.

I did see this issue on github but it doesn’t seem to be the same problem I have. I don’t see cron.php running anywhere. It just exits a few seconds after running it with no output.

Is this the first time you’ve seen this error?: Y

Steps to replicate it:

  1. Have running instance
  2. Gain shell inside app (fpm) container
  3. Run php -f /var/www/html/cron.php
  4. See no output and overview does not show a successful background job run.
  5. Debug logs show nothing at all related to cron immediately following a manual execution of cron.php (lots of noise from deprecation warnings though related to this discussion )

The output of your Nextcloud log in Admin > Logging:
There really isn’t anything to put here. There are no debug logs from cron, and there is no output from running the job either.

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'apps_paths' => 
  array (
    0 => 
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 => 
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'enabledPreviewProviders' => 
  array (
    0 => 'OC\\Preview\\Movie',
    1 => 'OC\\Preview\\PNG',
    2 => 'OC\\Preview\\JPEG',
    3 => 'OC\\Preview\\GIF',
    4 => 'OC\\Preview\\BMP',
    5 => 'OC\\Preview\\XBitmap',
    6 => 'OC\\Preview\\MP3',
    7 => 'OC\\Preview\\TXT',
    8 => 'OC\\Preview\\MarkDown',
  ),
  'instanceid' => '<redacted>',  
  'passwordsalt' => '<redacted>',                    
  'secret' => '<redacted>',                                      
  'trusted_domains' => 
  array (
    0 => '<redacted>',          
  ),
  'datadirectory' => '/var/www/html/data',
  'overwrite.cli.url' => '<redacted>',                  
  'dbtype' => 'pgsql',
  'version' => '20.0.6.1',
  'dbname' => 'nextcloud',
  'dbhost' => 'db',
  'dbport' => '',
  'dbtableprefix' => '',
  'dbuser' => 'nextcloud',
  'dbpassword' => '<redacted>',               
  'installed' => true,
  'mail_smtpmode' => 'smtp',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpsecure' => 'ssl',
  'mail_from_address' => 'admin',
  'mail_domain' => '<redacted>',          
  'mail_smtpauth' => 1,
  'mail_smtphost' => '<redacted>',    
  'mail_smtpport' => '<redacted>',
  'mail_smtpname' => '<redacted>',          
  'mail_smtppassword' => '<redacted>',      
  'maintenance' => false,
  'loglevel' => 1,
  'theme' => '',
  'app_install_overwrite' => 
  array (
    0 => 'spreed',
    1 => 'gpxedit',
  ),
  'overwriteprotocol' => 'https',
);

The output of your Apache/nginx/system log in /var/log/____:
nginx does not log local php calls. FPM does not have any logging either (because it’s executed with php directly, not through the web)

1 Like

Followup:

I have determined that this is in fact due to running occ preview:pre-generate . If I enable both of the following jobs, cron.php eventually stops working. It runs and exits 0 but has no output and debug logs show no cron tasks were run.

php -f /var/www/html/cron.php
/var/www/html/occ preview:pre-generate

I have tried several things to get cron.php to run again but the sure way to make it work is to delete all my previews from /var/www/html/data/appdata_<string>/preview. Cron.php begins running instantly when those files are removed.

I have thousands of images across multiple users. It is the main thing that I use nextcloud for. I can’t help wondering if I’ve hit some preview limit or something.

Doing a restart of the instance (it is containerized) does not always fix the problem, even if I disable the preview cronjob.

Here are some other things I tried when I get into this stuck state, for the most part none of them seem to make a difference:

occ maintenance:repair
occ -vvv files:scan --all
occ -vvv preview:generate-all #this takes days sometimes

I have configured nextcloud according to the performance tuning guide (within reason, I’m running on a pi4b with attached ssd storage). I use redis for distributed caching and file locking.

I have also tried adjusting the preview dimensions and quality which drastically reduced the size of my preview directory. But did not change this behavior.

So after days of debugging I narrowed down the problem but also reached the limit of my abilities.

Turns out the debug log when viewed with occ log:watch is missing quite a bit of information. I eventually just tailed the file nextcloud.log which provided a lot more information.

I added some debug output to the cron.php script which printed information about each job and between the additional output and the newly discovered verbose debug logs it turns out that all the jobs are running and completing successfully. The debug log shows each run and finish in the logs. For example:

":"--","message":"Run OCA\\Support\\BackgroundJobs\\CheckSubscription job with ID 2836","userAgent":"--","version":"20.0.7.1"}
{"reqId":"<redacted>","level":0,"time":"2021-02-12T14:25:01+00:00","remoteAddr":"","user":"--","app":"cron","method":"","url":"--","message":"Finished OCA\\Support\\BackgroundJobs\\CheckSubscription job with ID 2836 in 0 seconds","userAgent":"--","version":"20.0.7.1"}

What is not working is the very last line of the cron.php $config->setAppValue('core', 'lastcron', time()); script which updates the core lastcron config value. I created a test script which only contains the following:

<?php

require_once __DIR__ . '/lib/versioncheck.php';
require_once __DIR__ . '/lib/base.php';

#$logger = \OC::$server->getLogger();
$config = \OC::$server->getConfig();

print(serialize($config) . PHP_EOL);

print(time() . PHP_EOL);
$config->setAppValue('core', 'lastcron', time());
exit();

I ran this directly from inside the app container, just like the cron.php script is run, and that value could not be updated. No logs are generated, the setAppValue just never happens, just like in the cron.php script.

At this point I made a guess that this might be some kind of bug or limitation related to postgresql. I realize this was a long jump to a conclusion, however I had been on the fence about converting to mariadb anyway so I decided to try that as a next step. It is the recommended DB after all.

After converting my database from postgresql to mariadb 10.1, updating the charset and upgrading to 10.2, preview generation and all the background jobs are now working as expected.

I’m going to mark this solved since I haven’t had any responses anyway, but if someone else comes across this and has more time to dig into it than me, there is possibly a bug in the postgresql implementation in nextcloud which only surfaces when there are thousands of previews. I am not able to test anymore though since I’ve left postgres behind.

Posting here because I had very similar symptoms with a totally different cause: In my case, it was because an update of the server changed vanilla “php” to run php 8:

$ php -v
PHP 8.0.5 (cli) (built: May  3 2021 11:30:57) ( NTS )
Copyright (c) The PHP Group
Zend Engine v4.0.5, Copyright (c) Zend Technologies
    with Zend OPcache v8.0.5, Copyright (c), by Zend Technologies

When I changed the cron job for user www-data to specifically request php7.4, the problem went away:

$ sudo crontab -u www-data -l
# m h  dom mon dow   command
*/5  *  *  *  * php7.4 -f /var/www/nextcloud/cron.php

I’m facing the same problem:

# time sudo -u www-data php -f /var/www/html/nextcloud/cron.php
real    0m0,107s
user    0m0,067s
sys     0m0,040s

Finishes too fast and doesn’t output anything.
https://my-domain/nextcloud/index.php/settings/admin/overview tells me

Last background job execution ran 3 days ago. Something seems wrong.

right after manually executing the command above.

PHP version?

# php -v
PHP 7.4.3 (cli) (built: Jul  5 2021 15:13:35) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.3, Copyright (c), by Zend Technologies

suggests to me that @gwhitney 's solution doesn’t work for me, sine I’m already on php7.4 by default.

Preview generaton?

I have never done anything with preview generation, in other words I’ve left it as it is by default.
Let’s try running few of those commands @chosoi7cuh wrote about:

AHA!

cd /var/www/html/nextcloud/
sudo -u www-data php -f occ
An unhandled exception has been thrown:
OC\HintException: [0]: Memcache \OC\Memcache\APCu not available for local cache (Is the matching PHP module installed and enabled?)

but:

# apt-get install php-apcu
Reading package lists... Done
Building dependency tree       
Reading state information... Done
php-apcu is already the newest version (5.1.18+4.0.11-1build1).

Ah, but searching for this error message I found this bug entry:

So I’m now gonna try to find the appropiate php.ini file and ensure it contains:

apc.enable_cli = 1

Done = solved for me too :)

cd /var/www/html/nextcloud/
time sudo -u www-data php -f cron.php 
real    2m55,749s
user    0m18,341s
sys     0m7,874s

I’m having the same issue on 3 different instances after upgrading to 21.0.4!
Very frustrating.

The logs confirm that the cron job is running, but it doesn’t seem to be doing anything at all.
I’m still working on the issue and will post my findings here to help others.

An interesting wrinkle in one instancee is this message in the overview:

* It was not possible to execute the cron job via CLI. The following technical errors have appeared:
  * Your data directory is not writable Permissions can usually be fixed by giving the webserver write access to the root directory. See https://docs.nextcloud.com/server/21/go.php?to=admin-dir_permissions.
* Last background job execution ran last month. Something seems wrong. [Check the background job settings](https://cloud.vascowhite.co.uk/settings/admin#backgroundjobs)

Has anybody seen this one before?

All permissions are as they should be and the data directory is, indeed, writable by the www-data user.

It turned out to be this issue:

I had fixed it for occ, but not for cron. My crontab entry now looks like this:

*/05 * * * * php --define apc.enable_cli=1 -f /var/www/nextcloud/cron.php > /dev/null 2>&1
2 Likes