Cron.php no output and does not run background jobs. No debug logs either

Nextcloud version: 20.0.6
Operating system and version: Debian buster docker container
Apache or nginx version: nginx/1.19.6
PHP version: php-fpm 7.4.14

The issue you are facing:

Lately, maybe for the last month or so, cron jobs have been unstable and often show Last background job execution ran 3 hours ago. Something seems wrong. on the overview page. What’s very strange is that manually running php -f /var/www/html/cron.php as the www-data user in the container produces no output at all, and no debug logs (with occ log:manage --level=debug or with occ log:manage --level=info.

I also use preview generator which takes a very long time to run sometimes, but I have disabled that for 24 hours or so and even after server restarts I’m still having the same problem.
EDIT: This is in fact caused by preview:pre-generate somehow. See my follow up comment.

Once and a while, after taking down all containers and starting them again, I am able to get a few successful runs. After a while though cron.php stops working again and I start to see the error on the overview page again.

I did see this issue on github but it doesn’t seem to be the same problem I have. I don’t see cron.php running anywhere. It just exits a few seconds after running it with no output.

Is this the first time you’ve seen this error?: Y

Steps to replicate it:

  1. Have running instance
  2. Gain shell inside app (fpm) container
  3. Run php -f /var/www/html/cron.php
  4. See no output and overview does not show a successful background job run.
  5. Debug logs show nothing at all related to cron immediately following a manual execution of cron.php (lots of noise from deprecation warnings though related to this discussion )

The output of your Nextcloud log in Admin > Logging:
There really isn’t anything to put here. There are no debug logs from cron, and there is no output from running the job either.

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'apps_paths' => 
  array (
    0 => 
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 => 
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'enabledPreviewProviders' => 
  array (
    0 => 'OC\\Preview\\Movie',
    1 => 'OC\\Preview\\PNG',
    2 => 'OC\\Preview\\JPEG',
    3 => 'OC\\Preview\\GIF',
    4 => 'OC\\Preview\\BMP',
    5 => 'OC\\Preview\\XBitmap',
    6 => 'OC\\Preview\\MP3',
    7 => 'OC\\Preview\\TXT',
    8 => 'OC\\Preview\\MarkDown',
  ),
  'instanceid' => '<redacted>',  
  'passwordsalt' => '<redacted>',                    
  'secret' => '<redacted>',                                      
  'trusted_domains' => 
  array (
    0 => '<redacted>',          
  ),
  'datadirectory' => '/var/www/html/data',
  'overwrite.cli.url' => '<redacted>',                  
  'dbtype' => 'pgsql',
  'version' => '20.0.6.1',
  'dbname' => 'nextcloud',
  'dbhost' => 'db',
  'dbport' => '',
  'dbtableprefix' => '',
  'dbuser' => 'nextcloud',
  'dbpassword' => '<redacted>',               
  'installed' => true,
  'mail_smtpmode' => 'smtp',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpsecure' => 'ssl',
  'mail_from_address' => 'admin',
  'mail_domain' => '<redacted>',          
  'mail_smtpauth' => 1,
  'mail_smtphost' => '<redacted>',    
  'mail_smtpport' => '<redacted>',
  'mail_smtpname' => '<redacted>',          
  'mail_smtppassword' => '<redacted>',      
  'maintenance' => false,
  'loglevel' => 1,
  'theme' => '',
  'app_install_overwrite' => 
  array (
    0 => 'spreed',
    1 => 'gpxedit',
  ),
  'overwriteprotocol' => 'https',
);

The output of your Apache/nginx/system log in /var/log/____:
nginx does not log local php calls. FPM does not have any logging either (because it’s executed with php directly, not through the web)

Followup:

I have determined that this is in fact due to running occ preview:pre-generate . If I enable both of the following jobs, cron.php eventually stops working. It runs and exits 0 but has no output and debug logs show no cron tasks were run.

php -f /var/www/html/cron.php
/var/www/html/occ preview:pre-generate

I have tried several things to get cron.php to run again but the sure way to make it work is to delete all my previews from /var/www/html/data/appdata_<string>/preview. Cron.php begins running instantly when those files are removed.

I have thousands of images across multiple users. It is the main thing that I use nextcloud for. I can’t help wondering if I’ve hit some preview limit or something.

Doing a restart of the instance (it is containerized) does not always fix the problem, even if I disable the preview cronjob.

Here are some other things I tried when I get into this stuck state, for the most part none of them seem to make a difference:

occ maintenance:repair
occ -vvv files:scan --all
occ -vvv preview:generate-all #this takes days sometimes

I have configured nextcloud according to the performance tuning guide (within reason, I’m running on a pi4b with attached ssd storage). I use redis for distributed caching and file locking.

I have also tried adjusting the preview dimensions and quality which drastically reduced the size of my preview directory. But did not change this behavior.

So after days of debugging I narrowed down the problem but also reached the limit of my abilities.

Turns out the debug log when viewed with occ log:watch is missing quite a bit of information. I eventually just tailed the file nextcloud.log which provided a lot more information.

I added some debug output to the cron.php script which printed information about each job and between the additional output and the newly discovered verbose debug logs it turns out that all the jobs are running and completing successfully. The debug log shows each run and finish in the logs. For example:

":"--","message":"Run OCA\\Support\\BackgroundJobs\\CheckSubscription job with ID 2836","userAgent":"--","version":"20.0.7.1"}
{"reqId":"<redacted>","level":0,"time":"2021-02-12T14:25:01+00:00","remoteAddr":"","user":"--","app":"cron","method":"","url":"--","message":"Finished OCA\\Support\\BackgroundJobs\\CheckSubscription job with ID 2836 in 0 seconds","userAgent":"--","version":"20.0.7.1"}

What is not working is the very last line of the cron.php $config->setAppValue('core', 'lastcron', time()); script which updates the core lastcron config value. I created a test script which only contains the following:

<?php

require_once __DIR__ . '/lib/versioncheck.php';
require_once __DIR__ . '/lib/base.php';

#$logger = \OC::$server->getLogger();
$config = \OC::$server->getConfig();

print(serialize($config) . PHP_EOL);

print(time() . PHP_EOL);
$config->setAppValue('core', 'lastcron', time());
exit();

I ran this directly from inside the app container, just like the cron.php script is run, and that value could not be updated. No logs are generated, the setAppValue just never happens, just like in the cron.php script.

At this point I made a guess that this might be some kind of bug or limitation related to postgresql. I realize this was a long jump to a conclusion, however I had been on the fence about converting to mariadb anyway so I decided to try that as a next step. It is the recommended DB after all.

After converting my database from postgresql to mariadb 10.1, updating the charset and upgrading to 10.2, preview generation and all the background jobs are now working as expected.

I’m going to mark this solved since I haven’t had any responses anyway, but if someone else comes across this and has more time to dig into it than me, there is possibly a bug in the postgresql implementation in nextcloud which only surfaces when there are thousands of previews. I am not able to test anymore though since I’ve left postgres behind.

Posting here because I had very similar symptoms with a totally different cause: In my case, it was because an update of the server changed vanilla “php” to run php 8:

$ php -v
PHP 8.0.5 (cli) (built: May  3 2021 11:30:57) ( NTS )
Copyright (c) The PHP Group
Zend Engine v4.0.5, Copyright (c) Zend Technologies
    with Zend OPcache v8.0.5, Copyright (c), by Zend Technologies

When I changed the cron job for user www-data to specifically request php7.4, the problem went away:

$ sudo crontab -u www-data -l
# m h  dom mon dow   command
*/5  *  *  *  * php7.4 -f /var/www/nextcloud/cron.php