Background jobs unable to run via cron

strongthany · March 3, 2022, 6:43pm

Nextcloud version: 23.0.2
Operating system and version: 20.04.4 LTS
Apache or nginx version: 2.4.41
PHP version: 8.0.16

The issue you are facing: background jobs timers don’t function. The alert for last time a job was ran does not update, despite having configured a cron job.

When I try to run the command configured in the cron job in the cli, I get the following output.

root@cloud:/var/www/html/nextcloud/config# sudo -u www-data php -f /var/www/html/nextcloud/cron.php
PHP Warning:  Module "pdo_pgsql" is already loaded in Unknown on line 0
OCP\HintException: [0]: Memcache \OC\Memcache\APCu not available for local cache (Is the matching PHP module installed and enabled?)

Is there a module I need to activate in apache to resolve this? Or some setting or step I missed here that would take care of this?

Is this the first time you’ve seen this error? : Yes

Steps to replicate it:

Edit crontab with crontab -u www-data -e
Add line to the end of file */5 * * * * php -f /var/www/nextcloud/cron.php
enable cron with systemctl enable cron

The output of your Nextcloud log in Admin > Logging:
The full contents of the logs can be found here

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'trusted_proxies'   => ['10.10.10.4'],
  'overwriteprotocol' => 'https',
  'instanceid' => 'ocesm91tpsbv',
  'passwordsalt' => 'Aep0XHlurwzd4Kmp0LOM5bPsg+w1t8',
  'secret' => 'PA6qc5EqyOmnLQ3B0t+p7ydV7FvZE9zqD8IwJM6x+ai+0uvS',
  'trusted_domains' => 
  array (
          0 => 'cloud.domain.tld',
          1 => '10.10.10.243',
  ),
  'enable_previews' => true,
  'preview_libreoffice_path' => '/usr/bin/libreoffice',
  'enabledPreviewProviders' =>
   array (
      0 => 'OC\\Preview\\TXT',
      1 => 'OC\\Preview\\MarkDown',
      2 => 'OC\\Preview\\OpenDocument',
      3 => 'OC\\Preview\\PDF',
      4 => 'OC\\Preview\\MSOffice2003',
      5 => 'OC\\Preview\\MSOfficeDoc',
      6 => 'OC\\Preview\\PDF',
      7 => 'OC\\Preview\\Image',
      8 => 'OC\\Preview\\Photoshop',
      9 => 'OC\\Preview\\TIFF',
     10 => 'OC\\Preview\\SVG',
     11 => 'OC\\Preview\\Font',
     12 => 'OC\\Preview\\MP3',
     13 => 'OC\\Preview\\Movie',
     14 => 'OC\\Preview\\MKV',
     15 => 'OC\\Preview\\MP4',
     16 => 'OC\\Preview\\AVI',
   ),
  'datadirectory' => '/mnt/database/nextcloud/data',
  'dbtype' => 'pgsql',
  'version' => '23.0.2.1',
  'overwrite.cli.url' => 'http://cloud.domain.tld/',
  'htaccess.RewriteBase' => '/',
  'dbname' => 'nextclouddb',
  'dbhost' => '10.10.10.5:5432',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'nextcloud',
  'dbpassword' => 'Quickstep-Denture0-Pried',
  'installed' => true,
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'redis' => 
  array (
    'host' => '/run/redis/redis-server.sock',
    'port' => 0,
    'dbindex' => 0,
    'password' => 'secret',
    'timeout' => 1.5,
  ),
  'default_phone_region' => 'US',
  'mail_smtpmode' => 'smtp',
  'mail_smtpsecure' => 'ssl',
  'mail_sendmailmode' => 'smtp',
  'mail_from_address' => 'noreply',
  'mail_domain' => 'notifications.domain.tld',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpauth' => 1,
  'mail_smtphost' => 'smtp.mailhost.tld',
  'mail_smtpport' => '465',
  'mail_smtpname' => 'noreply@notifications.domain.tld',
  'mail_smtppassword' => 'shreklich_mail_transmission_protocol',
  'twofactor_enforced' => 'true',
  'twofactor_enforced_groups' => 
  array (
  ),
  'twofactor_enforced_excluded_groups' => 
  array (
  ),
);

The output of your Apache/nginx/system log in /var/log/____:

[Thu Mar 03 00:00:01.309626 2022] [mpm_event:notice] [pid 1645:tid 140025513307200] AH00489: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1f configured -- resuming normal operations
[Thu Mar 03 00:00:01.309646 2022] [core:notice] [pid 1645:tid 140025513307200] AH00094: Command line: '/usr/sbin/apache2'
[Thu Mar 03 18:18:10.252704 2022] [mpm_event:notice] [pid 1645:tid 140025513307200] AH00491: caught SIGTERM, shutting down
[Thu Mar 03 18:18:14.378208 2022] [mpm_event:notice] [pid 188:tid 140462151593024] AH00489: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1f configured -- resuming normal operations
[Thu Mar 03 18:18:14.378491 2022] [core:notice] [pid 188:tid 140462151593024] AH00094: Command line: '/usr/sbin/apache2'

tflidd · March 4, 2022, 9:30am

Checkout the orange-brown box in the documentation:
https://docs.nextcloud.com/server/23/admin_manual/configuration_server/caching_configuration.html

strongthany · March 4, 2022, 8:39pm

Thank you! I tried this but still am having the same issue. If I try running that change for the cron config I get this:

root@cloud:~# sudo -u www-data php -f /var/www/nextcloud/cron.php --define apc.enable_cli=1
PHP Warning:  Module "pdo_pgsql" is already loaded in Unknown on line 0
Could not open input file: /var/www/nextcloud/cron.php

Is there a php module I’m missing?

tflidd · March 4, 2022, 9:48pm

In your first post, you wrote that you installed nextcloud here:

/var/www/html/nextcloud

strongthany · March 4, 2022, 9:56pm

Correct, I did install NC to var/www/html/nextcloud, but I did not notice the issue in the command until you pointed it out there.

If i try running the command with the correct path, I get this:

root@cloud:~# sudo -u www-data php -f /var/www/html/nextcloud/cron.php --define apc.enable_cli=1
PHP Warning:  Module "pdo_pgsql" is already loaded in Unknown on line 0

It then stays there and doesn’t finish loading. Is this to be expected? should I wait longer?

tflidd · March 4, 2022, 10:11pm

If it has been a long time, the cronjob can take same time. Open a second console and check if there is some activity.

Hendrik · March 6, 2022, 12:32am

Try something like this:
/usr/local/php74/bin/php -f /home/user/domains/domain/public_html/cloud/cron.php

So, do explicit include the php version. I presume you create the cron job for a specific user. So do not run this as root, but as the user!

strongthany · March 7, 2022, 9:54pm

It did eventually finish, which I imagine it took a while to do because it’s been a long time since the automated jobs were able to run.

As a side note, the command Hendrik posted does not work as I don’t have that php version in my path. I am running php version 8.0, with a couple of 8.1 modules installed too, thought I didn’t do that and don’t know where they came from.

tflidd · March 8, 2022, 12:47pm

Now it is quick and you can run it on a regular basis (topic solved)?

strongthany · March 8, 2022, 3:46pm

Yes I would say topic solved. Thank you all for the help!

Hendrik · March 8, 2022, 4:45pm

With php8 I get errors too. But if you are with DirectAdmin, you can set php8.0 or 8.1 if installed. It will use the modules that belong to the selected version.
After deleting the ACPu line in the config, Nextcloud login started again. But with errors.

‘The process control (PCNTL) extensions are required’ is for the cron jobs. That older php module was not installed - see https://www.php.net/manual/en/book.pcntl.php. After installing Error while executing cron
/usr/local/php80/bin/php -f /home/username/domains/domain.eu/public_html/cloud/occ dav:send-event-reminders
The current PHP memory limit is below the recommended value of 512MB.’ Even when this is set to 1G and that is what System tab within NC confirms to be set to 1GB. So where does this memory counting come from?

And
‘memcache.local’ => ‘\OC\Memcache\APCu’,
should be
‘memcache.local’ => ‘\OC\Memcache\redis’,
if Redis is used. APCu is no longer supported in php8 as I have found out. So documentation should be updated to be in line with the php recommodation of Nextcloud.

bb77 · March 8, 2022, 5:17pm

That’s not true. APCu works fine with PHP8.0. Just add the following line to /etc/php/8.0/mods-available/apcu.ini

apc.enable_cli=1

Or you can add the following parameter to your cronjob:

--define apc.enable_cli=1

This is also highlighted in the Nextcloud documentation:

https://docs.nextcloud.com/server/stable/admin_manual/configuration_server/caching_configuration.html?#id1

strongthany · March 8, 2022, 5:31pm

Is there a performance difference between the two, APCU and Redis?

bb77 · March 8, 2022, 5:41pm

I am not a expert. But if you have enough RAM and can allocate enough memory to PHP, it can be faster than Redis. Especially in typical home user scenarios where only a few PHP processes are running and always from the same users and applications. If you have a larger setup with many users and or multiple applications, Redis is the way to go because it scales way better than APCu. You will still need more RAM then, but probably not as much more as if you were still using APCu.

Also APCu can only do local memory caching and no distributed caching or transactional file locking.

Hendrik · March 9, 2022, 12:38am

I did that --define. But SSH does not work normal with occ or I get an error warning in Nextcloud.
Perhaps APCu discussion I read was wrong. So I can try again. But ACPu, opcache and Redis as caching together?
And suddenly also php asks for the PCNTL module to be installed. And a lot of more trouble with opcache also. And serveral tables need again to be created with occ. And the cron job for occ dav:send-event-reminders needed to be altered by adding memory_limit. The one in php.ini is not used by the cli.

NC 23.0.2 is the most undocumented update so-far. Our server is up-to-date. I had no warnings before. So if NC needs other settings or php modules as before, I’d expect them to check that before installing the update and worn me to do some things as installing extra mods and change settings. And still working to get rid of opache messages that keep comming.

bb77 · March 9, 2022, 6:52am

Sorry but I don’t know what any of this has to do with SSH.

They can be used for diffrent things. You can use APCu for local memory caching and Redis for transactional file locking.

My config.php looks like this:

'memcache.local' => '\OC\Memcache\APCu',
'memcache.locking' => '\OC\Memcache\Redis',
'redis' =>
array (
'host' => '/var/run/redis/redis-server.sock',
'port' => 0,
'timeout' => 1.5,
),

OPcache on the otherhand is used for caching precompiled bytecode: Server tuning — Nextcloud latest Administration Manual latest documentation

Sorry I don’t know what exactley causes this. Never had this issue.

Not sure about this one either. But this are my php.ini configs reagrding Resource Limits:

/etc/php/8.0/cli/php.ini

Click here to view details

;;;;;;;;;;;;;;;;;;;
; Resource Limits ;
;;;;;;;;;;;;;;;;;;;

; Maximum execution time of each script, in seconds
; http://php.net/max-execution-time
; Note: This directive is hardcoded to 0 for the CLI SAPI
max_execution_time = 3600

; Maximum amount of time each script may spend parsing request data. It's a good
; idea to limit this time on productions servers in order to eliminate unexpectedly
; long running scripts.
; Note: This directive is hardcoded to -1 for the CLI SAPI
; Default Value: -1 (Unlimited)
; Development Value: 60 (60 seconds)
; Production Value: 60 (60 seconds)
; http://php.net/max-input-time
max_input_time = 3600

; Maximum input variable nesting level
; http://php.net/max-input-nesting-level
;max_input_nesting_level = 64

; How many GET/POST/COOKIE input variables may be accepted
;max_input_vars = 1000

; Maximum amount of memory a script may consume
; http://php.net/memory-limit
memory_limit = -1

/etc/php/8.0/fpm/php.ini

Click here to view details

;;;;;;;;;;;;;;;;;;;
; Resource Limits ;
;;;;;;;;;;;;;;;;;;;

; Maximum execution time of each script, in seconds
; http://php.net/max-execution-time
; Note: This directive is hardcoded to 0 for the CLI SAPI
max_execution_time = 3600

; Maximum amount of time each script may spend parsing request data. It's a good
; idea to limit this time on productions servers in order to eliminate unexpectedly
; long running scripts.
; Note: This directive is hardcoded to -1 for the CLI SAPI
; Default Value: -1 (Unlimited)
; Development Value: 60 (60 seconds)
; Production Value: 60 (60 seconds)
; http://php.net/max-input-time
max_input_time = 3600

; Maximum input variable nesting level
; http://php.net/max-input-nesting-level
;max_input_nesting_level = 64

; How many GET/POST/COOKIE input variables may be accepted
;max_input_vars = 1000

; Maximum amount of memory a script may consume
; http://php.net/memory-limit
memory_limit = 2048M

rellek · March 14, 2023, 5:50pm

This was the trick for me. Cron would not run without adding the --define param into the crontab

php --define apc.enable_cli=1 /var/www/nextcloud***/cron.php