Nextcloudcron.service stops working in ~12 hours

ℹ️ Support
,
1

Hi! I have setup systemd nextcloudcron.service to run cron.php job each 5 mins according to the official documentation with some little tuning with the idea to move from generic cron job executed via crontab.

Strangely, but it stops working in ~12 hours without clear understanding why. The trigger state in nextcoudcron.timer becames just “n/a”.

Units:

#cat /etc/systemd/system/nextcloudcron.service
[Unit]
Description=Nextcloud cron.php job
#Requires=apache2.service

[Service]
User=www-data
Nice=19
IOSchedulingClass=2
IOSchedulingPriority=7
Restart=on-failure
ExecStartPre=/bin/bash -c '/usr/bin/systemctl is-active --quiet apache2.service'
ExecCondition=/usr/bin/php -f /var/www/nextcloud/occ status -e
ExecStart=/usr/bin/php -f /var/www/nextcloud/cron.php
KillMode=process

# cat /etc/systemd/system nextcloudcron.timer
[Unit]
Description=Run Nextcloud cron.php every 5 minute

[Timer]
#OnCalendar=*:0/5:00
OnBootSec=5min
OnUnitActiveSec=5min
Unit=nextcloudcron.service

[Install]
WantedBy=timers.target

In logs I see:

 # cat /var/log/syslog | grep -n2 nextcloudcron
Mar 14 07:37:19 a3 systemd[1]: Starting Nextcloud cron.php job...
Mar 14 07:37:20 a3 systemd[1]: Started Nextcloud cron.php job.
Mar 14 07:37:25 a3 systemd[1]: nextcloudcron.service: Deactivated successfully.
Mar 14 07:37:25 a3 systemd[1]: nextcloudcron.service: Consumed 3.435s CPU time.
Mar 14 07:39:01 a3 CRON[939140]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Mar 14 07:39:01 a3 systemd[1]: Starting Clean php session files...
Mar 14 07:39:02 a3 systemd[1]: phpsessionclean.service: Deactivated successfully.
Mar 14 07:39:02 a3 systemd[1]: Finished Clean php session files.
Mar 14 07:40:01 a3 CRON[939410]: (smmsp) CMD (test -x /etc/init.d/sendmail && test -x /usr/share/sendmail/sendmail && test -x /usr/libexec/sendmail/sendmail && /usr/share/sendmail/sendmail cron-msp)
Mar 14 07:42:19 a3 systemd[1]: Starting Nextcloud cron.php job...
Mar 14 07:42:20 a3 systemd[1]: Started Nextcloud cron.php job.
Mar 14 07:42:23 a3 systemd[1]: nextcloudcron.service: Deactivated successfully.
Mar 14 07:42:23 a3 systemd[1]: nextcloudcron.service: Consumed 2.240s CPU time.
Mar 14 07:45:01 a3 CRON[940182]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Mar 14 07:45:01 a3 systemd[1]: Starting Nextcloud Preview Generator...
Mar 14 07:45:02 a3 systemd[1]: Started Nextcloud Preview Generator.
Mar 14 07:45:03 a3 systemd[1]: nextcloud-preview-generator.service: Deactivated successfully.
Mar 14 07:45:03 a3 systemd[1]: nextcloud-preview-generator.service: Consumed 1.781s CPU time.
Mar 14 07:46:42 a3 anacron[938018]: Normal exit (1 job run)
Mar 14 07:46:42 a3 systemd[1]: anacron.service: Deactivated successfully.
Mar 14 07:47:19 a3 systemd[1]: Starting Nextcloud cron.php job...
Mar 14 07:47:20 a3 systemd[1]: Started Nextcloud cron.php job.
Mar 14 07:47:23 a3 systemd[1]: nextcloudcron.service: Deactivated successfully.
Mar 14 07:47:23 a3 systemd[1]: nextcloudcron.service: Consumed 2.134s CPU time.
Mar 14 07:52:19 a3 systemd[1]: Starting Nextcloud cron.php job...
Mar 14 07:52:20 a3 systemd[1]: Started Nextcloud cron.php job.
Mar 14 07:55:01 a3 CRON[941702]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Mar 14 08:00:01 a3 systemd[1]: Starting Nextcloud Preview Generator...
Mar 14 08:00:01 a3 CRON[942459]: (smmsp) CMD (test -x /etc/init.d/sendmail && test -x /usr/share/sendmail/sendmail && test -x /usr/libexec/sendmail/sendmail && /usr/share/sendmail/sendmail cron-msp)
Mar 14 08:00:02 a3 systemd[1]: Started Nextcloud Preview Generator.
Mar 14 08:00:04 a3 systemd[1]: nextcloud-preview-generator.service: Deactivated successfully.
Mar 14 08:00:04 a3 systemd[1]: nextcloud-preview-generator.service: Consumed 1.842s CPU time.
Mar 14 08:02:01 a3 CRON[942830]: (logcheck) CMD (   if [ -x /usr/sbin/logcheck ]; then nice -n10 /usr/sbin/logcheck; fi)
Mar 14 08:09:01 a3 systemd[1]: Starting Clean php session files...
Mar 14 08:09:01 a3 CRON[944363]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Mar 14 08:09:03 a3 systemd[1]: phpsessionclean.service: Deactivated successfully.
Mar 14 08:09:03 a3 systemd[1]: Finished Clean php session files.

Filtering by service gives more clear report:

$ journalctl --since "2024-03-14 07:40:00" --until "2024-03-14 08:05:00" -u nextcloudcron.service
Mar 14 07:42:19 my.host systemd[1]: Starting Nextcloud cron.php job...
Mar 14 07:42:20 my.host systemd[1]: Started Nextcloud cron.php job.
Mar 14 07:42:23 my.host systemd[1]: nextcloudcron.service: Deactivated successfully.
Mar 14 07:42:23 my.host systemd[1]: nextcloudcron.service: Consumed 2.240s CPU time.
Mar 14 07:47:19 my.host systemd[1]: Starting Nextcloud cron.php job...
Mar 14 07:47:20 my.host systemd[1]: Started Nextcloud cron.php job.
Mar 14 07:47:23 my.host systemd[1]: nextcloudcron.service: Deactivated successfully.
Mar 14 07:47:23 my.host systemd[1]: nextcloudcron.service: Consumed 2.134s CPU time.
Mar 14 07:52:19 my.host systemd[1]: Starting Nextcloud cron.php job...
Mar 14 07:52:20 my.host systemd[1]: Started Nextcloud cron.php job.

So, somehow after 07:52 nextcloudcron.service stops working, while the other services (eg preview generator run each 15 mins) are continuing to work well.

I tried to play with OnCalendar option, added Restart=on-failure. It doesn’t help anyway.

Why I am using systemd instead of cron job, as I would like to stop executing cron.php job when backup process is run.

Steps to reproduce

  1. Create unit files for running cron.php job each 5 mins.
  2. Start nextcloudcron.timer.
  3. Observe that it is working and trigger runs the job each 5 mins.
  4. Check in ~12 hours and observe that the state of the trigger is n/a.
2

I’m not sure what exactly is causing this, but here are a few ideas of what you could try/check…

Have you ever tried without your tuning? Maybe it has something to do with the scheduling/priority settings you added or with the ExecStartPre option that checks whether apache is running.

Or maybe some app you have installed causes the job to fail, like in the following thread… Background cron jobs failing at regular intervals after v. 25.0.1 - #2 by n0plan

Also, if you’re using APCu as a memory cache, make sure that APCu is enabled for the CLI, otherwise some scripts like the ones in the cron.php tend to fail. See here and here.

3

Hi!

Yes, I tried without tuning and it is not helpful, that is why I tried to tune it up.

Yes, I use apcu and it is set up and enabled in php.ini (both apache and cli folders).

Cannon understand the nature of the issue. As a workaround I created the crontab task systemctl stop nextcloudcron.service run each 6 hours what is awkward.

At least it works.

4

So, trying to understand what may wrong I started to inspect the processes and found that there’s unfinished php process to run cron.php!

So, the question how to kill the process more actively if it is stuck?

Additionally I found a comment that the nextcloudcron.service may collide with nextcloud-preview-generator.service when both are run in the same time using php.

So, what I’ve done I’ve added the condition to start nextcloudcron.service when nextcloud-preview-generator.service is not active.

ExecStartPre=/bin/bash -c '! /usr/bin/systemctl is-active --quiet nextcloud-preview-generator.service'

Finally, my version of the nextcloudcron.service:

[Unit]
Description=Nextcloud cron.php job

[Service]
Type=simple
User=www-data
Nice=19
IOSchedulingClass=2
IOSchedulingPriority=7
ExecStartPre=/bin/bash -c '/usr/bin/systemctl is-active --quiet apache2.service'
ExecStartPre=/bin/bash -c '! /usr/bin/systemctl is-active --quiet nextcloud-preview-generator.service'
ExecStartPre=/bin/bash -c '! /usr/bin/systemctl is-active --quiet borgmatic.service'
ExecCondition=/usr/bin/php -f /var/www/nextcloud/occ status -e
ExecStart=/usr/bin/php -f /var/www/nextcloud/cron.php
TimeoutStopSec=350
KillMode=mixed

and nextcloudcron.timer:

[Unit]
Description=Run Nextcloud cron.php every 5 minutes

[Timer]
OnBootSec=5min
OnUnitActiveSec=5min
Unit=nextcloudcron.service

[Install]
WantedBy=timers.target

I tried to add ExecStopSec=/usr/bin/systemctl stop nextcloudcron.service to self-stop the service if 350 seconds passed, but it claims that the service should be stopped on behalf of root. If I prepend the command with sudo it says that www-data is obviously not in sudoers. Of course, I may create separate service to kill the nextcloudcron.service if it is stuck more than 5 mins.