Nextcloud version (eg, 20.0.5): 21.0.2
Operating system and version (eg, Ubuntu 20.04): Raspbian 10 with docker
Apache or nginx version (eg, Apache 2.4.25): newest nginx
PHP version (eg, 7.4): 7.x
The issue you are facing:
A new cron.php is created every 10 minutes and 100% CPU load.
cronjob never ends…
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 200 4 ? Ss May29 0:00 s6-svscan -t0 /var/run/s6/services
root 38 0.0 0.0 200 4 ? S May29 0:00 s6-supervise s6-fdholderd
root 366 0.0 0.0 200 4 ? S May29 0:00 s6-supervise nginx
root 367 0.0 0.0 200 0 ? S May29 0:00 s6-supervise php-fpm
root 368 0.0 0.0 200 4 ? S May29 0:00 s6-supervise cron
root 374 0.0 0.0 5192 2508 ? Ss May29 0:00 nginx: master process /usr/sbin/nginx -c /config/nginx/nginx.conf
root 375 0.0 0.0 2820 596 ? Ss May29 0:00 bash ./run
root 386 0.0 0.0 2332 780 ? S May29 0:03 /usr/sbin/crond -f -S -l 0 -c /etc/crontabs
abc 2087 0.0 0.0 5376 1012 ? S May30 0:00 nginx: worker process
abc 2088 0.0 0.0 5376 1012 ? S May30 0:00 nginx: worker process
abc 2089 0.0 0.0 5376 1008 ? S May30 0:00 nginx: worker process
abc 2090 0.0 0.0 5376 1020 ? S May30 0:00 nginx: worker process
root 2092 0.0 0.6 254872 24136 ? Ss May30 0:13 php-fpm: master process (/etc/php7/php-fpm.conf)
abc 2094 0.0 0.1 254876 5872 ? S May30 0:00 php-fpm: pool www
abc 2095 0.0 0.1 254876 5876 ? S May30 0:00 php-fpm: pool www
abc 2622 0.0 1.1 261492 44964 ? S 07:30 0:01 php7 -f /config/www/nextcloud/cron.php
abc 2623 0.0 0.7 254872 27932 ? D 07:40 0:00 php7 -f /config/www/nextcloud/cron.php
abc 2625 0.0 0.7 254872 28220 ? D 07:50 0:00 php7 -f /config/www/nextcloud/cron.php
abc 2628 0.0 0.7 254872 28108 ? D 08:00 0:00 php7 -f /config/www/nextcloud/cron.php
abc 2629 0.0 0.7 254872 27896 ? D 08:10 0:00 php7 -f /config/www/nextcloud/cron.php
abc 2631 0.0 0.7 254872 28084 ? D 08:20 0:00 php7 -f /config/www/nextcloud/cron.php
abc 2633 0.0 0.7 254872 27980 ? D 08:29 0:00 php7 -f /config/www/nextcloud/cron.php
abc 2634 0.0 0.7 254872 28188 ? D 08:39 0:00 php7 -f /config/www/nextcloud/cron.php
abc 2636 0.0 0.7 254872 28036 ? D 08:49 0:00 php7 -f /config/www/nextcloud/cron.php
abc 2639 0.0 0.7 254872 27992 ? D 08:59 0:00 php7 -f /config/www/nextcloud/cron.php
abc 2640 0.0 0.7 254872 27920 ? D 09:09 0:00 php7 -f /config/www/nextcloud/cron.php
abc 2642 0.0 0.7 254872 28104 ? D 09:19 0:00 php7 -f /config/www/nextcloud/cron.php
abc 2644 0.0 0.7 254872 28144 ? D 09:29 0:00 php7 -f /config/www/nextcloud/cron.php
abc 2645 0.0 0.7 254872 28016 ? D 09:39 0:00 php7 -f /config/www/nextcloud/cron.php
abc 2647 0.0 0.7 254872 28148 ? D 09:49 0:00 php7 -f /config/www/nextcloud/cron.php
abc 2650 0.0 0.7 254872 28132 ? D 10:00 0:00 php7 -f /config/www/nextcloud/cron.php
abc 2651 0.0 0.7 254872 28108 ? D 10:10 0:00 php7 -f /config/www/nextcloud/cron.php
abc 2653 0.0 0.7 254872 27980 ? D 10:20 0:00 php7 -f /config/www/nextcloud/cron.php
abc 2655 0.0 0.7 254872 28256 ? D 10:30 0:00 php7 -f /config/www/nextcloud/cron.php
abc 2656 0.0 0.7 254872 28068 ? D 10:40 0:00 php7 -f /config/www/nextcloud/cron.php
abc 2658 0.0 0.7 254872 28080 ? D 10:50 0:00 php7 -f /config/www/nextcloud/cron.php
root 2667 0.0 0.0 2988 1700 pts/0 Ss 10:52 0:00 bash
root 2738 0.0 0.0 2560 720 pts/0 R+ 10:56 0:00 ps aux
oc_jobs just has 36 entries…
Nobody any idea??
Is it possible to find out which process triggers the cron job to hang up?
Ok, first thing to do is stop the cronjob from running. Then run the cronjob manually, and see if it completes – give it half an hour and/or monitor what its doing.
One of the problems you’re dealing with is your hardware. I take it from your OS line reading “Raspbian 10 with docker” that you have one of those underpowered trinket sbc’s, so any significant amount of data could easily overwhelm it. If the cron doesn’t finish within the available time, then the previous job will slow down the new job which will take even longer than the first, and before you know it, you’ve got an out of control cycle of new cronjobs starting.
You could also make a wrapper script for the cronjob that only lets it start a new run if the last one has terminated.
I think you are right.
I added the flock command to the cronjob to prevent a second instance to start.
I didn’t had any problems since then…
I will let you know if it works or not in a few days!
1 Like
I got some bad news at this…
Flock prevented cron.php to spawn endless times, but the one cron.php that’s running is gone to uninterruptible sleep, waiting for some I/O I think.
No way to kill a linux process in uninterruptible sleep state. I always have to hard cut the power cord of my raspberry. No kill command works, even killing the docker container doesn’t work. It prevents my whole system to shut down regularly.
Now we have to find out which part of the cron.php is the culprit.
But I am not familiar enough with php and nextcloud…
Here you can see the processes with the “D” state… it’s nginx and cron.php
root 1 0.0 0.0 200 4 ? Ss Jun16 0:00 s6-svscan -t0 /var/run/s6/services
root 39 0.0 0.0 200 4 ? S Jun16 0:00 s6-supervise s6-fdholderd
root 368 0.0 0.0 200 4 ? S Jun16 0:00 s6-supervise nginx
root 369 0.0 0.0 200 4 ? S Jun16 0:00 s6-supervise php-fpm
root 370 0.0 0.0 200 4 ? S Jun16 0:00 s6-supervise cron
root 373 0.0 0.0 5176 3492 ? Ss Jun16 0:00 nginx: master process /usr/sbin/nginx -c /config/nginx/nginx.conf
root 374 0.0 0.6 256404 26492 ? Ss Jun16 0:09 php-fpm: master process (/etc/php8/php-fpm.conf)
root 375 0.0 0.0 2820 1256 ? Ss Jun16 0:00 bash ./run
root 389 0.0 0.0 2328 816 ? S Jun16 0:01 /usr/sbin/crond -f -S -l 0 -c /etc/crontabs
abc 395 0.0 0.0 5376 2784 ? D Jun16 0:00 nginx: worker process
abc 396 0.0 0.0 5376 2784 ? D Jun16 0:00 nginx: worker process
abc 397 0.0 0.0 5376 2784 ? D Jun16 0:00 nginx: worker process
abc 398 0.0 0.0 5376 2784 ? D Jun16 0:00 nginx: worker process
abc 399 0.0 0.1 256400 6516 ? S Jun16 0:00 php-fpm: pool www
abc 400 0.0 0.1 256400 6516 ? S Jun16 0:00 php-fpm: pool www
abc 1393 0.0 0.7 256380 27636 ? D 11:20 0:00 php8 -f /config/www/nextcloud/cron.php
root 2009 0.0 0.0 2984 1736 pts/1 Ss 18:12 0:00 bash
root 2092 0.0 0.0 2560 728 pts/1 R+ 19:08 0:00 ps aux
So we can hold on, that it is not a problem with the performance of the raspberry.
There are a lot of users with the same problem running nextcloud on different hardware.
If the cronjob passes, it takes about 1 second.
It’s a fault in the process we have to debug.
Ah, i forgot, I still tried php8 instead of php7 for testing purposes, but the error is the same.
Hmm, if its in D state, it shouldn’t be consuming CPU.
Any network/remote filesystems in your setup? Maybe a bad SDCARD?
Yes. That’s right. It doesn’t consumes CPU any more since I am using “flock -n” for the cronjob, but the cron.php hangs forever and is preventing the container to work and blocks my whole system from shutting down normally.
No network file systems, but I am running nextcloud on my raspberry with docker from the internal SD-Card but the “data” and “config” is stored to an external USB-HDD.
I am using the external storage add-on in nextcloud, too…
Please take a look at my docker stack
version: “2”
services:
nextcloud:
image: linuxserver/nextcloud:version-21.0.2
container_name: nextcloud
networks:
- homeserver
environment:
- PUID=1000
- PGID=1000
- TZ=Europe/Berlin
volumes:
- /srv/dev-disk-by-uuid-77365390-c57e-4b8a-846f-42fa099bf411/docker/appdata/nextcloud/config:/config
- /srv/dev-disk-by-uuid-77365390-c57e-4b8a-846f-42fa099bf411/docker/appdata/nextcloud/data:/data
links:
- mariadb
labels:
- “diun.enable=true”
restart: unless-stopped
mariadb:
image: yobasystems/alpine-mariadb:armhf
container_name: mariadb
networks:
homeserver:
ipv4_address: 172.18.0.20
environment:
- PUID=1000
- PGID=1000
- TZ=Europe/Berlin
- MYSQL_DATABASE=nextcloud
- MYSQL_USER=nextcloud
- MYSQL_PASSWORD=xxx
- MYSQL_ROOT_PASSWORD=
volumes:
- /srv/dev-disk-by-uuid-77365390-c57e-4b8a-846f-42fa099bf411/docker/appdata/mariadb/logs:/var/lib/mysql/mysql-bin
- /srv/dev-disk-by-uuid-77365390-c57e-4b8a-846f-42fa099bf411/docker/appdata/mariadb/mysql/_data:/var/lib/mysql
restart: unless-stopped
phpmyadmin:
container_name: phpmyadmin-nextcloud
image: phpmyadmin
networks:
- homeserver
restart: unless-stopped
environment:
- PMA_HOST=172.18.0.20
- PMA_PORT=3306
ports:
- 8182:80
networks:
homeserver:
external:
name: homeserver
I’m afraid that docker is pretty much gibberish to me.