High CPU Load - every crontab - docker mysql

Some time ago (Jan. 23), I switched the the background job from AJAX to Cron. Since then, I have a very high CPU load with every crontab execution.

Data:

  • Ubuntu 22 LTS (up2date)
  • Intel i3-6100
  • NC 25.0.4
  • Docker 23
  • Users: 3-5 active

docker compose:

version: '3.9'

networks:
 nextcloud:

services:
  nextcloud:
    image: nextcloud:${VERSION}
    container_name: nextcloud
    networks:
      - nextcloud
    ports:
      - "127.0.0.1:8080:80"
    volumes:
      - ${NEXTCLOUD_ROOT}/app_${VERSION}:/var/www/html
      - /media/freigaben/nextcloud_230112:/srv/nextcloud/data
    extra_hosts:
      - "${NEXTCLOUD_FQDN}:${NEXTCLOUD_IPADDRESS}"
      - "${COLLABORA_FQDN}:${NEXTCLOUD_IPADDRESS}"
    depends_on:
      - mysql
      - redis
    environment:
      - NEXTCLOUD_TRUSTED_DOMAINS='${NEXTCLOUD_FQDN}'
      - NEXTCLOUD_DATA_DIR=/srv/nextcloud/data
      - MYSQL_DATABASE=${MYSQL_DATABASE}
      - MYSQL_USER=${MYSQL_USER}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD}
      - MYSQL_HOST=nextcloud-mysql
      - REDIS_HOST=nextcloud-redis
      - PHP_MEMORY_LIMIT=2G
      - PHP_UPLOAD_LIMIT=2G
    restart: unless-stopped

  mysql:
    image: mysql:8
    container_name: nextcloud-mysql
    restart: unless-stopped
    volumes:
     - /srv/db/nextcloud/lib_${VERSION}:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD}
      - MYSQL_DATABASE=${MYSQL_DATABASE}
      - MYSQL_USER=${MYSQL_USER}
    networks:
      - nextcloud

  redis:
    image: redis:7
    container_name: nextcloud-redis
    networks:
      - nextcloud
    restart: unless-stopped

  coturn:
    image: coturn/coturn:4
    container_name: nextcloud-coturn
    restart: unless-stopped
    ports:
      - "3478:3478/tcp"
      - "3478:3478/udp"
    networks:
      - nextcloud
    command:
      - -n
      - --log-file=stdout
      - --min-port=49160
      - --max-port=49200
      - --realm=${NEXTCLOUD_FQDN}
      - --use-auth-secret
      - --static-auth-secret=${COTURN_SECRET}

config.php:

<?php
$CONFIG = array (
  'instanceid' => '###',
  'passwordsalt' => '###',
  'secret' => '###',
  'trusted_domains' => 
  array (
    0 => '###.de',
    1 => '###.eu',
    2 => '10.1.1.4',
    3 => '[###]',
  ),
  'version' => '25.0.4.1',
  'dbtype' => 'mysql',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'installed' => true,
  'allow_local_remote_servers' => true,
  'datadirectory' => '/srv/nextcloud/data',
  'dbname' => 'nextcloud_170226',
  'dbhost' => 'nextcloud-mysql',
  'dbuser' => '###',
  'dbpassword' => '###',
  'trusted_proxies' => 
  array (
    0 => '10.42.0.0/16',
    1 => '103.21.244.0/22',
    2 => '103.22.200.0/22',
    3 => '103.31.4.0/22',
    4 => '104.16.0.0/12',
    5 => '108.162.192.0/18',
    6 => '131.0.72.0/22',
    7 => '141.101.64.0/18',
    8 => '162.158.0.0/15',
    9 => '172.64.0.0/13',
    10 => '173.245.48.0/20',
    11 => '188.114.96.0/20',
    12 => '190.93.240.0/20',
    13 => '197.234.240.0/22',
    14 => '198.41.128.0/17',
    15 => '2400:cb00::/32',
    16 => '2606:4700::/32',
    17 => '2803:f800::/32',
    18 => '2405:b500::/32',
    19 => '2405:8100::/32',
    20 => '2c0f:f248::/32',
  ),
  'htaccess.RewriteBase' => '/',
  'theme' => '',
  'loglevel' => 2,
  'maintenance' => false,
  'default_phone_region' => 'DE',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'apps_paths' => 
  array (
    0 => 
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 => 
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' => 
  array (
    'host' => 'nextcloud-redis',
    'password' => '',
    'port' => 6379,
  ),
  'overwriteprotocol' => 'https',
  'overwrite.cli.url' => 'https://###.eu',
  'simpleSignUpLink.shown' => false,
  'enabledPreviewProviders' => 
  array (
    0 => 'OC\\Preview\\PNG',
    1 => 'OC\\Preview\\JPEG',
    2 => 'OC\\Preview\\GIF',
    3 => 'OC\\Preview\\HEIC',
    4 => 'OC\\Preview\\BMP',
    5 => 'OC\\Preview\\XBitmap',
    6 => 'OC\\Preview\\MP3',
    7 => 'OC\\Preview\\TXT',
    8 => 'OC\\Preview\\MarkDown',
    9 => 'OC\\Preview\\OpenDocument',
    10 => 'OC\\Preview\\Krita',
  ),
  'app_install_overwrite' => 
  array (
    0 => 'apporder',
    1 => 'files_readmemd',
    2 => 'sharingpath',
  ),
  'memories.exiftool' => '/var/www/html/custom_apps/memories/exiftool-bin/exiftool-amd64-glibc',
  'memories.vod.path' => '/var/www/html/custom_apps/memories/exiftool-bin/go-vod-amd64',
);

crontab:

*/10    *  *  *  *  nc-usr        docker exec --user www-data nextcloud php /var/www/html/cron.php >> /srv/log/cron_nc.log 2>&1

CPU Load:

Disc I/O Load:

  • The data is stored on conventional HDDs.
  • NC itself and the database on a NVMe SSD
  • High load is only on one core. But this core is then constant to 100% load.
  • There are no error messages. Neither in the cron_nc.log nor in the NC instance itself.
  • When I manually start the NC cron, the same behavior occurs.
  • If I vary the execution time (5m to 1h), the behavior is the same: the cron is executed for about 6min. If I choose short periods (5m to 20min), some cron are terminated after few seconds (you can see in the diagram).
  • I have noticed this with NC v24. I have upgraded and it did not bring any improvement.
  • I tried to disable (almost) all NC apps. This also brought no improvement.

What could be the reason for this?
How can I optimize this?

Thanks a lot!

1 Like

Hmm, have you tried switching back to ajax for the moment? Perhaps similar topics on the forum or server github Issues.

Could you please confirm some specs: cpu model number, how much ram.

Check your docker logs. (Back to work)

Thank you just.

Currently it still runs via cronetab. Reason for the change from AJAX was that no server messages were sent anymore, no files transferred etc. “Background jobs” also said “Last job ran 3 months ago”.

Since the switch to the cron everything works great.

  • Intel i3-6100, 2C/4T, 3.70GHz
  • 16 GB DDR4-RAM, ECC
  • WD Red SN700 NVMe 500GB

RAM load is only at 30-50%. SWAP file exists (on the NVMe), but is not used.

Unfortunately, the log does not show anything, except the successful start. I restarted the container yesterday 19:12 again, since then the cron also runs every 20min, even with high load as before.

$ docker logs nextcloud-mysql
2023-05-15T19:12:01.306368Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.32).
2023-05-15T19:12:04.234953Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.32)  MySQL Community Server - GPL.
2023-05-15 19:12:05+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.32-1.el8 started.
2023-05-15 19:12:05+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2023-05-15 19:12:05+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.32-1.el8 started.
'/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock'
2023-05-15T19:12:05.792446Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
2023-05-15T19:12:05.794226Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.32) starting as process 1
2023-05-15T19:12:05.800732Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-05-15T19:12:05.943404Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2023-05-15T19:12:06.112894Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2023-05-15T19:12:06.112934Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2023-05-15T19:12:06.114036Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2023-05-15T19:12:06.133816Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2023-05-15T19:12:06.133870Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.32'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server - GPL.

I don’t know if it helps, but a cron.php always (!) runs 5:30-5:45min.
No matter if executed via crontab or manually.

Have you tried searching the forum and internet and documentation for more info on Nextcloud crontab

There are tons of topics on this. I do not have this issue so nothing more to add, but I found lots of related topics and info on these searches. Hope it helps.

Of course, that should be part of the basis before any question :wink:

A lot of questions, but no clear answer … and for me also no solution.

What I could exclude

Cron.php hangs causing 100% cpu load

caused by bookmarks add-in. I dont’ use this


A new cron.php every 10 minutes and 100% CPU load
Cron.php creats high mysql load and doesn't finish
The cron.php tasks run for hours, constant CPU load, queries on oc_filecache

“cronjob never ends…” → my cronjos end every 5-7min sucessfully. No need to break it up.


Cron.php running erratically

no answer


Why so many cron logs & high cpu load

only one cron is runing, on my machine


[SOLVED] Occ command; PHP Fatal error, Allowed memory size of XXX bytes exhausted - #57 by crobarcro

commenting out memcache.local’ => ‘\OC\Memcache\APCu’, from /var/www/nextcloud/config/config.php, letting the cron job run

No improvement

What I (also) had tried without success
  • Cron every 1min: It is executed every now and then. Mostly one start every 7-14min.
  • Cron every 2-10min: same behavior as above.
  • Cron >10min: it is always executed.
  • Cron every 1-6h: same as above

The duration of an execution is always very same. Always 5-7min, no matter if every 1min or 6h.


docker exec -ti --user www-data nextcloud php /var/www/html/occ db:add-missing-indices
docker exec -ti --user www-data nextcloud php /var/www/html/occ db:add-missing-primary-keys
docker exec -ti --user www-data nextcloud php /var/www/html/occ db:convert-filecache-bigin

No improvement


docker exec --user www-data nextcloud php --define apc.enable_cli=1 -f /var/www/html/cron.php

No improvement


-d memory_limit=4G

it’s allredy set in compose file to PHP_MEMORY_LIMIT=2G


“slow_query_log” has no entries about queries that have run for more than 2s


I had stopped all (!) NC apps → no success.

What I had not yet tried so far
  • Nextcloud Docker to fpm or fpm-alpine
  • find unprintable characters in MySQL DB (I had read somewhere, I can’t find it anymore)

The solution is/was:

  • Using MariaDB!

I created a dump from my mysql 8 db and loaded it into a mariadb 10 (both docker). Since then the problem no longer occurs: (since 48h)

MySQL:

MariaDB:

What I also noticed here:

  • the NC24 mysql database (lib folder) was 4.2GB
  • the NC25 mysql db is 16GB !
  • the dump of this db is 2,4GB uncompressed
  • the NC25 MariaDB is 4,1GB

Not much new data has been added to the database since NC24 (last about February) (about 6TB).
Since the switch to MariaDB I have not noticed any losses (of files, app data, …).

I will continue to follow the topic, but this one is closed (for now) for me.
If someone needs more information (about the 16GB mysql db) for debbugging reasons, just contact me.

1 Like

After a few days the same thing happend again:

Now I have NC 27 and still the same.

I wrote a script that restarts the database if the load is too high for more than 60 minutes. The script is executed by cron on full hour:

#!/bin/bash
# run as user

counter=0
while [ $counter -lt 6 ]; do
    cpu=$(docker stats nextcloud-mysql --no-stream --format "{{.CPUPerc}}" | cut -d'.' -f 1)
    if [ $cpu -gt 50 ]; then
        # CPU Load one Core over 50%
                counter=$((counter+1))
        sleep 10m
    else
                # Load < 50%
        exit 0
    fi
done

# More than 60 min Load too high, restart docker container
docker restart nextcloud-mysql

exit 0

The very crazy thing about the result is that the hanging of the database always starts at 10:50!
So twice a day, always 10:50 am and 10:50 pm!
Since I run the script never more, but also never less.

The Nextcloud cron currently runs every 5 min, as 10:45, 10:50, 10:55 …

Does anyone have a guess what is started at 10:50 am/pm in the cron?
Can this possibly be read out?