Nextcloud requires a lot of CPU

Hello, I’ve been having major issues with CPU usage on my Nextcloud server for a few days now, with the PostgreSQL database that I updated to version 16.3 two weeks ago.
One can see that the CPU problems increase from the moment the users start working:

Docker stats:

top:

Can someone tell me how I can isolate the problem and find out why I need so many resources? I currently have 4 cores and 16GB RAM on the machine. If I increase to 12 cores, it still uses the entire CPU, and I have about 30-40 active employees working on it.

Best regards,
Manu

I can’t really help. It is a little noticeable that there is also a lot going on at night between 18:00 and 03:00. If I assume that no one or almost no one is working at night, it could very possibly be an overlay of CRON processes, although I would have expected a slightly different output. I also don’t know if the memory utilisation of your database is OK.

Can you deactivate the CRON processes tonight when nobody is working and run CRON manually. How long does this run for? Does it perhaps run for longer than the period between two CRON processes?

Hello, I don’t think it’s a CRON problem. On days when there is a holiday, I don’t have any CPU issues. The problems start when the users begin working, but with 30-40 users, it shouldn’t be consuming so much CPU. Even when I significantly increase the machine’s specifications, I still encounter these issues. It appears that the processes from Nextcloud, specifically php-fpm, are using a lot of CPU. Additionally, when I examine the network process between the two containers (database and Nextcloud), I notice that whenever the CPU is overloaded, there is an equal amount of network traffic between Nextcloud and the database. The CRON jobs at night are mainly backups.

Just a question and some general thoughts:

Are you and your users expiriencing any actual problems while working? Like are you getting error messages, are downloads or uploads failing, are services hanging or stopping, are things extremely slow?

If not, doesn’t that mean that everything is working as it should? I mean, if my server’s CPU load was never, let’s say, above 30-40% while everyone was using it, I would probably consider getting a less powerful server or allocating less resources to the VM and consolidating things to save some money. :wink:

Or in other words: Couldn’t the high CPU usage also mean that the software is using the CPU efficiently, in the sense that it is using its full processing power to complete the tasks as fast as possible , similar to RAM usage where there is a saying: Unused RAM is wasted RAM. Or am I looking at this the wrong way?

2 Likes

Hello, we don’t have any error messages, but when I open the UI of Nextcloud, it takes 1-2 minutes to load. This is very bad for the user experience. In the meantime, we have increased the CPU and RAM significantly to make it load a little faster. With a large server of 12 cores, it is somewhat faster than with 4 cores. I think it is trying to complete the tasks as quickly as possible, and that’s why it consumes so much CPU. My question is: how can I understand why there are so many tasks and what tasks are being executed that are causing such high CPU load?

So just the initial login → dashboard transition is slow?

What does the browser inspector indicate, such as under the Network tab?

Maybe you can also share a bit more about your environment, such as what apps are in use (or merely installed)? (I.e. occ app:list)

Hello, it is primarily the sync and loading of data that is very slow.”

root@c3-data:~# docker exec -u www-data nextcloud php occ app:list
Enabled:
  - activity: 2.21.1
  - bruteforcesettings: 2.9.0
  - cloud_federation_api: 1.12.0
  - dav: 1.30.1
  - encryption: 2.17.0
  - federatedfilesharing: 1.19.0
  - files: 2.1.0
  - files_downloadlimit: 2.0.0
  - files_external: 1.21.0
  - files_pdfviewer: 2.10.0
  - files_reminders: 1.2.0
  - files_sharing: 1.21.0
  - impersonate: 1.16.0
  - logreader: 2.14.0
  - lookup_server_connector: 1.17.0
  - nextcloud_announcements: 1.18.0
  - oauth2: 1.17.0
  - password_policy: 1.19.0
  - privacy: 1.13.0
  - provisioning_api: 1.19.0
  - recommendations: 2.1.0
  - related_resources: 1.4.0
  - serverinfo: 1.19.0
  - settings: 1.12.0
  - support: 1.12.0
  - text: 3.10.1
  - theming: 2.4.0
  - twofactor_backupcodes: 1.18.0
  - twofactor_totp: 11.0.0-dev
  - updatenotification: 1.19.1
  - viewer: 2.3.0
  - workflowengine: 2.11.0
Disabled:
  - admin_audit: 1.19.0
  - circles: 29.0.0-dev (installed 26.0.0)
  - comments: 1.19.0 (installed 1.16.0)
  - contactsinteraction: 1.10.0 (installed 1.10.0)
  - dashboard: 7.9.0 (installed 7.6.0)
  - federation: 1.19.0 (installed 1.16.0)
  - files_linkeditor: 1.1.20 (installed 1.1.20)
  - files_rightclick: 0.15.1 (installed 1.6.0)
  - files_trashbin: 1.19.0 (installed 1.16.0)
  - files_versions: 1.22.0 (installed 1.22.0)
  - firstrunwizard: 2.18.0 (installed 2.15.0)
  - notifications: 2.17.0 (installed 2.14.0)
  - photos: 2.5.0 (installed 2.2.0)
  - sharebymail: 1.19.0 (installed 1.16.0)
  - survey_client: 1.17.0 (installed 1.14.0)
  - suspicious_login: 7.0.0
  - systemtags: 1.19.0 (installed 1.19.0)
  - user_ldap: 1.20.0
  - user_status: 1.9.0 (installed 1.6.0)
  - weather_status: 1.9.0 (installed 1.6.0)
root@c3-data:~# 


CONTAINER ID   NAME                      CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
e9d44f428af3   nginx-server              2.52%     35.97MiB / 15.61GiB   0.23%     286GB / 284GB     238MB / 649kB     5
4cf7bf2b0bad   nextcloud                 225.08%   1.075GiB / 15.61GiB   6.88%     800GB / 363GB     696MB / 1.11TB    93
df5917d685f0   c3-data-db-1              131.90%   1.405GiB / 15.61GiB   9.00%     113GB / 448GB     11.4GB / 110GB    86
b82dd58e10a0   c3-data-datadog-1         0.56%     243.7MiB / 15.61GiB   1.52%     308MB / 1.07GB    36GB / 61.4MB     40
00a91ac2874e   c3-data-s3-s3-restic-1    0.00%     218.1MiB / 15.61GiB   1.36%     3.35TB / 24.5GB   18.2GB / 1.26TB   18
0f2f89ae86dc   c3-data-restic_backup-1   0.00%     35.2MiB / 15.61GiB    0.22%     169MB / 16.1GB    59.6GB / 16.1GB   2
52cc81745a96   c3-data-car-1             0.09%     33.2MiB / 15.61GiB    0.21%     2.25MB / 143kB    39.6MB / 49.2kB   3
0dde882453e5   c3-data-redis-1           6.45%     5.676MiB / 15.61GiB   0.04%     48.8GB / 18.3GB   10.6MB / 1.53GB   6


Hi @3d0220fd0f9a5702d0f7 Manuel,

Are you using external storage, or what is the reason you use server side encryption?

You have been warned during activation, that is a CPU-Expensive feature.

I can’t say with absolute certainty whether this is the sole cause; I would have to take a much closer look at your setup, but you haven’t told us that much. :wink:


Much and good luck,
ernolf

root@c3-data:/var/app# cat docker-compose.yml 
version: '3'

services:
  db:
    image: postgres:16.3-alpine
    restart: unless-stopped
    volumes:
      - db:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_DB=${POSTGRES_DB}
      - POSTGRES_USER=${POSTGRES_USER}
      - TZ=${TZ}
    networks:
      - nextcloud
    command:
      - "postgres"
      - "-c"
      - "max_connections=1000"
      - "-c"
      - "shared_buffers=3GB"

  redis:
    image: redis:alpine
    restart: unless-stopped
    volumes:
      - redis:/data
    environment:
      - TZ=${TZ}
    networks:
      - nextcloud

  nextcloud:
    container_name: nextcloud
    image: $CI_REGISTRY_IMAGE/nextcloud-fpm:latest
    restart: unless-stopped
    stdin_open: true
    tty: true
    links:
      - db
      - redis
    expose:
      - '80'
      - '9000'
    volumes:
      - app_data:/var/www/html
    environment:
      - TZ=${TZ}
    networks:
      - nextcloud

  nginx-server:
    container_name: nginx-server
    image: $CI_REGISTRY_IMAGE/nc_red_nginx:latest
    environment:
      - CRON_RESTARTNGINX=0 3 * * * service nginx reload
      - TZ=${TZ}
    links:
      - nextcloud
    restart: unless-stopped
    ports:
      - "0.0.0.0:443:443"
    volumes:
      - ssl_certs:/etc/ssl-nginx
      - app_data:/var/www/html
    networks:
      - nextcloud

  car:
    image: gitlab.intra.name.ch:4567/name/infrastructure/car:latest
    restart: unless-stopped
    environment:
      - DNSIMPLE_API_TOKEN=${DNSIMPLE_API_TOKEN}
      - DOMAINS=c3-data.intra.name.ch
      - DHPARAM=2048
      - TZ=${TZ}
    volumes:
      - ssl_certs:/usr/app/data/certs

  datadog:
    image: datadog/agent:latest
    privileged: true
    restart: unless-stopped
    environment:
      - DD_LOGS_ENABLED=true
      - DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true
      - DD_API_KEY=${DATADOG_API_KEY}
      - DD_CONTAINER_EXCLUDE="name:nextcloud_datadog_1 name:nginx-server name:nextcloud name:nextcloud_db_1 name:nextcloud_elasticsearch_1 name:nextcloud_redis_1 name:onlyoffice-document-server image:onlyoffice/documentserver"
      - DD_CONTAINER_INCLUDE="name:nextcloud_db_dump_1 name:db_dump name:nextcloud_crypto-layer_1 name:crypto-layer name:nextcloud_restic_backup_1 name:restic name:nextcloud_car_1 name:car"
      - DD_PROCESS_AGENT_ENABLED=true
      - TZ=${TZ}
    volumes:
      - /opt/datadog-agent/run:/opt/datadog-agent/run:rw
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /proc/:/host/proc/:ro
      - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
    networks:
      - nextcloud

  db_dump:
    image: $CI_REGISTRY_IMAGE/nc_db_dump:latest
    environment:
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_DB=${POSTGRES_DB}
      - POSTGRES_USER=${POSTGRES_USER}
      - TZ=${TZ}
    volumes:
      - db_dump:/var/opt
    networks:
      - nextcloud

  restic_backup:
    image: gitlab.intra.name.ch:4567/name/infrastructure/red-restic
    restart: unless-stopped
    environment:
      - BACKUP_CRON=30 2,9,12,15,18 * * *
      - RESTIC_FORGET_ARGS=--prune --keep-last 10 --keep-hourly 24 --keep-daily 7 --keep-weekly 52 --keep-monthly 120 --keep-yearly 100
      - AWS_ACCESS_KEY_ID=${API_KEY}
      - AWS_SECRET_ACCESS_KEY=${SECRET_KEY}
      - RESTIC_REPOSITORY=${RESTIC_BACKUP_REPOSITORY}
      - RESTIC_PASSWORD=${RESTIC_BACKUP_PASSWORD}
      - TZ=${TZ}
      - GOMAXPROCS=1
    volumes:
      - app_data:/data/nextcloud/app_data
      - db_dump:/data/db
      # to be removed once red-restic is migrated to alpine
      - /etc/localtime:/etc/localtime:ro

  s3-s3-restic:
    image: gitlab.intra.name.ch:4567/name/infrastructure/c3-data/s3-s3-red-restic
    cap_add:
      - SYS_ADMIN
    devices:
      - /dev/fuse:/dev/fuse
    restart: unless-stopped
    environment:
      - BACKUP_CRON=30 18 * * *
      - RESTIC_FORGET_ARGS=--prune --keep-last 10 --keep-hourly 24 --keep-daily 7 --keep-weekly 52 --keep-monthly 120 --keep-yearly 100
      - AWS_ACCESS_KEY_ID=${BACKUP_API_KEY}
      - AWS_SECRET_ACCESS_KEY=${BACKUP_SECRET_KEY}
      - RESTIC_REPOSITORY=${BACKUP_REPOSITORY}
      - RESTIC_PASSWORD=${BACKUP_PASSWORD}
      - TZ=${TZ}
      - S3_BUCKET=${S3_BUCKET}
      - S3_BUCKET_ENDPOINT=${S3_BUCKET_ENDPOINT}
      - S3_ACCESS_KEY=${S3_ACCESS_KEY}
      - S3_SECRET_ACCESS_KEY=${S3_SECRET_ACCESS_KEY}
      - GOMAXPROCS=1

volumes:
  app_data:
  db:
  ssl_certs:
  redis:
  db_dump:

networks:
  nextcloud:
    driver: 'bridge'
root@c3-data:/var/app# 

<?php
$CONFIG = array (
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'apps_paths' => 
  array (
    0 => 
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 => 
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'instanceid' => 'id',
  'passwordsalt' => 'salt',
  'secret' => 'Jsecret',
  'trusted_domains' => 
  array (
    0 => 'c3-data.intra.name.ch',
    1 => 'nginx-server',
  ),
  'datadirectory' => '/var/www/html/data',
  'dbtype' => 'pgsql',
  'version' => '29.0.4.1',
  'overwrite.cli.url' => 'http://c3-data.intra.name.ch',
  'dbname' => 'nextcloud',
  'dbhost' => 'db:5432',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'oc_root',
  'dbpassword' => 'password',
  'installed' => true,
  'overwriteprotocol' => 'https',
  'redis' => 
  array (
    'host' => 'redis',
    'port' => '6379',
    'timeout' => '0.0',
  ),
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'skeletondirectory' => '/var/skeleton',
  'trashbin_retention_obligation' => '30, 31',
  'maintenance' => false,
  'mail_smtpmode' => 'smtp',
  'mail_sendmailmode' => 'smtp',
  'mail_from_address' => 'c3-data',
  'mail_domain' => 'name.ch',
  'mail_smtphost' => 'smtp.eu.mailgun.org',
  'mail_smtpauth' => 1,
  'mail_smtpname' => 'c3-data@mail.name.ch',
  'mail_smtppassword' => 'password',
  'mail_smtpport' => '587',
  'twofactor_enforced' => 'true',
  'twofactor_enforced_groups' => 
  array (
  ),
  'twofactor_enforced_excluded_groups' => 
  array (
  ),
  'loglevel' => 0,
);
root@4cf7bf2b0bad:/var/www/html/config#