High CPU Usage on NextCloud Server When Uploading Multiple Small Files

I have a NextCloud server, and I’m experiencing high CPU usage issues when uploading multiple files and folders.

The server performs very well 99% of the time, staying below 2% CPU usage and 40% memory usage. However, when a user uploads several small XML files (around 2,000 files, each less than 100KB), the system becomes unresponsive, with CPU usage hitting 100%, sometimes taking an hour to upload just a few MB.

This load of thousands of PDF and XML files happens approximately once a week and is done via a web browser.

Even after the upload is complete, the CPU usage remains around 80% due to the cron.php. After a few minutes, the system returns to normal.

Here are my current configurations:

Server: AWS EC2 t3a.medium (2vCPU - 4GB RAM)
OS: Ubuntu 22.04 LTS
Web Server: NGINX
Database: MySQL 8
Cache: Redis and APCu
PHP: 8.1
NextCloud: 29.0.1

NextCloud Configuration:

<?php
$CONFIG = array (
  'instanceid' => '*******',
  'passwordsalt' => '*******',
  'secret' => '*******',
  'trusted_domains' =>
  array (
    0 => '*******',
  ),
  'loglevel' => 3,
  'loglevel_frontend' => 3,
  'maintenance_window_start' => 4,
  'default_language' => 'pt_BR',
  'force_language' => 'pt_BR',
  'default_locale' => 'pt_BR',
  'force_locale' => 'pt_BR',
  'default_phone_region' => 'BR',
  'skeletondirectory' => '',
  'profile.enabled' => false,
  'knowledgebaseenabled' => false,
  'defaultapp' => 'files',
  'memcache.local' => '\OC\Memcache\APCu',
  'memcache.locking' => '\OC\Memcache\Redis',
  'redis' => [
        'host' => 'localhost',
        'port' => 6379,
        'timeout' => 0.0,
        ],
  'datadirectory' => '/var/www/*******/data',
  'dbtype' => 'mysql',
  'version' => '29.0.1.1',
  'overwrite.cli.url' => '*******',
  'dbname' => '*******',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => '*******',
  'dbpassword' => '*******',
  'installed' => true,
  'mail_from_address' => '*******',
  'mail_smtpmode' => 'smtp',
  'mail_sendmailmode' => 'smtp',
  'mail_domain' => '*******',
  'mail_smtphost' => '*******',
  'mail_smtpport' => '587',
  'mail_smtpauth' => 1,
  'mail_smtpname' => '*******',
  'mail_smtppassword' => '*******',
);

MySQL Configuration:

transaction_isolation = READ-COMMITTED
binlog_format = ROW
skip_name_resolve = 1
tmp_table_size= 64M
max_heap_table_size= 64M
slow_query_log = 1
slow_query_log_file = /var/log/mysql/slow.log
long_query_time = 1

There are no warnings or errors in the NextCloud logs, and the configuration page shows no missing settings.

In ‘htop’, it shows that the high CPU usage is due to “php-fpm: poll www” and “/usr/sbin/mysqld”. In MySQL, when running “SHOW FULL PROCESSLIST”, I always see that it’s inserting data into the “oc_filecache” table.

Additional Information:

On the same production server, I have an OwnCloud instance using the same MySQL but running PHP 7.4. OwnCloud is slightly faster, with CPU usage around 60%, but it does not block other access.

In my test environment, I created two servers:
A - A server with all recommended installation settings: Apache2; APCu; PHP-8.3; Redis via socket; MariaDB 10, etc.
B - A server using NextCloud AIO Docker

In both cases, the same issue occurs when uploading thousands of files. However, the problem does not occur when uploading these files via WinSCP and manually scanning:

sudo -u www-data php occ files:scan --path="/administrador/files/TESTE2"

Even with 3,000 files, the “occ files:scan” command executes in seconds without burdening the processor.

I was considering uploading a compressed folder and then extracting it using the Extract app, but I saw it is no longer compatible with NextCloud 29.

I understand NextCloud has some performance limitations with the number of files, but am I missing something? Has anyone else experienced this? Are there any configurations or guidance to improve mass file uploads?

You haven’t set the innodb_* buffer configuration settings, they perhaps need to be a bit adjusted to your table sizes:
https://docs.nextcloud.com/server/stable/admin_manual/configuration_database/linux_database_configuration.html#parameters

It would be great if the NC client could do such things on its own in order to speed up the transfer itself and reduce the number of db inserts.

Are these authenticated users or public uploads?

AFAIK the Web UI uploader uses the bulk upload functionality for small files.

'loglevel' => 3,

You’ll likely not see much at this level. 2 is the default.

'version' => '29.0.1.1',

You’re several maintenance releases behind. It might be worth confirming this on a supported version.

Thank you for the help.
I made a change that brought significant improvements, although it didn’t resolve everything 100%. Instead of having one server, I ended up splitting it into two servers. I also updated the systems to be more aligned with what the official documentation suggests.

Server1: AWS EC2 t3a.small (2vCPU - 2GB RAM)
OS: Ubuntu 22.04 LTS
Web Server: Apache
Cache: Redis and APCu
PHP: 8.1
NextCloud: 29.0.1

Server2: AWS EC2 t3a.small (2vCPU - 2GB RAM)
Database: MariaDB 10.11

With this, I have the WEB part on one server and the DATABASE on another server. This resulted in more processing power for the same cost. Now I am experiencing some I/O issues with the database, but I am managing to adjust based on InnoDB parameters that MySQL didn’t allow before.

makefile

Copiar código

skip_name_resolve = 1
innodb_buffer_pool_size = 512M
innodb_buffer_pool_instances = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_max_dirty_pages_pct = 90
innodb_large_prefix=on
innodb_file_format=barracuda
innodb_file_per_table=1
query_cache_type = 1
query_cache_limit = 2M
query_cache_min_res_unit = 2k
query_cache_size = 64M
tmp_table_size= 64M
max_heap_table_size= 64M
slow_query_log = 1
slow_query_log_file = /var/log/mysql/mariadb-log-slow.log
long_query_time = 1
transaction_isolation = READ-COMMITTED
binlog_format = ROW

This greatly improved the upload part, but the system is still a bit slow overall. However, it resolved the freezing issues that were present before. I will let you know as soon as I have more improvements.

1 Like