I have a NextCloud server, and I’m experiencing high CPU usage issues when uploading multiple files and folders.
The server performs very well 99% of the time, staying below 2% CPU usage and 40% memory usage. However, when a user uploads several small XML files (around 2,000 files, each less than 100KB), the system becomes unresponsive, with CPU usage hitting 100%, sometimes taking an hour to upload just a few MB.
This load of thousands of PDF and XML files happens approximately once a week and is done via a web browser.
Even after the upload is complete, the CPU usage remains around 80% due to the cron.php. After a few minutes, the system returns to normal.
Here are my current configurations:
Server: AWS EC2 t3a.medium (2vCPU - 4GB RAM)
OS: Ubuntu 22.04 LTS
Web Server: NGINX
Database: MySQL 8
Cache: Redis and APCu
PHP: 8.1
NextCloud: 29.0.1
NextCloud Configuration:
<?php
$CONFIG = array (
'instanceid' => '*******',
'passwordsalt' => '*******',
'secret' => '*******',
'trusted_domains' =>
array (
0 => '*******',
),
'loglevel' => 3,
'loglevel_frontend' => 3,
'maintenance_window_start' => 4,
'default_language' => 'pt_BR',
'force_language' => 'pt_BR',
'default_locale' => 'pt_BR',
'force_locale' => 'pt_BR',
'default_phone_region' => 'BR',
'skeletondirectory' => '',
'profile.enabled' => false,
'knowledgebaseenabled' => false,
'defaultapp' => 'files',
'memcache.local' => '\OC\Memcache\APCu',
'memcache.locking' => '\OC\Memcache\Redis',
'redis' => [
'host' => 'localhost',
'port' => 6379,
'timeout' => 0.0,
],
'datadirectory' => '/var/www/*******/data',
'dbtype' => 'mysql',
'version' => '29.0.1.1',
'overwrite.cli.url' => '*******',
'dbname' => '*******',
'dbhost' => 'localhost',
'dbport' => '',
'dbtableprefix' => 'oc_',
'mysql.utf8mb4' => true,
'dbuser' => '*******',
'dbpassword' => '*******',
'installed' => true,
'mail_from_address' => '*******',
'mail_smtpmode' => 'smtp',
'mail_sendmailmode' => 'smtp',
'mail_domain' => '*******',
'mail_smtphost' => '*******',
'mail_smtpport' => '587',
'mail_smtpauth' => 1,
'mail_smtpname' => '*******',
'mail_smtppassword' => '*******',
);
MySQL Configuration:
transaction_isolation = READ-COMMITTED
binlog_format = ROW
skip_name_resolve = 1
tmp_table_size= 64M
max_heap_table_size= 64M
slow_query_log = 1
slow_query_log_file = /var/log/mysql/slow.log
long_query_time = 1
There are no warnings or errors in the NextCloud logs, and the configuration page shows no missing settings.
In ‘htop’, it shows that the high CPU usage is due to “php-fpm: poll www” and “/usr/sbin/mysqld”. In MySQL, when running “SHOW FULL PROCESSLIST”, I always see that it’s inserting data into the “oc_filecache” table.
Additional Information:
On the same production server, I have an OwnCloud instance using the same MySQL but running PHP 7.4. OwnCloud is slightly faster, with CPU usage around 60%, but it does not block other access.
In my test environment, I created two servers:
A - A server with all recommended installation settings: Apache2; APCu; PHP-8.3; Redis via socket; MariaDB 10, etc.
B - A server using NextCloud AIO Docker
In both cases, the same issue occurs when uploading thousands of files. However, the problem does not occur when uploading these files via WinSCP and manually scanning:
sudo -u www-data php occ files:scan --path="/administrador/files/TESTE2"
Even with 3,000 files, the “occ files:scan” command executes in seconds without burdening the processor.
I was considering uploading a compressed folder and then extracting it using the Extract app, but I saw it is no longer compatible with NextCloud 29.
I understand NextCloud has some performance limitations with the number of files, but am I missing something? Has anyone else experienced this? Are there any configurations or guidance to improve mass file uploads?