Slow upload performance -> high CPU load

ℹ️ Support
, ,
#1

Hello,

if I upload a file (size doesn’t matter, e.g. 3MB) to my Nextcloud instance, it uploads it complete, but it not finishes the upload process (uploading progress 3MB / 3MB).
Then I look into the process list (htop) and one php-fpm process (with ~80%) and redis (with ~55%) has high cpu load for about 20 seconds. After the high cpu load of these two processes is gone, the upload process is completed.
But while the client who is still uploading and waiting for completing the upload process, other clients are already downloading the new uploaded file.

It is very annoying to have the slow upload performance. I had to increase the timeout value in nginx, so the client doesn’t abort the upload. Any other things on my Nextcloud instance have good performance.

Nextcloud version: 13.0.4
Operating system and version: CentOS 7.5
Apache or nginx version: nginx 1.15.0
PHP version: 7.2

Is this the first time you’ve seen this error? (Y/N): N

Steps to replicate it:

  1. Upload file

The output of your Nextcloud log in Admin > Logging: nothing in log
The output of your Apache/nginx/system log in: nothing in log

my memcach config

  'memcache.local' => '\\OC\\Memcache\\APCu',
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
    'host' => 'redis',
    'port' => 6379,
    'timeout' => 1.5,
  ),

I already cleaned up filecache with occ files:cleanup.

Every help is very appreciated. If you need some more information, please let me know. Thanks!

#2

Download is no problem, just upload? Did you check your database-logs? Perhaps enable to log slow-queries.

#3

No, Download is no problem. I already enabled the slow query log.
There are not much slow statements in there and it has nothing to do with file operations.

Here are two examples which are in the mariadb-slow.log:

# Time: 180630  0:30:50
# User@Host: root[root] @  [172.19.0.4]
# Thread_id: 19454  Schema: nextcloud  QC_hit: No
# Query_time: 42.486385  Lock_time: 0.000068  Rows_sent: 0  Rows_examined: 1
# Rows_affected: 1  Bytes_sent: 52
SET timestamp=1530318650;
UPDATE `oc_news_feeds` SET `last_modified` = '1530318608298466' WHERE `id` = 23;

# User@Host: root[root] @  [172.19.0.4]
# Thread_id: 19478  Schema: nextcloud  QC_hit: No
# Query_time: 17.752362  Lock_time: 0.000264  Rows_sent: 0  Rows_examined: 1
# Rows_affected: 1  Bytes_sent: 52
SET timestamp=1530318650;
UPDATE `oc_authtoken` SET `last_activity` = 1530318633 WHERE `id` = 3701;

The most are from the news app. So no file operations.
I am curious about the high redis cpu load. Do you have an idea what that could be caused?

#4

42s query time is a lot. Is the news app filling the database too much? And nearly 20s for authtoken is also a lot, do your cronjobs run on a regular basis (check on your admin page when the cronjob ran for the last time).

Regarding redis, I haven’t seen many configuration problems with it. Perhaps check the log level that it tells you more in the logs. If it’s on the same host, you can connect to it through a socket, this is more secure and perhaps a bit faster.

On CentOS, it might be a good idea to keep an eye on SELinux, it can block certain things and lead to strange behaviour of other services.

#5

I had heavy load on my server a few days ago. In normal operation there are no slow queries in the log (e.g. the last days). The news app has ~14k entries and cronjob is running fine every 15 min. So that is no problem at all.

Redis is running mostly in default configuration (official docker container). I will check a better log level. Maybe I can find something there.

The SELinux thing sounds very interesting. I can imagine that this could be the problem. I will investigate.

Thank you very much for your good input and advice! I need some time for it to check your points. I have atm not much free time to look at it. Thanks!

Greets
nxtcld

#6

I just found the problem which caused the upload freezes:
I have >15k directories in the preview folder and ~1k from them had the attribute “size” set to -1 in the corresponding entry of the database table oc_filecache. Which triggers a scan each upload process and caused the freezes.
After running occ files:scan-app-data the entries gone away and the problem was fixed.

thanks for your help and greets
nxtcld

1 Like