Uploads of many small files max out CPU

ā„¹ļø Support šŸ“¦ Appliances (Docker, Snappy, VM, NCP, AIO)
#1

I have Nextcloud on a VPS, with the data directory on an encrypted rclone remote on GDrive. So far it has worked well, but Iā€™ve only given it very light use.

A few weeks ago I made an account for a friend, who started uploading his files for backing up. He has a huge amount of small files, and this is making the server go nuts - mysqld causes the load to go very high, CPU usage at 95% to 100%, and everything else suffers.

Iā€™ve looked at ā€˜andā€™ to auto-renice the process but it doesnā€™t touch root processes (and mysqld is a root process, probably because I have installed Nextcloud via snap), and cpulimit (not sure if/how I can automate that as process IDs change all the time).

Is this normal? Do you think itā€™s happening purely because of the rclone remote setup - can anyone comment on whether this behaviour is seen on Nextcloud servers with local storage?

Any ideas for solutions are very welcome. I would love to somehow automate a check so that if mysqld or a related Nextcloud process starts hogging the CPU, itā€™s reniced to allow other processes to take over if needed.

Can 3rd-party sync apps be disabled?
#2

Use redis if possible. The database cache can change the speed enormously. I have done a few tests a long time ago (https://github.com/owncloud/core/issues/20967), you find newer topics here. I donā€™t know if the snap lets you change all these configurations.

#3

Yes, use a cache server such as Redis to assist Nextcloud for large numbers of files or for large file sizes. An alternative to setting up a cache server is to use Seafile instead, which has the best performance out of the box and doesnā€™t need a cache server, but is pretty much just for cloud storage. Nextcloud has a good selection of apps, so if a Redis server fixes the performance issue, Nextcloud is a great system to use.

#4

As far as I understand, I have set up the snap to use redis. The settings are there in config.php, and a redis-server process is running. I donā€™t think I can configure it in any way other than the way itā€™s already configured due to it running inside the snap though.
Redis has always been running and these issues occurred with it running unfortunately.

#5

Have you configured the ā€˜memcache.lockingā€™ => ā€˜\OC\Memcache\Redisā€™,
in the nextcloud.config file?
And when going to settings --> overview
do you pass all the security & setup warnings?