I am running Nextcloud on my Server. First some Server-Specs:
Model: HP Microserver Gen8 (Intel Celeron CPU)
OS: VMware ESXi 6.0.0 (Virtual Maschine: NAS4Free)
The Datatransfer via a CIFS/SMB -Share is very fast (80-100 MB/s), but the synchronisation between Nextcloud and the PC is very slow (10 KB/s to max. 2 MB/s). I access Nextcloud in the local network. In addition, the CPU utilization is very high (40-90%).
What process is using 40-90% of the CPU?
Can you also check if iowait is limiting your performance (can be seen within top).
Did you set up redis for locking-cache?
The database creates an important load when uploading a large number of small files especially when not optimized. Try rather large files if you can reach higher speeds. Do you use the encryption app? External storage?
Thanks a lot.
The proprocesses are “lighttpd” and “php-cgi”. I don’t use the encryption app or external storage. It is a lot faster, when I put all files to a .zip or .rar File
The difference for a lot of small files is that your database needs to create a lot of entries. You can optimize your mysql settings with help of scripts like tuning-primer.sh. I did some test in the past where I found some parameters that can increase the speed considerably, part is the innodb_flush-stuff that @victorbw mentioned (however there is a risk that some queries will be lost in case your server crashes):
I should run this tests again with the current version of NC.
Would switching oc_activity, oc_file_cache and oc_file_locks to MYISAM affect data integrity? I think it would, but that it would be acceptable for home use.
Switch to using php-fpm, make sure redis is configured for file locking, configure a fast local memcache (apcu is usually the best option) and use php7 if possible if you’re currently on 5.x.
@icewind
pretty much the answer I expected. The thing is though, it does not help with the issue at all. For a small low-end server with two or three users the speed difference is hardly noticable if memcache and redis file locking is active. Without it uploads one (small) file per second and with it too. Meanwhile my experience is that setting innodb_flush_log_at_trx_commit = 2 or converting the responsible tables to MYISAM makes nextcloud upload more than thousand (small) files per minute.
Besides, I don’t want to ever touch redis again, after basically troubleshooting for about a month due to locking issues.
Edit: php-fpm does not help either. You basically described the setup I ran for a year with innodb_flush_log_at_trx_commit = 2
Further investigation revealed that switching to MyIsam will generate problems with indices down the line and could lead to data loss. So those hacks are not recommended if you want to keep your data safe.
With a value of 2, the contents of the InnoDB log buffer are written to the log file after each transaction commit and the log file is flushed to disk approximately once per second. Once-per-second flushing is not 100% guaranteed to happen every second, due to process scheduling issues. Because the flush to disk operation only occurs approximately once per second, you can lose up to a second of transactions in an operating system crash or a power outage.
[…] InnoDB’s crash recovery works regardless of the innodb_flush_log_at_trx_commit setting. Transactions are either applied entirely or erased entirely.
Can anyone elaborate on what the solution discussed here is? I have the same excruciatingly slow sync process with about 1 small file per second. There is reference to my.cnf, which is a file which I cannot find on my RPi2.
Further onto this: i am syncing my Mint Home folder with abotu 100GB to Cloudbox (1TB Western Digital HD + RPi2). Not wanting to flood my network, I copied all files to the HD via USB and ran commands to update the NextCloud box (so it was aware of the new files). I therefore expected the synchronisation to be ready in a few minutes (as everything was already in sync). Yet, 2 days later, we have done about 20%.
The command top shows several apache2 processes running at 50 to 80% of CPU time, yet my bandwidth use is a few kb/s.
I dont know, that OS is installed on the Nextcloud-Box, but I would install Ubuntu Server in the RPi and setup Nextcloud there. If you have Problems, I can help
Thanks for the offer for help, but I am not keen to start all over again.
Western Digital offered a WD 1TB drive with Ubuntu snappy preloaded SD card. I never booted on my RPi2, so I loaded Pixel Raspbian which I believe is based on Ubuntu 14.04 or 16.04. The RPi2 runs at 900MHz, 4 core CPU, 1GB ram, probably not comparable with the hardware you run your VM on.
There are several references above made by insiders to configuration changes that need a little more guidance for less experienced users, like clearly stating which configuration file they refer to, from which application (Nextcloud, PHP, MySQL, Apache).