Slow synchronisation between Desktop-Client and Server?

🚧 Installation
1

Good Afternoon,

I am running Nextcloud on my Server. First some Server-Specs:
Model: HP Microserver Gen8 (Intel Celeron CPU)
OS: VMware ESXi 6.0.0 (Virtual Maschine: NAS4Free)

The Datatransfer via a CIFS/SMB -Share is very fast (80-100 MB/s), but the synchronisation between Nextcloud and the PC is very slow (10 KB/s to max. 2 MB/s). I access Nextcloud in the local network. In addition, the CPU utilization is very high (40-90%).

Is that “normal” or where/what is the “mistake”?

Thanks a lot :smiley:

2

are these mostly small files?

when yes, try this setting in your mysql-config:

Ubuntu 14.04: nano /etc/mysql/my.cnf
Ubuntu 16.04: nano /etc/mysql/mysql.conf.d/mysqld.cnf

innodb_flush_log_at_trx_commit = 2

*edit:

probably not this setting at all, was a bit to fast in reading and typing :smiley:

what does your client say about it?

to activate your logging, start the owncloud-client (I assume you’re using windows) as following:

"C:\Program Files (x86)\ownCloud\owncloud.exe" --logwindow --logfile "C:\owncloudclient.log"

3

What process is using 40-90% of the CPU?
Can you also check if iowait is limiting your performance (can be seen within top).
Did you set up redis for locking-cache?

The database creates an important load when uploading a large number of small files especially when not optimized. Try rather large files if you can reach higher speeds. Do you use the encryption app? External storage?

4

Thanks a lot.
The proprocesses are “lighttpd” and “php-cgi”. I don’t use the encryption app or external storage. It is a lot faster, when I put all files to a .zip or .rar File :smiley:

5

I use NAS4Free, but via SSH I can reach all Files from there. But it is a lot faster, when I put all files to a .zip or .rar File

6

we’re still hoping to get your log-files, without them its just a guessing around for us :smiley:

7

The difference for a lot of small files is that your database needs to create a lot of entries. You can optimize your mysql settings with help of scripts like tuning-primer.sh. I did some test in the past where I found some parameters that can increase the speed considerably, part is the innodb_flush-stuff that @victorbw mentioned (however there is a risk that some queries will be lost in case your server crashes):

I should run this tests again with the current version of NC.

8

I found a post where someone solved that issue by converting some tables to myisam. Has anyone seen an improvement trying it? Are there risks?

https://forum.owncloud.org/viewtopic.php?t=30083#p119093

9

@icewind @nickvergessen @rullzer

Would switching oc_activity, oc_file_cache and oc_file_locks to MYISAM affect data integrity? I think it would, but that it would be acceptable for home use.

10

Switch to using php-fpm, make sure redis is configured for file locking, configure a fast local memcache (apcu is usually the best option) and use php7 if possible if you’re currently on 5.x.

11

@icewind
pretty much the answer I expected. The thing is though, it does not help with the issue at all. For a small low-end server with two or three users the speed difference is hardly noticable if memcache and redis file locking is active. Without it uploads one (small) file per second and with it too. Meanwhile my experience is that setting innodb_flush_log_at_trx_commit = 2 or converting the responsible tables to MYISAM makes nextcloud upload more than thousand (small) files per minute.

Besides, I don’t want to ever touch redis again, after basically troubleshooting for about a month due to locking issues.

Edit: php-fpm does not help either. You basically described the setup I ran for a year with innodb_flush_log_at_trx_commit = 2

12

@oparoz MYISAM is not supported:
https://docs.nextcloud.com/server/11/admin_manual/installation/system_requirements.html#database-requirements-for-mysql-mariadb

2 - InnoDB storage engine (MyISAM is not supported)

13

Thanks guys.

Further investigation revealed that switching to MyIsam will generate problems with indices down the line and could lead to data loss. So those hacks are not recommended if you want to keep your data safe.

14

No idea why my post is marked as a solution, as it obviously is not since both hacks can result in data loss.

15

havent experienced any data-loss since using innodb_flush_log_at_trx_commit = 2 in my.cnf of mysql.

as far as I know, this setting is only there to tell mysql how to write its logs - nothing else … but i could be terribly wrong of course :smile:

if you need to handle a certain speed for small files, use this setting BUT also start doing backups tho.

1 Like
16

http://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_flush_log_at_trx_commit

With a value of 2, the contents of the InnoDB log buffer are written to the log file after each transaction commit and the log file is flushed to disk approximately once per second. Once-per-second flushing is not 100% guaranteed to happen every second, due to process scheduling issues. Because the flush to disk operation only occurs approximately once per second, you can lose up to a second of transactions in an operating system crash or a power outage.
[…]
InnoDB’s crash recovery works regardless of the innodb_flush_log_at_trx_commit setting. Transactions are either applied entirely or erased entirely.

17

Can anyone elaborate on what the solution discussed here is? I have the same excruciatingly slow sync process with about 1 small file per second. There is reference to my.cnf, which is a file which I cannot find on my RPi2.

Thanks

18

Further onto this: i am syncing my Mint Home folder with abotu 100GB to Cloudbox (1TB Western Digital HD + RPi2). Not wanting to flood my network, I copied all files to the HD via USB and ran commands to update the NextCloud box (so it was aware of the new files). I therefore expected the synchronisation to be ready in a few minutes (as everything was already in sync). Yet, 2 days later, we have done about 20%.

The command top shows several apache2 processes running at 50 to 80% of CPU time, yet my bandwidth use is a few kb/s.

1 Like
19

I solved the problem in this way:

  • setup new VM in ESXi
  • install Ubuntu Server 16.04 LTS
  • install Nextcloudserver on it

Works!

I dont know, that OS is installed on the Nextcloud-Box, but I would install Ubuntu Server in the RPi and setup Nextcloud there. If you have Problems, I can help :smiley:

20

Thanks for the offer for help, but I am not keen to start all over again. :worried:

Western Digital offered a WD 1TB drive with Ubuntu snappy preloaded SD card. I never booted on my RPi2, so I loaded Pixel Raspbian which I believe is based on Ubuntu 14.04 or 16.04. The RPi2 runs at 900MHz, 4 core CPU, 1GB ram, probably not comparable with the hardware you run your VM on.

There are several references above made by insiders to configuration changes that need a little more guidance for less experienced users, like clearly stating which configuration file they refer to, from which application (Nextcloud, PHP, MySQL, Apache).