Nextcloud limited to exactly 15 Mbit - What is the bottleneck / max performance of Raspberry Pi?

yawicz · September 8, 2019, 7:34pm

Nextcloud version: 16.0.4
Operating system and version: latest Raspbian 9 (Stretch) / Debian 9.9
Apache or nginx version: Apache 2.4.25
PHP version: 7.2.22

The issue you are facing:
Download and upload speeds max out at 15Mbit, even in LAN, instead of possible 100Mbit.

Detailed description/what I tried so far:
Since I setup my Raspberry 3B+ with Nextcloud last year, it’s generally running like a charm, but the performance was always something I wanted to have a closer look at one day.
Now the day came and I spent quite some time reading through all those useful tuning tips, especially those specifically targeting the setup on a Raspberry Pi. Thus, I applied
the “common” tweaks to speed up the performance: Install and activate Redis, adjust MySQL settings (see below), adjust OPCache settings, upgrade PHP7.1 to PHP7.2.
Afterwards, I monitored the performance when up- and downloading one big file via web browser access, WebDAV access and direct Samba fileshare in my local area network at home.
Results -> nothing really changed to before in terms of transfer speeds:
Web browser and WebDAV via SSL (as expected) exactly the same: very constant 14,9-15,4 MBit/s
Samba: 85-95 Mbit/s (which seems to be the maximum possible with my current setup)

Thus, to me it looks very much like there would be some sort of fixed limit / bottleneck when up-/downloading via Nextcloud. I know of course the Raspberry Pi is not a performance
monster - but still, shouldn’t there be a bit more performance possible considering the Samba performance? I monitored the hardware usage during the upload and the resources were
constantly used at around 5-25% with some very short CPU peaks at around 50-65% - that’s it. I also know that there’s a severe performance decrease when uploading a big number of
small files - thus I tried to find the maximum limit by only uploading one big file.
I tried to find comparisons of others using Nextcloud on the Raspberry Pi, but couldn’t really find much to be honest. The only thing I found is some other threads with a similar
topic, but they were never really solved in the end. (see Very Slow upload speed or File Uploads From Web Interface Limited to 10 Mbit/s)

Thus my main questions would be:

Is this already the maximum speed possible on a Raspberry Pi, although the resources don’t seem to be used at even half of what they could?
What could be the bottleneck? After trying around a lot and optimizing the “main suspects” (Memcaching, PHP, MySQL, OPCache), my only idea left would be SSL, could that be?
If it should be SSL: I still don’t understand why the spare CPU power wouldn’t be used to speed things up a bit - any ideas for further optimization options?

Just FYI: I’m using Ethernet on the Pi and an 2,5" external HDD as data storage, thus I am aware that both are sharing the USB controller maximum speed. But still I guess the Samba file upload
shows what would be the final writing speed limitations of this setup.

Is this the first time you’ve seen this error?: N

Steps to replicate it:

Open Nextcloud Website or WebDAV connection, upload or download a big file (around 500MB)
Monitor network usage on Client laptop
Repeat step 1 using Samba file share instead
Monitor network usage on Client laptop and compare values

The output of your Nextcloud log in Admin > Logging:
No output during test period.

MySQL settings:

max_connections         = 20
table_cache             = 64
table_open_cache        = 144
thread_concurrency      = 10
sort_buffer_size        = 512K
net_buffer_length       = 8K
read_buffer_size        = 256K
read_rnd_buffer_size    = 512K
myisam_sort_buffer_size = 8M

query_cache_type        = 1
query_cache_limit       = 1M
query_cache_size	= 16777216

innodb_buffer_pool_size = 180M
innodb_buffer_pool_instances = 1
innodb_additional_mem_pool_size = 2M
innodb_log_file_size = 8M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50
innodb_large_prefix=true
innodb_file_format=barracuda
innodb_file_per_table=1
innodb_file_format_max=barracuda

# Logging of slow queries --> No entries so far
slow_query_log_file     = /var/log/mysql/mariadb-slow.log
long_query_time = 10
log_slow_rate_limit     = 1000
log_slow_verbosity      = query_plan
log-queries-not-using-indexes

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => 'INSTANCE_ID_HERE',
  'passwordsalt' => 'PASSWORD_HERE',
  'secret' => 'SECRET_HERE',
  'trusted_domains' =>
  array (
    0 => 'URL_0',
    1 => 'URL_1',
    2 => 'URL_2',
    3 => 'URL_3',
    4 => 'URL_4',
    5 => 'URL_5',
    6 => 'URL_6',
  ),
  'datadirectory' => '/path/to/data/directory',
  'overwrite.cli.url' => 'http://x.x.x.x/cloud',
  'dbtype' => 'mysql',
  'version' => '16.0.4.1',
  'dbname' => 'DBNAME_HERE',
  'dbhost' => 'localhost:3306',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'ADMIN_USER_HERE',
  'dbpassword' => 'DBPASSWORD_HERE',
  'installed' => true,
  'forcessl' => true,
  'forceSSLforSubdomains' => true,
  'maintenance' => false,
  'theme' => '',
  'loglevel' => 2,
  'updater.release.channel' => 'stable',
  'mail_smtpmode' => 'smtp',
  'mail_smtpsecure' => 'ssl',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_from_address' => 'mail',
  'mail_domain' => 'domain.de',
  'mail_smtpauth' => 1,
  'mail_smtphost' => 'smtp.domain.de',
  'mail_smtpport' => '465',
  'mail_smtpname' => 'mail@domain.de',
  'mail_smtppassword' => 'SMTP_PASSWORD_HERE',
  'memcache.local' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'filelocking.enabled' => 'true',
  'redis' =>
  array (
    'host' => '/var/run/redis/redis.sock',
    'port' => 0,
    'timeout' => 0.0,
  ),
  'mysql.utf8mb4' => true,
);

The output of your Apache system log in /var/log/apache2:
No relevant errors during the time of testing in logs error.log and access.log, just regular GET/POST/PUT/MOVE entries. Can be posted if considered as necessary.

I hope I provided as much detail as possible relevant to my questions and am just in general very interested in your opinion and experiences. If that’s the max speeds possible on a
Raspberry Pi, then so be it, still great. But if there’s still ideas and options/settings I didn’t consider that are limiting my setup, I would of course be happy to adjust them
and use the maximum speeds possible, especially since even my Internet connection would allow for up to 50 Mbit/s.

Thanks a lot for your answers already and in general to the team and great community behind Nextcloud - Keep up the great work!
Best, yawicz

yawicz · September 9, 2019, 11:10pm

Ok…so I finally solved this by myself and it’s the classic example of how sometimes the most difficult appearing problems actually have a super easy reason/solution. Thus, I even feel a bit ashamed to clarify what was the solution in the end, because it’s just too simple and stupid. I also don’t think it will help many others since it was in the end very setup-specific…but anyways, there we go:

First of all, my setup in short:
ISP --> FritzBox -LAN-> Switch -LAN-> Asus WiFi Router -WLAN-> Laptops, Smartphones, etc.
|||||||||||||||||||||||||-LAN-> Raspberry Pi 3B+

To my defense I have to say that it was actually a combination of two factors making up the perfect illusion that it would be a Nextcloud related problem, which it wasn’t actually.
Problem a) was:
My (headless) Raspberry, although connected via LAN, still had the WiFi actived which is why it was also connected to the WiFi Router at the same time (which wasn’t planned of course). But that explained why the Samba filetransfer was so much faster than the rest: it was using the well working WiFi instead of the LAN connection, which was misconfigured, see -->
Problem b):
Bit by bit I started to narrow the problem down: a speedtest on my Raspberry showed that the internet/LAN connection itself was fine, thus it was not the LAN port/cable/connection in general. The switch was also fine, but it became more and more clear that in the end the problem needed to be the connection between both subnetworks, since the WiFi network itself was also working fine.
In the end it turned out, that deep inside the settings of the Asus Router there was a so called “Quality of Service” option activated, that had a speed limit set to exactly the above mentioned 15Mbit (who does that?). The weird thing is: I can’t recall ever setting this up plus it was supposed to only affect additional guest WiFi networks. Instead, it affected the connection of the two subnetworks between the routers - but not the internet or WiFi connection itself, otherwise I would have obviously noticed earlier. What an unnecessarily long search…and text.

Anyways, now the transfer speeds are on par and I’m more than happy with the result!

weB3DynamIT · September 12, 2019, 6:08pm

Really nice effort. Thank you very much for the detailed explanation.