Self-hosted nextcloud instance downloads/uploads speed slow, even with performant server and gigabit internet

Nextcloud version: 19.0.5
Maybe important: nextcloud is not able to detect the cpu and memory available, or the gateway. I am able to reach Nextcloud from outside because I have an open port on my router targeted at the FreeNAS jail
Operating system and version: FreeBSD 11.3-RELEASE-p14 amd64_ (FreeNAS jail)
Apache or nginx version (eg, Apache 2.4.25): Apache/2.4.46 (FreeBSD)
PHP version (eg, 7.1):

Version: *7.3.21*
Memory Limit: *4 GB*
Max Execution Time: *3600*
Upload max size: *78.1 GB*

The issue you are facing:
Hello,
I own a FreeNAS server (located in my home) and I’ve been struggling to identify why upload and download speed are slow.

I used the following guide for the original setup: https://www.samueldowling.com/2020/07/24/install-nextcloud-on-freenas-iocage-jail-with-hardened-security/ and have been happily running Nextcloud for a couple of years.

My home server (home as in: it is located in my home) is a dual for 16 cores/32 threads. I set cpuset=all on my jail and I noticed that when I’m downloading a 1GB file from NextCloud, apache24 uses 13% of the CPU (which seems excessive!). I also have 32 GB available and the system load is essentially 0% the entire time. All this to say, I doubt this is a hardware problem. When downloading the same file through SFTP, I get ~800 mbit in download speed, while I understand it’s a different protocol, there is something that could be done for sure to improve nextcloud upload/download speed.

Both my laptop and my server are connected through cable.

Any recommendation on how to improve the situation? I’m quite confident it’s a configuration problem, but I can’t identify what exactly is given the many parts at play.

What information could I provide to help debugging the issue?

Is this the first time you’ve seen this error? (Y/N): No, it’s always been like that.

Steps to replicate it:

Dragging-and-dropping a file results in slow upload speed. Downloading any file results in slow download speed

The output of your Nextcloud log in Admin > Logging:
There isn’t a specific line related to this issue. When I downloaded the file, no log entry was written

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'apps_paths' => 
  array (
    0 => 
    array (
      'path' => '/usr/local/www/nextcloud/apps',
      'url' => '/apps',
      'writable' => true,
    ),
    1 => 
    array (
      'path' => '/usr/local/www/nextcloud/apps-pkg',
      'url' => '/apps-pkg',
      'writable' => false,
    ),
  ),
  'logfile' => '/var/log/nextcloud/nextcloud.log',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'instanceid' => 'myinstanceid',
  'passwordsalt' => 'mysalt',
  'secret' => 'mysecret',
  'trusted_domains' => 
  array (
    0 => '192.168.1.21',
    1 => 'mydomain.something',
    2 => 'mydomain.something:10021',
  ),
  'datadirectory' => '/mnt/data',
  'dbtype' => 'mysql',
  'version' => '19.0.5.2',
  'overwrite.cli.url' => 'http://192.168.1.21',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost:/tmp/mysql.sock',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud_admin',
  'dbpassword' => 'somedbpassword',
  'installed' => true,
  'redis' => 
  array (
    'host' => '/tmp/redis.sock',
    'port' => 0,
  ),
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'mail_from_address' => 'somemailaddress',
  'mail_smtpmode' => 'smtp',
  'mail_smtpsecure' => 'tls',
  'mail_sendmailmode' => 'pipe',
  'mail_domain' => 'somemaildomain',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpauth' => 1,
  'mail_smtphost' => 'somesmtphost',
  'mail_smtpname' => 'somesmtp@mailaddress',
  'mail_smtppassword' => 'somesmtppassword',
  'mail_smtpport' => '587',
  'theme' => '',
  'loglevel' => 2,
  'maintenance' => false,
  'updater.release.channel' => 'stable',
  'updater.secret' => 'updatersecret',
);

The output of your Apache/nginx/system log in /var/log/httpd_access.log:

192.168.1.1 - - [15/Dec/2020:21:48:07 -0800] "PROPFIND /remote.php/dav/files/francesco/ HTTP/1.1" 207 371
192.168.1.1 - - [15/Dec/2020:21:48:01 -0800] "GET /remote.php/webdav/MyBigFile.zip HTTP
/1.1" 200 124314448
127.0.0.1 - - [15/Dec/2020:21:48:11 -0800] "OPTIONS * HTTP/1.0" 200 -
127.0.0.1 - - [15/Dec/2020:21:48:13 -0800] "OPTIONS * HTTP/1.0" 200 -
192.168.1.1 - - [15/Dec/2020:21:48:16 -0800] "PROPFIND /remote.php/dav/files/francesco/ HTTP/1.1" 207 391
192.168.1.1 - - [15/Dec/2020:21:48:24 -0800] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 304 -

The output of your Apache/nginx/system log in /var/log/httpd_error.log:

[Tue Dec 15 21:05:23.829393 2020] [http2:warn] [pid 54876] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm d
etermines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not
do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.
[Tue Dec 15 21:05:23.831542 2020] [mpm_prefork:notice] [pid 54876] AH00163: Apache/2.4.46 (FreeBSD) OpenSSL/1.0.2s-freebsd configured -
- resuming normal operations
[Tue Dec 15 21:05:23.831592 2020] [core:notice] [pid 54876] AH00094: Command line: '/usr/local/sbin/httpd -D NOHTTPACCEPT'
[Tue Dec 15 21:06:39.904452 2020] [core:notice] [pid 54876] AH00052: child pid 54879 exit signal Segmentation fault (11)
[Tue Dec 15 21:06:39.904533 2020] [core:notice] [pid 54876] AH00052: child pid 54882 exit signal Segmentation fault (11)
[Tue Dec 15 21:06:39.904680 2020] [mpm_prefork:notice] [pid 54876] AH00169: caught SIGTERM, shutting down
[Tue Dec 15 21:06:40.016437 2020] [http2:warn] [pid 54942] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm d
etermines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not
do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.
[Tue Dec 15 21:06:40.018915 2020] [mpm_prefork:notice] [pid 54942] AH00163: Apache/2.4.46 (FreeBSD) OpenSSL/1.0.2s-freebsd configured -
- resuming normal operations
[Tue Dec 15 21:06:40.018964 2020] [core:notice] [pid 54942] AH00094: Command line: '/usr/local/sbin/httpd -D NOHTTPACCEPT'
[Tue Dec 15 21:48:31.212004 2020] [core:notice] [pid 54942] AH00052: child pid 54948 exit signal Segmentation fault (11)
[Tue Dec 15 21:48:31.229965 2020] [core:notice] [pid 54942] AH00052: child pid 55508 exit signal Segmentation fault (11)
[Tue Dec 15 21:48:31.300492 2020] [core:notice] [pid 54942] AH00052: child pid 58633 exit signal Segmentation fault (11)
[Tue Dec 15 21:48:31.300623 2020] [mpm_prefork:notice] [pid 54942] AH00169: caught SIGTERM, shutting down
[Tue Dec 15 21:48:31.411492 2020] [http2:warn] [pid 58672] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm d
etermines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not
do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.
[Tue Dec 15 21:48:31.414032 2020] [mpm_prefork:notice] [pid 58672] AH00163: Apache/2.4.46 (FreeBSD) OpenSSL/1.0.2s-freebsd configured -
- resuming normal operations
[Tue Dec 15 21:48:31.414094 2020] [core:notice] [pid 58672] AH00094: Command line: '/usr/local/sbin/httpd -D NOHTTPACCEPT'

Hi,

I looked at the tutorial, and it doesn’t seem to cover your mysql/mariadb config.
What does your mysql config look like?

You’ll find it in /usr/local/etc/mysql/my.conf for mysql.

# $FreeBSD$

[mysqldump]
max_allowed_packet=512MB

[mysqld]
innodb_buffer_pool_size=2G
innodb_io_capacity=4000

I often suggest this as a minimum on FreeBSD based installs, otherwise mysql starts chewing up swap space, which obviously makes it very slow. The high apache usage is actually because it is waiting for mysql to respond to a request.

-G

I configured the file (located in /var/db/mysql/my.cnf based on my ps output) with the settings you recommended, but it didn’t make any difference. However I discovered something very important though: when I access nextcloud through direct LAN ip address (192.168.1.21) and download the file from there, I get ~95MB speed, which is what I would expect. So the problem is limited to WAN access (when accessing from a domain pointing to my external IP address).

EDIT: I setup a digitalocean server with 4 cpus and executed some tests there. I discovered I was able to reach 288 mbit in upload speed (~35MB) using iperf and ~28MB/sec with scp.

The router might be a bottleneck, however the difference in speed is so wide that I’m not completely confident this is the case. Not to mention, I get full speed over LAN.

Hi,

Well the good news is that Nextcloud is not slow over LAN. I would start by looking at the router, and working my way in. If you are able to, spin up snap of Nextcloud on hard metal and see what results you get over WAN?

Lot’s of variables to work through, you are going to have to eliminate them all one by one.
Best luck! Let me know!

-G

I can’t spin up nextcloud on baremetal easily, however I spun up nginx on my laptop (baremetal) and served a 1GB file straight from there. Then I used wget from a server and saw the same problem. I’ll try with the file directly in RAM and a very beefy server and see what happens, but it seems like I need a new router.

I did some more serious tests this morning and it seems like I can upload ~760mbit/sec to a friend (has gigabit too) by serving a file with NGINX from my laptop (directly from RAM). I am able to download a file with the same speed from a server. Both have about 5 to 10% cpu usage of my router.

However, when I download a file “from myself” when passing through the internet, the cpu usage skyrockets to 100% and I get 150mbit speed.

It seems like it’s a router misconfiguration (or a firmware issue, who knows). I’ll start investigating!

1 Like

OK, I was able to solve this nightmare without purchasing a new router.
So apparently when I hit computers on my local network from WAN, my router uses “NAT Loopback” (which makes sense). However this functionality does not run on hardware (no hardware acceleration), hence the CPU usage.
My solution was to use dnsmasq (preinstalled on the router) to point the domains to local IP addresses. (Hosts file on the router would have achieved the same).
It’s not perfect, but it does the trick, I’m now getting ~90MB/sec download speed.