Up-/Download to slow (only 2MB/s) after installation despite good hardware

BoboN7 · October 24, 2020, 1:33pm

Hi, I have already set up Nextcloud several times with the help of great guides and learned a lot. Unfortunately I am still a beginner.

I have a performance problem in every installation which I just can’t get fixed, so I wanted to ask here. Both my upload and download are max 2-2.5 MB/s.

The hardware should not be the problem.

Does anyone have an idea? I am happy to give more information if someone has an idea.

System

Guide : HowTo: Ubuntu + Docker + Nextcloud + Talk + Collabora from KarlF12
Nextcloud version : Nextcloud 20.0.0
Operating system and version : Ubuntu 20.04
Apache or nginx version : Server version: Apache/2.4.41
PHP version : PHP 7.4.11 (inside the docker container)
PHP version : no php (outside the docker container)

I think the only thing I did differently than in the guide is:

Newer Ubuntu Server 20.04
The Redis Authentication Fix: HowTo: Ubuntu + Docker + Nextcloud + Talk + Collabora
And maybe my btrfs formatting

The issue:
Both my upload and download are max 2-2.5 MB/s.

Is this the first time you’ve seen this error?
No, i tried multiple installations via the guide above.

config.php and logs

The output of your Nextcloud log in Admin > Logging:

Nothing unusual, a few unsuccessfull login attempts from my smart phone.

Warnings in Admin > Overview:

None

The output of your config.php file in /path/to/nextcloud : I dont know how to get the output, but here is the config.php

<?php
$CONFIG = array (
  'htaccess.RewriteBase' => '/',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'apps_paths' =>
  array (
    0 =>
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 =>
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
    'host' => XXX,
    'password' => XXX,
    'port' => XXX,
  ),
  'instanceid' => XXX,
  'passwordsalt' => 'XXX',
  'secret' => 'XXX',
  'trusted_domains' =>
  array (
    0 => 'XXX',
  ),
  'datadirectory' => '/srv/nextcloud/data',
  'dbtype' => 'mysql',
  'version' => '20.0.0.9',
  'overwrite.cli.url' => 'XXX',
  'dbname' => 'nextcloud',
  'dbhost' => 'nextcloud-mariadb',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud',
  'dbpassword' => 'XXX',
  'installed' => true,
  'trusted_proxies' =>
  array (
    1 => 'XXX',
  ),
  'overwritehost' => 'XXX',
  'overwriteprotocol' => 'https',
);

KarlF12 · October 24, 2020, 2:44pm

Just the same, would you mind sharing hardware details?

I would start by narrowing down the cause. You can run iperf or iperf3 to rule out a possible network issue. You can (very carefully) use dd or another tool to test the effective read/write speed of your storage. Run htop and watch during upload/download to make sure you have sufficient processor and memory resources and little/no swap usage.

If there are no problems there then you might try putting some file in your Apache webroot and downloading it in your browser to see what the speed is like outside Nextcloud. The slowness has to start somewhere.

sambilanet · October 24, 2020, 3:17pm

What is your speed at check https://speedtest.net ?

BoboN7 · October 24, 2020, 3:26pm

Thank you for the reply!
I will apply the tools as soon as I get home tomorrow.

The hardware details (from my memory):

ssd 128gb from Samsung, no raid yet (as long as speed is an issue)
gigabit lan and new cat 7 cable
nextcloud and client are in the same network on different devices.
i7 7600, 32GB RAM

My Internet speed should be 13MB down and 4MB upload.

tflidd · October 24, 2020, 3:53pm

Can you check the upload and download speed via SFTP as a benchmark? You won’t get the same speed, it will be certainly less but you have to find the bottleneck. This can be tricky (don’t hesitate to search for similar threads where people already shared some tools and how to use them).

basil · October 25, 2020, 4:17am

Watch your units here. You do mean 13Mb/s down and 4Mb/s up? If so this equates to 1.6 MB/s down and 0.5 MB/s up, which is closer to the speed noted in your topic title.

BoboN7 · October 25, 2020, 8:50am

I checked on different a provider, here are the results:

My initial guess for the internet speed came from my router data:

even if the connection was via the Internet (although both are on the same network), it is too slow. Especially when downloading

I will tackle iperf3 and the SFTP Benchmark next and hope to find something.
As tflidd and KarlF12 have pointed out, finding the bottleneck is the first step. But sometimes it can be a challenge for me to even see where the bottle goes on

BoboN7 · October 25, 2020, 8:55am

I have already adjusted the units to make it easier for others to compare
Both are really (rounded down) MB/s, not Mbit/s.

anon71540698 · October 25, 2020, 9:15am

What are the devices you connect to Nextcloud?
How do they get onto the network? Wi-Fi? What “flavour”?
That’s a good candidate for the bottleneck…

sambilanet · October 25, 2020, 4:37pm

It´s more on your side and less “nextcloud” I guess. You might install the nextcloud “testaccount” and compare.

BoboN7 · October 25, 2020, 9:15pm

Hi, after lerning what iperf and an Apache webroot is (thanks, i feel a lot smarter now ), i managed to do some testing. I might have found some clues with the webroot thing. The transfer speed in iperf was is quite good.

iperf3 results:

Next i copied a 1GB video on the webroot ${NEXTCLOUD_ROOT}/html and started a download from my browser. This was kind of a success, because the download was very slow.

I stoped nextcloud and started a nginx container with:
sudo docker run --name mynginx1 -p 8080:80 -v /home/Username/nginxData:/mount1 -d nginx
Since the port is the same as the one nextcloud gets in the .yml file, i hoped that my reverse proxy wouldn’t know the difference. So i used this to emulate my nextcloud with a different webserver. The download was quite slow again.

Than i tried to download the file via LOCAL_IP/testvideo3.mp4 in my browser instead of FQDNs/testvideo3.mp4 and the download was very quick (about 116 MB/s) and certainly no bottleneck in sight .

I guess the video was downloaded from the Apache reverse proxy + Nginx webroot again, which had the slow download before. But this time within the local network.
I got the same result from Apache reverse proxy + Nextcloud webroot.

So everything seems to be fine (incl. the read/write speed of my storage) with the hardware performance.

I have another question @KarlF12: (sorry this is surely a stupid question)

You mentioned the concept of

split-horizon DNS

in your guide. Should this already be included via the instructions in the guide? Or would i need to configure this seperatly?
Also, you mentioned:

so I tried to test weather i got this hairpin behaviour as follows - someone else downloaded the same video FQDNs/testvideo3.mp4 remotely and got exactly the same slow speed as i had. What does this mean concerning my router firewall?

Sidenote:

Downloading it in parallel (not locally) did cut the speed in halve, as expected.
I don’t know if this is relevant in any way: But i use ddns from noip.com for my nextcloud.

KarlF12 · October 27, 2020, 1:57am

Do this:

nslookup FQDN

on your local network and see if it returns the local IP or outside IP. Your problem may be due to hairpin routing.

Hairpin routing means that you are accessing a local server via a public IP (incorrectly configured DNS or no local DNS server). What this means is you are accessing the local server by way of your router instead of directly. Most good quality routers don’t even allow this, but in any case, it wastes bandwidth and is bad for performance. The proper solution to this problem is to run a local DNS server that returns the local IP for the FQDN while you’re on your LAN, but you can also leverage the hosts file or use the local IP.

A quick test you can do to determine if that’s the cause is to add the local IP and FQDN to your hosts file and try your download again.

wwe · October 27, 2020, 9:02am

I have similar problem. it takes multiple days to sync my docs to the clients - both on Win and Linux in local LAN with 1GBits/s. The folder contains 70k files with 40GB. I have local DNS server which points Nextcloud FQDN to local IP so there is no loop through Internet (split-brain DNS). I don’t see exhausted resource on server: CPU is running 30%-70% plenty of RAM is free, no error messages. if there is a big file I see transfer speeds up to 50-70 MB/s but the average is really bad as often it only syncs 1-2 files per seconds. Upload/Download via Browser works with 40-50 MB/s. But syncing small files is terribly slow - I see the client only syncs 1-2 small files per second - even this are only KB. Web interface is pretty slow on initial load 5-10 sec spinning after entering a folder - but once it’s loaded one can view/download files really fast. I don’t see any errors in logs.

anybody has an idea how to analyze?

KarlF12 · October 27, 2020, 11:34am

@wwe You’ll need to repeat all of the above troubleshooting, and I suggest you also test the read/write speed of your disk. Your issue seems different since your system is having trouble just loading the pages.

wwe · October 27, 2020, 12:51pm

@KarlF12 I appreciate your help. from my feeling disk speed is not an issue as big files transfer fast, but I would like to proof/verify the assumptions. Copying the data from/to the system via NFS or SMB is fast and saturates the 1GBit/s network adapter. Could you point me to the way how I could verify the problem is caused by slow disk access? Is there any logging I can see delays when the system access the disk/database?

download via Browser

Upload via Browser

I made a test and uploaded a video vile with 134MB with a browser - it took~20 sec with very little pauses every 10MB (chunks) the speed shown in the progress bar was 3MB/s to 20MB/s. The file appeared on the local directory 30 sec later - sync took ~6 sec according to the client log, which is absolutely acceptable and shows disk and network perform fast. The problem I have is when lot of small files exist on a directory it takes ages…

2020-10-27 13:30:03:934 [ info nextcloud.gui.application ]:	Sync state changed for folder  "https://<my.nextcloud.fqdn>/remote.php/dav/files/<user>/Documents" :  "Sync Running"
2020-10-27 13:30:03:935 [ info nextcloud.sync.propagator ]:	Starting INSTRUCTION_NEW propagation of "Ask me anything, with Mark Russinovich on cloud, Sysinternals, security, and more-P1pk9VBdwUs.mp4" by OCC::PropagateDownloadFile(0x219c4d63090)
2020-10-27 13:30:03:936 [ info nextcloud.sync.accessmanager ]:	2 "" "https://<my.nextcloud.fqdn>/remote.php/dav/files/<user>/Documents/Ask me anything, with Mark Russinovich on cloud, Sysinternals, security, and more-P1pk9VBdwUs.mp4" has X-Request-ID "d30cff50-5b8f-4fb6-92f8-2a3fc1ce6c0a"
2020-10-27 13:30:03:937 [ info nextcloud.sync.networkjob ]:	OCC::GETFileJob created for "https://<my.nextcloud.fqdn>" + "/Documents/Ask me anything, with Mark Russinovich on cloud, Sysinternals, security, and more-P1pk9VBdwUs.mp4" "OCC::PropagateDownloadFile"
2020-10-27 13:30:04:491 [ info nextcloud.sync.credentials.webflow ]:	request finished
2020-10-27 13:30:04:491 [ info nextcloud.sync.networkjob.etag ]:	Request Etag of QUrl("https://<my.nextcloud.fqdn>/remote.php/dav/files/<user>/Contacts") FINISHED WITH STATUS "OK"
2020-10-27 13:30:08:313 [ info nextcloud.sync.credentials.webflow ]:	request finished
2020-10-27 13:30:08:326 [ info nextcloud.sync.checksums ]:	Computing "SHA1" checksum of "W:/Nextcloud/Documents/.Ask me anything, with Mark Russinovich on cloud, Sysinternals, security, and more-P1pk9VBdwUs.mp4.~fa9" in a thread
2020-10-27 13:30:08:660 [ info nextcloud.sync.database ]:	Updating file record for path: "Ask me anything, with Mark Russinovich on cloud, Sysinternals, security, and more-P1pk9VBdwUs.mp4" inode: 53 modtime: 1507790579 type: 0 etag: "e8cc653832d587e1493e7a777b34c60f" fileId: "00475173ocshh0oigtc5" remotePerm: "WDNVR" fileSize: 140892641 checksum: "SHA1:d2021affa8db72d65f0220b44f07004a74eef3e9" e2eMangledName: "" isE2eEncrypted: false
2020-10-27 13:30:08:662 [ info nextcloud.sync.propagator ]:	Completed propagation of "Ask me anything, with Mark Russinovich on cloud, Sysinternals, security, and more-P1pk9VBdwUs.mp4" by OCC::PropagateDownloadFile(0x219c4d63090) with status 4
2020-10-27 13:30:08:662 [ warning nextcloud.gui.activity ]:	Item  "Ask me anything, with Mark Russinovich on cloud, Sysinternals, security, and more-P1pk9VBdwUs.mp4"  retrieved resulted in  ""
2020-10-27 13:30:08:663 [ warning nextcloud.gui.activity ]:	Item  "Ask me anything, with Mark Russinovich on cloud, Sysinternals, security, and more-P1pk9VBdwUs.mp4"  retrieved successfully.
2020-10-27 13:30:08:663 [ info nextcloud.gui.activity ]:	Successfully added to the activity list:  ""
2020-10-27 13:30:08:664 [ info nextcloud.gui.folderwatcher ]:	Detected changes in paths: QSet("W:/Nextcloud/Documents/Ask me anything, with Mark Russinovich on cloud, Sysinternals, security, and more-P1pk9VBdwUs.mp4")
2020-10-27 13:30:08:665 [ info nextcloud.sync.database ]:	Closing DB "C:/Users/<user>/AppData/Roaming/Nextcloud/._sync_6d0d8033c51e.db"
2020-10-27 13:30:08:675 [ info nextcloud.sync.engine ]:	CSync run took  6145 ms

from the log it look like the client i doing many many thing just to replicate one file - is it expected?

The file appeared on the local client few seconds later. I checked the client log during the sync and see it executed bunch SQL queries for ALL existing files in the directory where I put the upload is this expected?

2020-10-27 13:16:02:962 [ info nextcloud.sync.csync.updater ]:	Database entry found for Readme.md, compare: 1597168289 <-> 1597168289, etag: 0aba3361429e9afd483f17bb8be7644a <-> 0aba3361429e9afd483f17bb8be7644a, inode: 0 <-> 1116, size: 136 <-> 136, perms: 9f <-> 9f, checksum:  <->  , ignore: 0,  e2e: 
2020-10-27 13:16:02:962 [ info nextcloud.sync.csync.updater ]:	file: Readme.md, instruction: INSTRUCTION_NONE <<=
2020-10-27 13:16:02:962 [ debug nextcloud.sync.csync.updater ]	[ csync_walker ]:	file: Welcome to Nextcloud Hub.docx [file_id=00000210ocshh0oigtc5 size=25150]
2020-10-27 13:16:02:963 [ debug nextcloud.sync.database.sql ]	[ OCC::SqlQuery::bindValue ]:	SQL bind 1 QVariant(qlonglong, 3622897607555501721)
2020-10-27 13:16:02:963 [ debug nextcloud.sync.database.sql ]	[ OCC::SqlQuery::exec ]:	SQL exec "SELECT path, inode, modtime, type, md5, fileid, remotePerm, filesize,  ignoredChildrenRemote, contentchecksumtype.name || ':' || contentChecksum, e2eMangledName, isE2eEncrypted  FROM metadata  LEFT JOIN checksumtype as contentchecksumtype ON metadata.contentChecksumTypeId == contentchecksumtype.id WHERE phash=?1"

I already tuned my mysql with the following settings:

innodb_flush_log_at_trx_commit = 2
innodb_flush_log_at_timeout = 5
max_connections = 250

which made the system feel little more responsive but still not really good to sync thousands of files. having max_connections = 100 I had an error in the logs that DB stopped responding because of “too many connections” - why does the system with 2 users and 3 clients need more then 100 sql connections?

KarlF12 · October 27, 2020, 5:34pm

@wwe Do the iperf test mentioned above and the dd test I linked in my last post.

You should start your own thread too.

wwe · October 27, 2020, 9:17pm

@KarlF12: thank you for your help. I started new thread with more streamlined description and additional data: Slow desktop client sync

BoboN7 · October 30, 2020, 9:54pm

nslookup FQDN did return the outside IP, so the problem is most likely hairpin routing.
I was able to place something in the nextcloud webroot and access it via LOCAL_IP/testvideo3.mp4 without hairpin routing. But if i try just LOCAL_IP, i get redirected to FQDN right away.
So my assumption was, that this might be a problem with my nextcloud configuration?

My only idea to fix this was to run my own dns server and use it in my router to resolve the FQDN with my local IP within the local network. But following a guide to run a dns docker called

sameersbn/bind

in http://www.damagehead.com/blog/2015/04/28/deploying-a-dns-server-using-docker/ did fail with the following message:

So i assume there is already something using the port.

Is there another (maybe easier) solution? Can i turn of the redirect for nextcloud (since it doesn’t seem to come from my router)?

Btw, here is my .yml. Maybe my error was somewhere in there:

version: '3.7'

networks:
 nextcloud:

services:
  nextcloud:
    image: nextcloud:custom
    build: .
    container_name: nextcloud
    networks:
      - nextcloud
    ports:
      - "127.0.0.1:8080:80"
    volumes:
      - ${NEXTCLOUD_ROOT}/html:/var/www/html
      - ${NEXTCLOUD_ROOT}/data:/srv/nextcloud/data
    extra_hosts:
      - "${NEXTCLOUD_FQDN}:${NEXTCLOUD_IPADDRESS}"
    depends_on:
      - mariadb
      - redis
    environment:
      - NEXTCLOUD_TRUSTED_DOMAINS='${NEXTCLOUD_FQDN}'
      - NEXTCLOUD_DATA_DIR=/srv/nextcloud/data
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
      - MYSQL_PASSWORD=${MYSQL_PASSWORD}
      - MYSQL_HOST=nextcloud-mariadb
      - REDIS_HOST=nextcloud-redis
    restart: unless-stopped

  mariadb:
    image: mariadb
    container_name: nextcloud-mariadb
    restart: unless-stopped
    volumes:
      - ${NEXTCLOUD_ROOT}/mariadb:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD}
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
    networks:
      - nextcloud

  redis:
    image: redis
    container_name: nextcloud-redis
    #command: redis-server --requirepass XYZ
    networks:
      - nextcloud
    restart: unless-stopped

  coturn:
    image: instrumentisto/coturn
    container_name: nextcloud-coturn
    restart: unless-stopped
    ports:
      - "3478:3478/tcp"
      - "3478:3478/udp"
    networks:
      - nextcloud
    command:
      - -n
      - --log-file=stdout
      - --min-port=49160
      - --max-port=49200
      - --realm=${NEXTCLOUD_FQDN}
      - --use-auth-secret
      - --static-auth-secret=${COTURN_SECRET}

KarlF12 · October 31, 2020, 1:03pm

No, this is an IP routing issue. Technically has nothing to do with Nextcloud.

It would seem so. Port 53 is used only for DNS. So what is it? netstat -lntp

Easier short term or long term? You could add the FQDN and local IP to your hosts file, but this is a “solution,” not a solution.

I guess I need to write a guide for setting up bind since this is such a common issue.

BoboN7 · November 1, 2020, 10:01pm

I meant something already build in that i am just not aware of

I did search for something exactly like netstat -lntp, so thank you again!

It revealed that systemd-resolve seems to block that port:

It was a system service that provides network name resolution to local applications. And this sounds as something that could already give me what i needed. But i disabled the service

systemctl disable --now systemd-resolved

and tried something i found meanwhile that seeemed to be exactly what i wanted: dnsmasq
I used this guide: Configure Local DNS Server using Dnsmasq on Ubuntu 20.04 - kifarunix.com
and it worked for me

Finally my tests confirmed that the original problem was solved and my speed is better than ever!

I was very happy! I did shut down the server and put in my raid controller raid 1 that were waiting to be used for the new nextcloud setup. Configuring them was no problem. But once ubuntu started again, i couldn’t connect via ssh and my router didn’t even recognize the system was connected.
Even ifconfig on the server didn’t return the local network. My assumption is, I caused something to block my ethernet connection? I couldn’t figure out what it could be yet. Maybe someone with more experience has an idea?