Limited upload speed on docker container

[pepesuriano]

Hi, I am willing to migrate my running nextcloud from my debian install to a docker container. When deploying containers (with this guide Docker Hub) everything is fine and my desktop sync client can connect, but for some reason the upload speed is limited to 30MB/s.

Using my “traditional” installation directly on debian this issue is not present, I can sync files maxing out my gigabit connection (110MB/s upload speed). It gets 30MB/s limited speed when using the docker container (download speed is fine at 110MB/s). I ran an iperf test from my docker container to my PC and I successfully get my 110MB/s upload speed so it is some configuration limitation.

I tried changing php memory limit as shown here (PHP memory limit to 128 ? · Issue #447 · nextcloud/docker · GitHub) to 512M and it didn’t work either.

Any ideas of what could be limiting my upload speed?

[pepesuriano]

Let’s try again, bump

[Mageunic]

Hi @pepesuriano,
I don’t know if it it can solve your issue but try this in nextcloud directory and refresh your web page:
sudo -u www-data php occ config:app:set files max_chunk_size --value 0

[pepesuriano]

@Mageunic

sudo -u www-data php occ config:app:set files max_chunk_size --value 0

I ran this on the nextcloud container and besides the command was successful, my speed is still limited to 30MBps

Here is the output of the command:
sudo -u www-data php occ config:app:set files max_chunk_size --value 0
Config value max_chunk_size for app files set to 0

[pepesuriano]

Let’s try again, Bump!

[Mageunic]

@pepesuriano do you use antivirus for files app ?

[pepesuriano]

Nope, it’s used from nextcloud’s debían client.

[pepesuriano]

Hi, I tried nextcloud 15 image and speed improved significantly! Also the auth step in the add accound wizard is working fine, with nextcloud 21 it fails and I have to put the link manually on my browser.

Steps to reproduce:
1- Use this docker-compose:

version: ‘2’

volumes:
nextcloud:
db:

services:
db:
image: mariadb
restart: always
command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
volumes:
- db:/var/lib/mysql

app:
image: nextcloud:15.0
restart: always
ports:
- 8080:80
links:
- db
volumes:
- nextcloud:/var/www/html

2- Add an account on your desktop client, copy a big file (2GB would be fine) to the nextcloud folder and wait for it to sync to see the speed, you should see more or less 80MB/s with a gigabit network.

3- Remove the nextcloud containers.

4- Modify the docker-compose by switching the version to latest (image: nextcloud:latest), then spin up the containers again

5- Repeat step 2, you will see now that your files sync at more or less 30MB/s speed.

When you switch from nextcloud 15 to nextcloud 21 the transfer speed get capped, any ideas of what should be the issue?

[wwe]

the topic clearly describes Traefik limits the speed but it’s definitely not such bad as you describe. I got this results wit Traefik and Docker:

NC download speed slower through traefik

I have no idea regarding Raspi - my Nextcloud with MariaDB behind Traefik running as Docker on x86 Intel Celeron based Qnap-453 NAS reaches 200+ Mbit/s when moving big files around (movies, .iso files etc.) with the client, when the client is on Wi-Fi. Browser up-/download in fact seems to be limited at 100-150 MBit/s = 10-20MB/s… Maybe the limit is related to Wi-Fi (unfortunately all my LAN ports are used now…) Interesting fact I have to investigate deeper. [image] the screenshot shows par…

Today I measured upload speeds from up to 700Mbit/s. NC Client is faster but even the browser goes over 200Mbit/s so at least in my installation I don’t hit any limits caused by traefik:

NC download speed slower through traefik

In the meanwhile I moved my docker to PC (Debian 10 on i3 6th Gen+16GB RAM+spinning disk). While the speed is unchanged with WiFi, with LAN connection I get much higher throughput. Browser transfer is >100MBIt/s for 1GB file and client sync goes over 500MBit/s (spikes to >800MBit/s). My conclusion is the problem is not related to Traefik itself as my tests proof it CAN handle faster transfers. [image] [image] [image] [image]

[pepesuriano]

Yes, 700Mbit/s is what I get when I deploy the nextcloud 15 container, if I just erase it and install nextcloud 21 and make the same test, the speed goes down to 200Mbit/s.

So it’s something different between NC21 and NC15, is it possible to you to make the test without traefik? That would be a good scenario for replicating the issue.

Cheers

[pepesuriano]

Small followup, I tried different nextcloud versions going from 15 up to 21 to see in which version the issue shows up. I found that when switching from nextcloud 18 to nextcloud 19 the speed cap appears, in nextcloud 18 I can have 700Mbit/s and changing to a nextcloud 19 server caps it to 200Mbit/s. Always using docker official images of nextcloud of course.

Hope this give more guidance.

[wwe]

I don’t exactly get what you are looking for. It may be the case NC15 ist faster then NC21. I assume current version has contains move moving parts.

My test show that NC itself and traefik CAN saturate the network - see another test, here I added 4 files with total size of 3,5GB. It was interesting initially I had send speed close to 1 GBit/s, then short pause and then another block at ~300-500MBit/s. complete transfer was finished in less then 40 seconds on NC21 container with traefik, with HTTPS

Downloading this 4 files (uploaded via Web) happens in two chunks with >500Mbit/s speed, again complete 3,5GB is received in less then a minute.

definitely if you calculate an average network load it would be far less then real transfer speed. Additionally all the tests show similar behavior - a transfer starts much faster and settles at lower rate later - most likely there is some caching which saturates after 1-2GB and falls back to the bare-metal limits (in my case most likely spinning disk)

My earlier statement regarding syncing small files is slower doesn’t meant overhead as “additional network traffic” but everything inside the system, adding new files to DB, updating indices and so on. I don’t have a proof but I would expect the system is busy doing some things in background an just doesn’t use the network at this time - this is the reason huge files sync lot faster as small files because such overhead is almost constant per file an doesn’t really matter if you sync iso files or movies but hurts if you upload thousands of small text files.

your system may have some bottleneck but I conclude with my tests NC21 itself and traefik don’t apply this limits if the hardware performs well. New versions consume more resources then older versions - and offer you more functionality. You may try to remove apps you don’t need like activity, deck, unified search - maybe this results in a speed improvement which satisfy you needs.

[pepesuriano]

Can you share the docker-compose if you have it or the docker command to run your setup of NC + traefic? It’s worth considering that my tests are done by transfering one big file, so any processing bottlenecks on the NC server that happen when you send several small files don’t occur, plus I checked and I don’t get CPU, memory o hard disk saturation.

Perhaps trying your setup will give us more clues, remember that we are talking about a very specific change when switching from NC 18 to NC19, if it was just that newer software require more processing power, the step should not be so big.

[wwe]

i use docker-compose similar to this. There are few additional settings I didn’t mentioned in that post (but most likely this is not the reason for the performance gap):

./nextcloud/php/conf.d/uploads.ini

memory_limit=512M
upload_max_filesize = 500M
post_max_size = 500M

./nextcloud/db-config/my.cnf

[server]
skip-name-resolve
innodb_buffer_pool_size = 256M
innodb_buffer_pool_instances = 1
innodb_flush_log_at_trx_commit = 2
innodb_flush_log_at_timeout = 5
innodb_log_buffer_size = 32M
innodb_max_dirty_pages_pct = 75
query_cache_type = 1
query_cache_limit = 2M
query_cache_min_res_unit = 2k
query_cache_size = 16M #was 64M, tuning primer said mybe less
tmp_table_size= 64M
max_heap_table_size= 64M
slow-query-log = 1
slow-query-log-file = /var/log/mysql/slow.log
long_query_time = 1
max_connections = 250
join_buffer_size = 1M
key_buffer = 64M
[client]
default-character-set = utf8mb4
[mysqld]
character-set-server = utf8mb4
collation-server = utf8mb4_general_ci
transaction_isolation = READ-COMMITTED
binlog_format = ROW
innodb_large_prefix=on
innodb_file_format=barracuda
innodb_file_per_table=1
default_authentication_plugin=mysql_native_password
max_binlog_size = 134217728
expire_logs_days = 3

I understand you tried made the tests using one big file - this is what I did as well. I have to admit I moved my NC to real PC few weeks ago - it is equipped with Core i3 6th Gen, 16GB RAM, whole Docker Nextcloud is on magnetic HDD (traefik as well) . I believe this more powerful as Odroid but may be I’m wrong. I don’t remember exactly but I think I got 50-70MB/s syncing big files with Qnap NAS (Celeron) with NC19. I didn’t made lot of testing with NC21 on Qnap but from my feeling the sync was same as on 19…

[pepesuriano]

Bump, issue still happening even at NC 23

[wwe]

I’m sorry there is nothing wrong with Nextcloud running as Docker container behind Traefik. Below results from Windows 10 desktop client 3.3.6 against Nextcloud Docker container 22.2.3 running on old Core i3/16GB RAM where container lives on magnetic HDD with ZFS filesystem:

Download from NAS is definitely faster but NC is close enough on hardware limits… 570 Mbps is far more you need for daily usage…