Nextcloud does not use the full speed of the LAN connection

Nextcloud version: NextCloudHub 8 (29.0.2)
Operating system and version: VM Ubuntu server 24.04 inside Proxmox
Apache or nginx version: default Apache with Dicker AIO. Nginx PM - reverse proxy
PHP version: 8.3.8

The issue you are facing:

Nextcloud is installed in Docker (official AIO image) on an Ubuntu Server virtual machine inside Proxmox. Proxmox, Ubuntu and Docker are installed on an NVME SSD. User files are stored on LVM2 stripe SATA SSD. The read speed of user data tested with dd and hdparm is over 700MB/sec. SATA SSD directly bind to the virtual machine. The connection between the server with Nextcloud and the client PC in the local network is 2.3…2.5 Gbit/s (iperf test).
Server and client PC are connected via 2.5 Gbit switch and Keenetic Ultra router. NAT loopback is enabled on the router by default, so the connection to the NextCloud server is via LAN.
When uploading/downloading files in the local network via the web-interface the speed is no more than 1…1.3 Gbit/s, most often 800-900 Mbit/s. The Windows application for some reason uses the speed of 500-600 Mbps.
I’ve checked virtual machine configuration, various storage settings, network adapter configurations, changed patch cords, changed local DNS settings (in case NAT loopback is down for some reason) - it has no result. The speed of downloading files in the local network is also no more than 1.3 Gbit/s. At the same time, uploading/downloading files to proxmox/ubuntu-server is carried out at full speed - 2.3…2.5 Gbit/s.
How can I configure Nextcloud to upload or download files on my local network at least at 2 Gbps?
Server configuration:
CPU: xeon E5 2640V4 (10 cores)
RAM: 16 Gb (available 15,6 Gb)
NVME storage: samsung 970 evo plus
SATA storage: samsung 870 QVO

870QVO has SLC cache, but the size of the file downloaded or uploaded to nextcloud does not matter, the speed is always ± the same.

Additionally, it is worth mentioning that access from the Internet to nextcloud is performed using Nginx proxy manager, the settings of which correspond to the recommendations (all-in-one/reverse-proxy.md at main · nextcloud/all-in-one · GitHub).

Is this the first time you’ve seen this error? (Y/N): Y

The output of your Nextcloud log in Admin > Logging:

all ok

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'one-click-instance' => true,
  'one-click-instance.user-limit' => 100,
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'maintenance_window_start' => 100,
  'apps_paths' =>
  array (
    0 =>
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 =>
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'appsallowlist' => false,
  'check_data_directory_permissions' => false,
  'memcache.local' => '\OC\Memcache\APCu',
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
    'host' => 'nextcloud-aio-redis',
    'password' => 'psw',
    'port' => 6379,
  ),
  'overwritehost' => 'mydomain',
  'overwriteprotocol' => 'https',
  'passwordsalt' => 'psw',
  'secret' => 'secret',
  'trusted_domains' =>
  array (
    0 => 'localhost',
    1 => 'mydomain',
    2 => '192.168.2.127',
  ),
  'datadirectory' => '/mnt/ncdata',
  'dbtype' => 'pgsql',
  'version' => '29.0.2.2',
  'overwrite.cli.url' => 'mydomain',
  'dbname' => 'nextcloud_database',
  'dbhost' => 'nextcloud-aio-database',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'oc_nextcloud',
  'dbpassword' => 'psw',
  'installed' => true,
  'instanceid' => 'oclw3qayfqrk',
  'maintenance' => false,
  'updatedirectory' => '/nc-updater',
  'loglevel' => '2',
  'log_type' => 'file',
  'logfile' => '/var/www/html/data/nextcloud.log',
  'log_rotate_size' => '10485760',
  'log.condition' =>
  array (
    'apps' =>
    array (
      0 => 'admin_audit',
    ),
  ),
  'preview_max_x' => 8192,
  'preview_max_y' => 8192,
  'jpeg_quality' => 60,
  'enabledPreviewProviders' =>
  array (
    1 => 'OC\\Preview\\Image',
    2 => 'OC\\Preview\\MarkDown',
    3 => 'OC\\Preview\\MP3',
    4 => 'OC\\Preview\\TXT',
    5 => 'OC\\Preview\\OpenDocument',
    6 => 'OC\\Preview\\Movie',
    7 => 'OC\\Preview\\Krita',
    0 => 'OC\\Preview\\Imaginary',
  ),
  'enable_previews' => true,
  'upgrade.disable-web' => true,
  'mail_smtpmode' => 'smtp',
  'trashbin_retention_obligation' => 'auto, 30',
  'versions_retention_obligation' => 'auto, 30',
  'activity_expire_days' => '30',
  'simpleSignUpLink.shown' => false,
  'share_folder' => '/Shared',
  'one-click-instance.link' => 'https://nextcloud.com/all-in-one/',
  'upgrade.cli-upgrade-link' => 'https://github.com/nextcloud/all-in-one/discussions/2726',
  'allow_local_remote_servers' => true,
  'davstorage.request_timeout' => 9800,
  'htaccess.RewriteBase' => '/',
  'dbpersistent' => false,
  'files_external_allow_create_new_local' => true,
  'trusted_proxies' =>
  array (
    0 => '127.0.0.1',
    1 => '::1',
    2 => '192.168.2.18',
    10 => '172.20.0.1/32',
  ),
  'preview_imaginary_url' => 'http://nextcloud-aio-imaginary:9000',
  'memories.exiftool' => '/var/www/html/custom_apps/memories/bin-ext/exiftool-amd64-musl',
  'memories.vod.path' => '/var/www/html/custom_apps/memories/bin-ext/go-vod-amd64',
  'memories.vod.ffmpeg' => '/usr/bin/ffmpeg',
  'memories.vod.ffprobe' => '/usr/bin/ffprobe',
  'memories.vod.disable' => false,
  'memories.video_default_quality' => '-2',
  'twofactor_enforced' => 'false',
  'twofactor_enforced_groups' =>
  array (
    0 => 'group',
  ),
  'twofactor_enforced_excluded_groups' =>
  array (
  ),
  'preview_imaginary_key' => 'key',
  'memories.db.triggers.fcu' => true,
  'defaultapp' => '',
  'auth.bruteforce.protection.enabled' => true,
  'ratelimit.protection.enabled' => true,
  'data-fingerprint' => 'fingerprint',
  'preview_max_memory' => 1280,
  'preview_max_filesize_image' => 500,
  'memories.viewer.high_res_cond_default' => 'always',
);



Hi, see https://github.com/nextcloud/all-in-one?tab=readme-ov-file#how-can-i-access-nextcloud-locally

Thanks for the reply! I saw this section with the information, I will try to do as it says. However, as I wrote in the first post, my router supports NAT loopback, which means a direct local connection to the server if it is on the same network. Also, I configured DNS rewriting in AdGuardhome for exactly local connection to Nextcloud. And this connection is local, because my internet provider doesn’t give me speeds of more than 1 Gbps, and when downloading a file I can clearly see downloads up to 1.3 Gbps. It is highly unlikely that my ISP allows such a speed variation. In any case, in all other tasks I have never seen download speeds from the Internet higher than 1 Gbps.

In addition to the above, I have now disconnected the provider’s optical cable from my media converter, which means that I do not have internet access. Access to the next cloud via the web interface has been preserved, as well as access through the application. However, the download speeds remained unchanged.

To understand how the network is built at home.

  1. The provider’s optical cable connects to the Huawei media converter, the media converter is in bridge mode and is designed for authorization in the provider’s networks and optical signal conversion.
  2. The media converter is connected to the Keenetic Ultra router, which is a DHCP server and manages the entire internal network.
  3. The server and the client PC are connected to another router, which has only 2.5 Gbps connection ports (strictly speaking, there are 2 more SFP connectors, but they are not involved).
  4. The switch from point 3 is connected to the Keenetic Ultra router (the router has 1 port with a speed of 2.5 Gbit/s). Thus, both the server and the client PC have a 2.5 Gbit connection between each other and a 1 Gbit connection to access the Internet.

No, it merely means the public IP is accessible from inside your network. And it means traffic takes a round trip to your router via the LAN then back out to your LAN.

It won’t use your Internet connection - which is good - but you’ll still be using 2x of your LAN capacity. And that assumes your router is capability of doing that loopback at even that level.

Given you’re seeing consistently exactly 50% of your max possible throughout it sounds a bit like that’s the barrier you’re hitting.

Or on one of your other network paths.

What is the layer 3 (IP) topology between NPM and AIO?

Are there any VLANs involved?

What happens when you test using IP addressing only located in exactly the same subnet as your test client w/o NPM in the mix (e.g. http://IP_address:11000 I believe with an AIO deployment).

(You may have to reconfigure Nextcloud a bit to do this test since the overwrite* config parameters may kick the traffic back out to the DNS based names and HTTPS via your Proxy).

The test also be unnecessary if you clarify your internal layer 3 topology.

2 Likes

Hi! Thanks for your reply.

No, it merely means the public IP is accessible from inside your network. And it means traffic takes a round trip to your router via the LAN then back out to your LAN.

Hmm, indeed, when using the tracert command in Windows, only the public IP address appears, not the local one.

What is the layer 3 (IP) topology between NPM and AIO?

If the question is exactly how the network is built between NPM and AIO, then there are no specific settings. NPM is installed in proxmox lxc container, has its own IP address received from the Keenetic router, AIO, installed in virtual machine also has its own IP address received from the same router.
VLAN is not used.
The 2.5 Gb Switch does not interfere with the network in any way and performs the role of switching only.

What happens when you test using IP addressing only located in exactly the same subnet as your test client w/o NPM in the mix (e.g. http://IP_address:11000 I believe with an AIO deployment).

If i using AIO IP without NPM (http://localip:11000) there is redirection to the mydomain, then connection error.

As far as I know, access by ip address is not provided in Nextcloud (I saw this in the documentation)

All of my home services have ip in the same network (like that: proxmox - 192.168.2.2, homeassistant - 192.168.2.3, nextcloud - 192.168.2.4, etc).

Update. A few more details about network configuration.
Port 443 is open in the Keenetic, requests from which are sent to NPM. Inside NPM, all requests for cloud.mydomain.com are sent to the local Nextcloud address, ex. 192.168.2.15:11000.

I am pretty sure that this is what limits your speed. It should use the internal ip-address of your reverse proxy in your home network for the best speed.

1 Like

Still not working properly.
What I tried to do:

  1. I installed Pi-hole instead of Adguard (AdGuard wasn’t working properly for some reason), in Pi-hole I overwrote the DNS record as follows: cloud.mydomain.net → /nginx-proxy-manager-ip/
  2. Once this record was established, the CMD command trecert cloud.mydomain.net pointed directly to the NPM IP address.
  3. When navigating to the Nextcloud IP address, the IP address in the address bar was automatically replaced with cloud.mydomain.net and then my Nextcloud was loaded.
  4. Download/upload speed is still limited to ~1 Gbps. BUT sometimes, the speed jumps up to 2 Gbps for a few seconds and then drops. This behavior happened before, but the duration of these jumps was shorter.
  5. Assuming that there might be a VM configuration error, I installed truenas/freenas, xpenology - in ALL cases the speed of data exchange with services was about 2-2.3 Gbps.
    Obviously there is some limitation specifically in the Nextcloud configuration. And frankly speaking, it’s starting to get annoying. Any other services can use all the bandwidth of my local network, but not Nextcloud.
  6. Assuming that I might have made a mistake in rewriting DNS in Pi-hole, I explicitly specified in the windows hosts file: 192.168.2.4 cloud.mydomain.net. This had no effect whatsoever.
  7. Assuming that somehow the Keenetic router could interfere with the data exchange, I excluded it from the chain (hence excluding Internet access, NAT loopback option and other possible interference) by connecting the server and the client PC directly through the switch. The situation did not change in any way. Nextcloud data download/transfer speed is limited to ~1 Gbps with rare, for a few seconds, jumps up to 2 Gbps.

What else can be done to solve this problem? I don’t feel like changing Nextcloud for another similar open-source service.

By the way, I tried to download 2 files simultaneously (files in zip archive format) and the average download speed was 1.5-2 Gbps. That is, when downloading two files the channel is fully utilized, and when downloading one file - only half of it.
When I download files via Win desktop app the speed is ~400-500 Mbps.
This is some kind of strange magic.

I think I was able to get this to work at 1.7-2 Gbps. But only downloading through the browser. Downloading in the application has a speed of 500 Mbps and I don’t know what to do about it yet. At the same time, the application specifies a local domain like mycloud.local, which according to tracert definitely does not go beyond my network.
Below I will briefly describe my actions, maybe someone will find it useful. I apologize that the actions look haphazard, by this time I was already tired of fighting for speed and acted a bit rambling.

Спойлер
  1. Installed Pi-hole instead of Adguard, made Pi-hole the DNS server for the whole local network.
  2. Rewrote DNS in Pi-hole so that the global domain would map to the IP address of the local proxy.
  3. Added an additional local DNS record of the form mycloud.local, which also maps to the IP address of the reverse proxy. In the reverse proxy I added a record similar to the global domain. It turns out that when querying mycloud.local DNS returns proxy-IP, which means that mycloud.local should be redirected to Nextcloud IP. The settings for this proxy are the same as for the global proxy. The additional domain has been written to eliminate the NAT loopback operation and to make sure that local DNS rewriting works.
  4. Path tracing was done with tracert on the client PC. When tracing mycloud.local - DNS returned the IP address of my reverse proxy. When tracing the global domain cloud.mydomain.com - DNS returned the external IP address (more on this below)
  5. This is probably an important item for Keenetic router owners. In the router’s administrative panel, there is an option to set Traffic Priorities. The priorities that I set up were probably not well designed, and as a result, it could affect the logic of the router as a router. I set all the values to “Default”. Previously, 3 devices on the network had the same “Critical” priority at the same time. Which included the client PC, proxmox server and Nextcloud server.
  6. I reset the network adapters configuration in proxmox. I have 2 network adapters: discrete and integrated. I disabled the discrete one back when I installed proxmox. In this case vbridge is tied to the discrete adapter. Now I have enabled both integrated and discrete, but I have not changed the vbridge settings. (This may play a role, as I have a Machinist motherboard from aliexpress (I know it’s not the best choice, but that’s the way it is at the moment) and it’s not known for sure how good the motherboard firmware is).
  7. I disabled Multiqueue in the virtual machine’s network adapter settings.
  8. After all the above actions, the first connection to any domain (global or local) resulted in high download speed.
  9. Apparently, with the NAT loopback feature, DNS rewriting in Pi-hole is not required after all. I disabled this rewriting, rebooted the server, router, PC, connected to Nextcloud via browser and got the same 2 Gbps.

So, the problem with slow download speeds in the Nextcloud app on Windows remains as it is.
I tried CyberDuck and it works fine, with an average speed of about 1.9 Gbps. However, this app is not quite what I need.
The reasons for the low speed of the NextCloud app are still ongoing…

I am still searching for a solution and have come to an intermediate conclusion.
I installed the second instance of NextCloud by building it from binary files according to the official documentation and with the help of this guide: Nextcloud_Ubuntu/nextcloud.md at main · jameskimmel/Nextcloud_Ubuntu · GitHub.
And it turned out that it works faster, access is possible by ip-address, and both download and upload speeds are 2 Gbps without any jumps or drops. It does not matter how I access NC: by ip address or by my domain (NC is also accessible from the Internet). In this deployment, done according to the official documentation, ssl certificates are obtained on the NC server side using Certbot, and for AIO certificates are handled by Nginx Proxy Manager. I noticed that if for a new NC deployment the certificates are obtained in NPM, the speed matches what I see on AIO. If i switch to Certbot on the same server, everything is fine. So far I can’t find an explanation for this, most likely I’m misconfiguring something in the case of using NPM for both AIO and new install. Although it’s worth noting that NPM is configured according to the official Nextcloud documentation.
Is it possible to somehow configure AIO so that certificate retrieval happens on the same host on which AIO is installed? And NPM would only do proxying, without taking certificates into account.

Yes, AIO does the tls proxying by itself if you set APACHE_PORT=443. However port 443 needs to be forwarded to the server that runs aio in that case.

So, i try something, but its not working for some reason.
This is my home network.

Спойлер

And here my docker-compose file

Спойлер
services:
  nextcloud-aio-mastercontainer:
    image: nextcloud/all-in-one:latest
    init: true
    restart: always
    container_name: nextcloud-aio-mastercontainer 
    volumes:
      - nextcloud_aio_mastercontainer:/mnt/docker-aio-config 
      - /var/run/docker.sock:/var/run/docker.sock:ro 
    ports:
     # - 80:80
      - 8181:8080
     # - 443:443 
    environment: 
     - APACHE_PORT=11000
     - APACHE_IP_BINDING=127.0.0.1
     - NEXTCLOUD_DATADIR=/media/cloud/data/ 
     - NEXTCLOUD_MOUNT=/media/mntnextcloud/ 
     - NEXTCLOUD_UPLOAD_LIMIT=65G 
     - NEXTCLOUD_MAX_TIME=9800 
     - NEXTCLOUD_MEMORY_LIMIT=2048M 
volumes: 
  nextcloud_aio_mastercontainer:
    name: nextcloud_aio_mastercontainer 

The NPM configuration is the same as here: all-in-one/reverse-proxy.md at main · nextcloud/all-in-one · GitHub

When I set APACHE_PORT=443, turn off SSL in NPM, change addressing in NPM to AIO_IP:443 (or to 80 port) - NC won’t open, browser gives error: unable to establish a connection to the site.

Studying, once again, the documentation on organizing local access in AIO I have comments to the documentation:

…the recommended way how to access Nextcloud locally, is to set up a local dns-server like a pi-hole and set up a custom dns-record for that domain that points to the internal ip-address of your server that runs Nextcloud AIO

This cannot be done because the local DNS record does not allow you to specify an access port, which is required if you are setting up AIO behind a reverse proxy. If you do not specify a port, you cannot access the AIO. Just like access to AIO is not possible over a direct hop like 192.168.1.10:11000 via browser. If you don’t install AIO, there is access by IP address. What could be the problem with AIO - I don’t know. There are absolutely no ideas.

The documentation on Local instance AIO clearly states that

...add a custom DNS-record for your domain and overwrite the A-record (and possibly the AAAA-record, too) to point to the private ip-address of your reverse proxy.

This confirms what I said above.

Although AIO seems to be a very convenient installation option, I will have to use the binary option since it works exactly as expected on the local network.
The question remains as to why the speed drops and becomes unstable when accessed by domain name. Most likely, the proposed solutions have not been tested in an environment where the AIO has access to a network greater than 1Gbps.
In addition, I have not seen any topics or reports indicating that AIO can utilize a local channel greater than 1Gbps in a stable manner. All mentions of Nextcloud working with a network greater than 1Gbps are generally not related to the AIO deployment option.

This is the end of my active search for a way to fix this problem. I’m leaving this topic open in case I don’t give up trying in the near future or someone comes who has AIO installed, has local and web access and can use a channel more than 1Gbps via LAN.