Web interface download speeds slower than expected (15-50MB/s on gigabit connection)

[tali055]

Nextcloud version: 26.0.1
Install Method: From scratch
Operating system and version: Ubuntu 22.04.2 LTS
Apache or nginx version: Apache 2.4.52
PHP version: 8.1 FPM
DB: MariaDB
Hardware Specs:
CPU: AMD Ryzen 5 3600 6c 12t
Memory: 64GB 2666MT/s
Storage: 2x 2TB Toshiba HDD (Nextcloud and DB on 1, data on 2)

The issue you are facing:

When downloading from the Nextcloud web interface I’m seeing only about half the download bitrate that I’m expecting and sometimes even just a 10th or so. Download speeds ramp up to about 55MB/s and hover there. I am downloading singular large zip files for testing this so all of that is sequential.

I have verified the following:

• Hard-drive read and write performance with IO benchmarking and manually copying files from disk 2 (nextcloud data) to disk 1 (nextcloud application and db). Speeds easily reach 190MB/s there
• Bandwidth and connection speed between the server and my home machine from which I’m trying to download using Iperf with on average 970mbit/s speeds. Additionally, SCP and downloading a file directly off a webserver reach the expected gigabit download speeds
• I’m confident I can exclude any hardware bottlenecks, here are some HTOP screenshots
  First one is downloading a large file
  
  
  
  Second one is downloading a file that I assume has been chached (has been downloaded before) since no disk IO is displayed here
  
  
  
  Here are processes running during download sorted by memory usage
  
  

Some additional info:
PHP:
max_execution_time = 7200
memory_limit = 8G
post_max_size = 0 (disabled)
pm = dynamic
pm.max_children = 90
pm.start_servers = 24
pm.min_spare_servers = 12
pm.max_spare_servers = 24
pm.max_spawn_rate = 32 (default)
pm.process_idle_timeout = 10s (default)
pm.max_requests = 0

MariaDB
[server]
skip_name_resolve = 1
[mysqld]
transaction_isolation = READ-COMMITTED
binlog_format = ROW
innodb_buffer_pool_size= 2G
innodb_io_capacity = 4000
join_buffer_size = 214748364
table_definition_cache = 600

Opcache is enabled, APCu is enabled, Redis is used for caching and file locking.

Is this the first time you’ve seen this error? (Y/N): Y

The output of your Nextcloud log in Admin > Logging:
Log level 1 output of starting and cancelling a file download

{"reqId":"pXeDi9bHXnIL0XpwVQfF","level":0,"time":"2023-05-21T09:50:08+00:00","remoteAddr":"IPV6ADDRESS","user":"tali","app":"logreader","method":"GET","url":"/settings/admin/logging","message":"Could not find resource file \"/apps/logreader/css/../js/logreader-main.css\"","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36","version":"26.0.1.1","data":{"app":"logreader"}}
{"reqId":"V2bKHv0AZJbt28KBRT8L","level":1,"time":"2023-05-21T09:50:12+00:00","remoteAddr":"IPV6ADDRESS","user":"tali","app":"no app in context","method":"GET","url":"/apps/files/","message":"Deprecated event type for \\OCP\\Collaboration\\Resources::loadAdditionalScripts: null","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36","version":"26.0.1.1","data":[]}
{"reqId":"oiLZGbu5m1zbK5I2kXUY","level":1,"time":"2023-05-21T09:50:13+00:00","remoteAddr":"IPV6ADDRESS","user":"tali","app":"no app in context","method":"GET","url":"/apps/files/ajax/getstoragestats?dir=%2F","message":"Deprecated event type for \\OCP\\Collaboration\\Resources::loadAdditionalScripts: null","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36","version":"26.0.1.1","data":[]}
{"reqId":"P3B98kUGtAm5cT8j0WKJ","level":1,"time":"2023-05-21T09:50:14+00:00","remoteAddr":"IPV6ADDRESS","user":"tali","app":"no app in context","method":"GET","url":"/apps/files/ajax/getstoragestats?dir=%2FVarious","message":"Deprecated event type for \\OCP\\Collaboration\\Resources::loadAdditionalScripts: null","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36","version":"26.0.1.1","data":[]}
{"reqId":"YIcke9vhwjaPwnq52XJ0","level":1,"time":"2023-05-21T09:50:15+00:00","remoteAddr":"IPV6ADDRESS","user":"tali","app":"no app in context","method":"GET","url":"/apps/files/ajax/getstoragestats?dir=%2FVarious%2FSoftware","message":"Deprecated event type for \\OCP\\Collaboration\\Resources::loadAdditionalScripts: null","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36","version":"26.0.1.1","data":[]}
{"reqId":"SS2LueE0CV5164SocQ31","level":0,"time":"2023-05-21T09:50:26+00:00","remoteAddr":"IPV6ADDRESS","user":"tali","app":"no app in context","method":"GET","url":"/.well-known/webfinger","message":"2 well known handlers registered","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36","version":"26.0.1.1","data":[]}
{"reqId":"gMM57nqeQCybKJFVWzyl","level":0,"time":"2023-05-21T09:50:26+00:00","remoteAddr":"IPV6ADDRESS","user":"tali","app":"no app in context","method":"GET","url":"/.well-known/nodeinfo","message":"2 well known handlers registered","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36","version":"26.0.1.1","data":[]}
{"reqId":"5XH6bRIn2n5cAPiUexUv","level":1,"time":"2023-05-21T09:50:27+00:00","remoteAddr":"IPV6ADDRESS","user":"tali","app":"no app in context","method":"GET","url":"/settings/ajax/checksetup","message":"Deprecated event type for OCP\\IDBConnection::CHECK_MISSING_PRIMARY_KEYS: Symfony\\Component\\EventDispatcher\\GenericEvent is used","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36","version":"26.0.1.1","data":[]}
{"reqId":"5XH6bRIn2n5cAPiUexUv","level":1,"time":"2023-05-21T09:50:27+00:00","remoteAddr":"IPV6ADDRESS","user":"tali","app":"no app in context","method":"GET","url":"/settings/ajax/checksetup","message":"Deprecated event type for OCP\\IDBConnection::CHECK_MISSING_INDEXES: Symfony\\Component\\EventDispatcher\\GenericEvent is used","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36","version":"26.0.1.1","data":[]}
{"reqId":"5XH6bRIn2n5cAPiUexUv","level":1,"time":"2023-05-21T09:50:27+00:00","remoteAddr":"IPV6ADDRESS","user":"tali","app":"no app in context","method":"GET","url":"/settings/ajax/checksetup","message":"Deprecated event type for OCP\\IDBConnection::CHECK_MISSING_COLUMNS: Symfony\\Component\\EventDispatcher\\GenericEvent is used","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36","version":"26.0.1.1","data":[]}
{"reqId":"P6vRJsJ4XK30VwCTclKD","level":0,"time":"2023-05-21T09:50:28+00:00","remoteAddr":"IPV6ADDRESS","user":"tali","app":"logreader","method":"GET","url":"/settings/admin/logging","message":"Could not find resource file \"/apps/logreader/css/../js/logreader-main.css\"","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36","version":"26.0.1.1","data":{"app":"logreader"}}
{"reqId":"zgOK269drtZtyc0LwqPd","level":1,"time":"2023-05-21T09:51:15+00:00","remoteAddr":"IPV6ADDRESS","user":"tali","app":"no app in context","method":"GET","url":"/apps/files/","message":"Deprecated event type for \\OCP\\Collaboration\\Resources::loadAdditionalScripts: null","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36","version":"26.0.1.1","data":[]}
{"reqId":"n9My3xWuRx9zcpcG0QPv","level":1,"time":"2023-05-21T09:51:16+00:00","remoteAddr":"IPV6ADDRESS","user":"tali","app":"no app in context","method":"GET","url":"/apps/files/ajax/getstoragestats?dir=%2F","message":"Deprecated event type for \\OCP\\Collaboration\\Resources::loadAdditionalScripts: null","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36","version":"26.0.1.1","data":[]}
{"reqId":"IrmNhlrfsv84B1l2D96V","level":1,"time":"2023-05-21T09:51:17+00:00","remoteAddr":"IPV6ADDRESS","user":"tali","app":"no app in context","method":"GET","url":"/apps/files/ajax/getstoragestats?dir=%2FVarious","message":"Deprecated event type for \\OCP\\Collaboration\\Resources::loadAdditionalScripts: null","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36","version":"26.0.1.1","data":[]}
{"reqId":"gC0ahtWjPFqHbxaGWAkW","level":1,"time":"2023-05-21T09:51:18+00:00","remoteAddr":"IPV6ADDRESS","user":"tali","app":"no app in context","method":"GET","url":"/apps/files/ajax/getstoragestats?dir=%2FVarious%2FSoftware","message":"Deprecated event type for \\OCP\\Collaboration\\Resources::loadAdditionalScripts: null","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36","version":"26.0.1.1","data":[]}
{"reqId":"vWEvmz11BLdqj5vNYD6D","level":1,"time":"2023-05-21T09:51:32+00:00","remoteAddr":"IPV6ADDRESS","user":"tali","app":"no app in context","method":"GET","url":"/settings/ajax/checksetup","message":"Deprecated event type for OCP\\IDBConnection::CHECK_MISSING_PRIMARY_KEYS: Symfony\\Component\\EventDispatcher\\GenericEvent is used","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36","version":"26.0.1.1","data":[]}
{"reqId":"vWEvmz11BLdqj5vNYD6D","level":1,"time":"2023-05-21T09:51:32+00:00","remoteAddr":"IPV6ADDRESS","user":"tali","app":"no app in context","method":"GET","url":"/settings/ajax/checksetup","message":"Deprecated event type for OCP\\IDBConnection::CHECK_MISSING_INDEXES: Symfony\\Component\\EventDispatcher\\GenericEvent is used","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36","version":"26.0.1.1","data":[]}
{"reqId":"vWEvmz11BLdqj5vNYD6D","level":1,"time":"2023-05-21T09:51:32+00:00","remoteAddr":"IPV6ADDRESS","user":"tali","app":"no app in context","method":"GET","url":"/settings/ajax/checksetup","message":"Deprecated event type for OCP\\IDBConnection::CHECK_MISSING_COLUMNS: Symfony\\Component\\EventDispatcher\\GenericEvent is used","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36","version":"26.0.1.1","data":[]}
{"reqId":"rcTK2zCw6eT3w7UmZDcc","level":1,"time":"2023-05-21T09:55:43+00:00","remoteAddr":"IPV6ADDRESS","user":"tali","app":"no app in context","method":"GET","url":"/apps/files/","message":"Deprecated event type for \\OCP\\Collaboration\\Resources::loadAdditionalScripts: null","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36","version":"26.0.1.1","data":[]}
{"reqId":"YFyEl35uZFnjwdi83MFu","level":1,"time":"2023-05-21T09:55:44+00:00","remoteAddr":"IPV6ADDRESS","user":"tali","app":"no app in context","method":"GET","url":"/apps/files/ajax/getstoragestats?dir=%2F","message":"Deprecated event type for \\OCP\\Collaboration\\Resources::loadAdditionalScripts: null","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36","version":"26.0.1.1","data":[]}
{"reqId":"LOvS3eWBzLquGWJWKJyB","level":1,"time":"2023-05-21T09:55:45+00:00","remoteAddr":"IPV6ADDRESS","user":"tali","app":"no app in context","method":"GET","url":"/apps/files/ajax/getstoragestats?dir=%2FVarious","message":"Deprecated event type for \\OCP\\Collaboration\\Resources::loadAdditionalScripts: null","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36","version":"26.0.1.1","data":[]}
{"reqId":"bgvvAYcc76lDGu9ueAxV","level":1,"time":"2023-05-21T09:55:47+00:00","remoteAddr":"IPV6ADDRESS","user":"tali","app":"no app in context","method":"GET","url":"/apps/files/ajax/getstoragestats?dir=%2FVarious%2FSoftware","message":"Deprecated event type for \\OCP\\Collaboration\\Resources::loadAdditionalScripts: null","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36","version":"26.0.1.1","data":[]}

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => 'octofdhgx91h',
  'passwordsalt' => 'XXXX',
  'secret' => 'XXXX',
  'trusted_domains' =>
  array (
    0 => 'nextcloud.XXXX.com',
  ),
  'datadirectory' => '/data/nextcloud',
  'dbtype' => 'mysql',
  'version' => '26.0.1.1',
  'overwrite.cli.url' => 'https://nextcloud.XXXX.com',
  'htaccess.RewriteBase' => '/',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'tali',
  'dbpassword' => 'XXXX',
  'installed' => true,
  'defaultapp' => 'files',
  'memcache.local' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'filelocking.enabled' => 'true',
  'redis' =>
  array (
    'host' => '/var/run/redis/redis-server.sock',
    'port' => 0,
    'timeout' => 0.0,
  ),
  'simpleSignUpLink.shown' => false,
  'app_install_overwrite' =>
  array (
    0 => 'openotp_auth',
  ),
  'twofactor_enforced' => 'true',
  'twofactor_enforced_groups' =>
  array (
  ),
  'twofactor_enforced_excluded_groups' =>
  array (
  ),
  'updater.secret' => 'XXXX',
  'maintenance' => false,
  'theme' => '',
  'loglevel' => 1,
  'mail_smtpmode' => 'smtp',
  'mail_smtphost' => 'mail.XXXX.com',
  'mail_smtpsecure' => 'tls',
  'mail_smtpauth' => 1,
  'mail_smtpname' => 'littletali',
  'mail_smtppassword' => 'XXXX',
  'mail_from_address' => 'nextcloud',
  'mail_sendmailmode' => 'smtp',
  'mail_domain' => 'XXXX.com',
  'mail_smtpport' => '2525',
  'mail_smtpdebug' => 'true'
);

The output of your Apache/nginx/system log in /var/log/____:

Nothing in error.log, nothing in access.log

other_vhosts_access.log:

nextcloud.example.com:443 IPV6ADDRESS - - [21/May/2023:11:59:37 +0200] "GET /apps/files/?dir=/Various/Software&fileid=4121 HTTP/2.0" 200 12867 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
nextcloud.example.com:443 IPV6ADDRESS - - [21/May/2023:11:59:38 +0200] "GET /ocs/v2.php/search/providers?from=%2Fapps%2Ffiles%2F%3Fdir%3D%2FVarious%2FSoftware%26fileid%3D4121 HTTP/2.0" 200 936 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
nextcloud.example.com:443 IPV6ADDRESS - - [21/May/2023:11:59:38 +0200] "GET /apps/files_sharing/api/externalShares HTTP/2.0" 200 736 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
nextcloud.example.com:443 IPV6ADDRESS - - [21/May/2023:11:59:38 +0200] "GET /ocs/v2.php/apps/text/workspace?path=%2FVarious%2FSoftware HTTP/2.0" 404 835 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
nextcloud.example.com:443 IPV6ADDRESS - - [21/May/2023:11:59:38 +0200] "PROPFIND /remote.php/dav/files/tali/Various/Software HTTP/2.0" 207 2723 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
nextcloud.example.com:443 IPV6ADDRESS - - [21/May/2023:11:59:39 +0200] "GET /apps/files/ajax/getstoragestats?dir=%2FVarious%2FSoftware HTTP/2.0" 200 12840 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
nextcloud.example.com:443 IPV6ADDRESS - - [21/May/2023:11:59:40 +0200] "GET /ocs/v2.php/apps/text/workspace?path=%2FVarious%2FSoftware%2FVST HTTP/2.0" 404 835 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
nextcloud.example.com:443 IPV6ADDRESS - - [21/May/2023:11:59:40 +0200] "PROPFIND /remote.php/dav/files/tali/Various/Software/VST HTTP/2.0" 207 2821 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
nextcloud.example.com:443 IPV6ADDRESS - - [21/May/2023:11:59:40 +0200] "GET /apps/files/ajax/getstoragestats?dir=%2FVarious%2FSoftware%2FVST HTTP/2.0" 200 12847 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
nextcloud.example.com:443 IPV6ADDRESS - - [21/May/2023:11:59:43 +0200] "PROPFIND /remote.php/dav/files/tali/ HTTP/1.1" 207 5655 "-" "Mozilla/5.0 (Windows) mirall/3.8.2stable-Win64 (build 20230516) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)"
nextcloud.example.com:443 IPV6ADDRESS - - [21/May/2023:11:59:42 +0200] "GET /remote.php/webdav/Various/Software/VST/13_03_2022_Waves_v_13_Plugins_only_%26_v_12_%20StudioRack.zip?downloadStartSecret=3y1057kt77c HTTP/2.0" 200 520994280 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
nextcloud.example.com:443 IPV6ADDRESS - - [21/May/2023:12:00:13 +0200] "PROPFIND /remote.php/dav/files/tali/ HTTP/1.1" 207 5656 "-" "Mozilla/5.0 (Windows) mirall/3.8.2stable-Win64 (build 20230516) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)"
nextcloud.example.com:443 IPV6ADDRESS - - [21/May/2023:12:00:39 +0200] "GET /apps/files/api/v1/stats HTTP/2.0" 200 881 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
nextcloud.example.com:443 IPV6ADDRESS - - [21/May/2023:12:00:43 +0200] "PROPFIND /remote.php/dav/files/tali/ HTTP/1.1" 207 5655 "-" "Mozilla/5.0 (Windows) mirall/3.8.2stable-Win64 (build 20230516) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)"
nextcloud.example.com:443 IPV6ADDRESS - - [21/May/2023:12:01:13 +0200] "PROPFIND /remote.php/dav/files/tali/ HTTP/1.1" 207 5654 "-" "Mozilla/5.0 (Windows) mirall/3.8.2stable-Win64 (build 20230516) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)"
nextcloud.example.com:443 IPV6ADDRESS - - [21/May/2023:12:01:39 +0200] "GET /apps/files/api/v1/stats HTTP/2.0" 200 881 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
nextcloud.example.com:443 IPV6ADDRESS - - [21/May/2023:12:01:43 +0200] "PROPFIND /remote.php/dav/files/tali/ HTTP/1.1" 207 5655 "-" "Mozilla/5.0 (Windows) mirall/3.8.2stable-Win64 (build 20230516) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)"
nextcloud.example.com:443 IPV6ADDRESS - - [21/May/2023:12:02:13 +0200] "PROPFIND /remote.php/dav/files/tali/ HTTP/1.1" 207 5655 "-" "Mozilla/5.0 (Windows) mirall/3.8.2stable-Win64 (build 20230516) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)"
nextcloud.example.com:443 IPV6ADDRESS - - [21/May/2023:12:01:52 +0200] "GET /remote.php/webdav/Various/Software/VST/13_03_2022_Waves_v_13_Plugins_only_%26_v_12_%20StudioRack.zip?downloadStartSecret=ez94npij98h HTTP/2.0" 200 1331462614 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
nextcloud.example.com:443 IPV6ADDRESS - - [21/May/2023:12:02:38 +0200] "GET /apps/files/api/v1/stats HTTP/2.0" 200 881 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
nextcloud.example.com:443 IPV6ADDRESS - - [21/May/2023:12:02:43 +0200] "PROPFIND /remote.php/dav/files/tali/ HTTP/1.1" 207 5656 "-" "Mozilla/5.0 (Windows) mirall/3.8.2stable-Win64 (build 20230516) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)"
nextcloud.example.com:443 IPV6ADDRESS - - [21/May/2023:12:03:13 +0200] "PROPFIND /remote.php/dav/files/tali/ HTTP/1.1" 207 5656 "-" "Mozilla/5.0 (Windows) mirall/3.8.2stable-Win64 (build 20230516) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)"
nextcloud.example.com:443 IPV6ADDRESS - - [21/May/2023:12:03:38 +0200] "GET /apps/files/api/v1/stats HTTP/2.0" 200 881 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
nextcloud.example.com:443 IPV6ADDRESS - - [21/May/2023:12:03:43 +0200] "PROPFIND /remote.php/dav/files/tali/ HTTP/1.1" 207 5655 "-" "Mozilla/5.0 (Windows) mirall/3.8.2stable-Win64 (build 20230516) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)"

I tried to supply all the information I could think of, if there’s anything I’ve missed please let me know.

I’m interested to hear your takes on this I’ve already worked my way through 5 google pages trying to find similar issues though with all cases the issue was either pretty apparent and was easily fixed or the situation didn’t apply to my setup.

[jtr]

Some thoughts in no particular order (but I’m afraid no smoking gun):

• I see 96 MB/s (768 Mb/s) for downloads from my main NC host when physically connected via GigE. My main NC host is Ubuntu 22.04 on i7-8700T/16GB/1TB SSD and I’m testing against an NC26/Apache/non-FPM Docker instance.

• Any chance you’re testing via wireless on your home machine? The workstation I’m at (on wi-fi) gets about 16 MB/s (128 Mbit/s) downloads at the moment (and about double that for uploads). These roughly match the currently negotiated wi-fi connect speeds for this device (which also bounces around).

• You don’t appear to be CPU or DISK I/O bound.

• I can’t speak to the iPerf tests without knowing whether it was iperf2 or 3 and what your command lines looked like.

• I also note it appears you’re using IPv6 (at least for the NC connection). Do you know if the iperf and/or SCP tests were using IPv4 or IPv6 paths?

• Any chance you can try some curl tests to eliminate the browser from the picture? e.g.

curl -u USERNAME:PASSWORD "http[s]://nc.domain.com/remote.php/dav/files/USERNAME/bigfile.zip" -o /dev/null

• You seem to have a good head on yourself when it comes to performance tuning. Anything special about your Apache setup you can think of?

• This may sound like a random suggestion - and a counterintuitive one - but disable http2.

[tali055]

Hi jtr, thanks so much for your reply!

It’s good to know that you’re exceeding the speeds I am so I can probably rule out the limitation being the webdav or general nextcloud overhead. Are you also on 26.0.1?

My home machine is definitely tethered, it’s my main work-place and I couldn’t stand wifi connectivity stability for that

I agree that CPU or IO doesn’t seem to be an issue unless my testing is somehow flawed. I am wondering a bit though what exactly is caching these files for download since downloading a cached file results in 0 disk IO, I’m not sure if these HDDs have a flash or nand storage cache, if they did would usage of that be factored into general disk IO? It’s not my browser caching it since the same behaviour is seen in an incognito tab or a different browser.
It could be a memory limit problem with either of the components running nextcloud although I did check php and mariadb and all relevant logs and nothing pointed toward any memory limit being close to saturated.

Regarding iPerf, I used iPerf3, I will post a few additional screenshots of the tests run right now:

• Iperf3 UDP IPv6 Reverse mode iperf3 -c example.com -b 0 -R -u :
  

  

• Iperf3 UDP IPv4 Reverse mode iperf3 -4 -c example.com -b 0 -R -u :
  

  

I just tried your suggestion using curl and just like on the browser it ramps up to about 55-60MB/s and hovers there:

I can’t think of anything special about my apache setup, I’m mostly using a suggested configuration from the Nextcloud installation page that’s a bit adapted to my environment. Here’s the nextcloud virtual host config:

<VirtualHost *:80>
	ServerName nextcloud.example.com

	RewriteEngine on
	RewriteCond %{SERVER_NAME} =nextcloud.example.com
	RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

<VirtualHost *:443>
	SSLEngine on
	SSLCertificateFile      /etc/letsencrypt/live/example.com/fullchain.pem
	SSLCertificateKeyFile   /etc/letsencrypt/live/example.com/privkey.pem

	# enable HTTP/2, if available
	Protocols h2 http/1.1

	# HTTP Strict Transport Security (mod_headers is required) (63072000 seconds)
	Header always set Strict-Transport-Security "max-age=63072000"

	DocumentRoot /var/www/nextcloud/
	ServerName  nextcloud.example.com

	ProxyTimeout 7200
	ProxyPreserveHost On
	ProxyRequests Off

	<Directory /var/www/nextcloud/>
		Require all granted
		AllowOverride All
		Options +FollowSymLinks
		Options MultiViews
		<IfModule mod_dav.c>
			Dav off
		</IfModule>
	</Directory>
	
</VirtualHost>
LimitRequestBody 0
RequestReadTimeout handshake=0 header=0 body=0

SSLCryptoDevice dynamic

# intermediate configuration
SSLProtocol             all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite          ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305
SSLHonorCipherOrder     off
SSLSessionTickets       off

SSLUseStapling On
SSLStaplingCache "shmcb:logs/ssl_stapling(32768)"

Good catch with the HTTP2, I’ve actually tried that already since I was having issues with upload speeds before and for some reason HTTP1 was performing a lot better. What ended up causing that issue was that in the HTTP2 Apache Module, the H2WindowSize parameter default value was too low, 0.06MB to be exact, raising this to a comfortable 10-20MB fixed the upload issues.
Manually disabling HTTP2 for the nextcloud Vhost sadly makes no difference in download speeds.

[jtr]

So I think there may be a misunderstanding here about those iPerf results.

When using UDP, which is a bit unusual to test with honestly unless you have a specific reason for doing so, you won’t get useful results (for the most part) with -b 0. That just tells the client to send at the constant max rate it’s local NIC (mostly) can handle without any regard for feedback from the network. That’s why you’re getting nearly 100% packet loss in your test on the other side. (For the record the results in my test runs are a tad clearer and make this more obvious in the summary… which I think is simply an artifact of the much older iperf3 version you’re using).

Since you’re running in -R (reverse) mode you have to flip which end we’re calling the client, but the details are the same.

I suggest a TCP based test (which is -b 0 / unlimited by default - but that’s okay because TCP has congestion control!) Also HTTP/HTTPS (aka Nextcloud) is all going to be TCP so that’s all you probably care about.

Your results - even with those caveats - actually tell me this:

• your server likely has a ~958 Mbit/s link capacity (i.e. so roughly GigE)
• your client received almost none of the traffic (look at the Lost/Total at the end on the client-side!)
• you’re running a really old version of iPerf3

If you do want to do UDP-based tests, you can’t really do so until you’ve got a really good idea of the sustainable link capacity. And a good starting place is the number you get back from the TCP test. Then you can start bumping the -b X parameter up a bit until you start seeing heavy packet loss/unacceptable jitter/high latency. But, again, I don’t think the UDP test is even what you want in your use case. That’d be more for testing capacity for VOIP or Video streams and such.

Try something like:

iperf3 -c <IP address>
or
iperf3 -c <IP address> -R

Given your host is remote from your workstation, even if you end up having a theoretical link capacity (end to end that is - since that’s all that counts!) of, say, 1000 Mbit/s the actual throughput will be a factor of the delay (latency) too. It’s highly unlikely you’ll be higher than 80% of that figure even under ideal conditions other than on a LAN (I’m generalizing here).

Given your cURL test results, I suspect your iperf results (with TCP!) are going to end up in the 600 Mbit/s to 700 Mbit/s range but that’s mostly a wild guess.

Certainly share your results!

[tali055]

I see the issue now, thank you so much for clearing this up! I’m still learning about networking as I go.
I also didn’t realise the latest iperf Windows version was this old, I ran the tests again on my linux NAS with version 3.7 (latest supported) and in TCP mode. Interestingly I wasn’t able to get iperf3 to run on IPv6.
However, on IPv4 your estimates were spot on for single connection sustained throughput:

With rerunning this a few times it looks like 600-610 Mbits/s. Although this still works out higher than what I’m seeing from Nextcloud, it’s a lot closer.
Out of interest I also ran it with the -P 2 option and here I saw it more or less saturating the link fully:

The server side reported CWND is about 3.15 MBytes for the first test and about 3.08 for the second test.

Does this mean I can still optimize my network settings server-side to provide this full throughput on a single connection or is this simply an inherent limit of sorts?

[tali055]

I’ve got the issue! The limiting factor was indeed the TCP window size, however, not on my server, but on my client. Since the server is located in Helsinki and I’m in Germany, the latency is quite large, to achieve gigabit throughput the server and client TCP window maximums have to be increased.

Now the question is if there’s any way around actually doing that on the client, somehow improving the latency maybe since I could figure out how to increase the TCP window size on my Windows 10 home machine and all my other devices I use, but I’d rather find a more elegant solution that doesn’t end up requiring manual registry fiddling

Edit: FALSE ALARM
This only got iperf3’s speeds to gigabit, the actual download speeds from Nextcloud are still throttling to what they were before…

[jtr]

Keep in mind your benchmarking is getting the benefit of ignoring Slow Start since you’re using -O 3. It’s going to be artificially higher than anything real-world since real transactions can’t do that. Though, admittedly, the longer (in clock time) your connection is, the less of a factor that’ll be.

No matter what, NC is going to be lower than the theoretical maximum due to different overhead for HTTP/HTTPS.

I’d like to get more info on your SCP results. You didn’t share them, but mentioned in your first post that they were faster than NC. If you get a chance to share those it would be interesting.

I suspect that given how large file downloads take place, that other than the initial background bits required (authentication/etc) at the startup of the connection, the performance at this point isn’t NC driven, but is dictated by:

• Apache ( (ingrained + tunable for your use case)
• HTTP/HTTPS (ingrained + tunable for your use case)
• Server TCP stack (ingrained + tunable for your use case)
• Path characteristics between Server and Client (things outside your control including intermediate latency/queues/etc)
• Client TCP stack (ingrained + tunable for your use case)

Your parallel results are interesting and do suggest a possible opportunity.

I’m headed out, but I may look closer at your updates later.

You may find this interesting: https://speed.cloudflare.com/

[tali055]

Yes that’s true! Iperf also doesn’t use TLS so I guess that’s why there’s better throughput. I actually started capturing a few scenarios with Wireshark yesterday and am currently still comparing them seeing if I can find anything interesting, it’s sadly the first time I’m using wireshark so I’m still working on trying to interpret the results but I can also post a few screenshots down here.

Out of curiosity I also tried running the whole thing on Nginx instead of Apache and was sometimes getting about 85-90MB/s download speeds through the web interface which is probably the theoretical maximum. However, that still wasn’t very consistent, it would sometimes only get to about 75 and sometimes it also throttled to 20-40 like Apache does without ever ramping up beyond that.

Here’s an SCP I ran:

Repeated tests all max out at around 49MB/s at the moment.

Here are some Wireshark statistics:

Iperf 3

Apache

Nginx

Sadly these don’t mean very much to me at this point, I will have to dig deep into how to interpret this, if there’s anything to gain from this data at all.

Edit:
I’m still digging through Wireshark and found the option to show the packet length distribution that might be related to the issues I’m seeing or might not.

Apache:

Nginx:

Apparently Apache was sending a lot more smaller packets and I wonder if that’s related to the slower speed

[tali055]

Well look at that:

After switching from HTTP2 to HTTP1 downloads now peak at 100MB/s.

But I wouldn’t want to call the issue fully resolved yet. I’ve been downloading the same file multiple times and sometimes it sustains this speed for prolonged periods of time but almost always it still somehow throttles back, sometimes to 50MB/s, sometimes as low as 10-20.
This is super inconsistent and erratic behaviour that I have no real explanation for…
Besides, I would prefer keeping HTTP2

[tali055]

I’ve run Wireshark again at the server side with the download speeds exhibiting throttling to about 15-20MB/s. The amount of Duplicate ACK messages is pretty staggering. From what I read online it’s very unusual to have this many duplicate ACKs but next to no retransmissions:

[tali055]

I’m ready to throw in the towel on this one.
I’ve ruled out everything that is possible for me to rule out, this issue is only exhibited on Nextcloud, downloading files directly off my apache document root doesn’t have this issue.
PHP doesn’t seem to be the issue either, so I’m ready to call this some issue with my nextcloud installation. But I’m not really excited to re-install it from scratch again…