Help ! Downloads slow to start - Seems to be linked with http2 and/or MPM

Nextcloud version (eg, 12.0.2): 18.0.6
Operating system and version (eg, Ubuntu 17.04): U18.0.4
Apache or nginx version (eg, Apache 2.4.25): Apache/2.4.41
PHP version (eg, 7.1): PHP 7.3.12

Download of large files takes an excrutiating time to START… (transfer rates are fine)

Good day to all,

I’ve been having this problem for quite some time now, and can’t seem to get my head around it.

Downloading any large file (>500Mo) through the web interface, or through a share link, takes an awfully long time to start.

Once “Download” is selected, the browser starts loading, like when you load a large or slow page, and then eventually starts the download after minutes. Sometimes after a minute or two, but for larger files (>2Go) can take about 15 minutes. I haven’t tried for files larger than 4Go, but can only imagine…
Transfer rates are fine once download starts, it’s just the time it takes to start.

This is of course extremely frustrating. And have no idea where to start looking…
Description screenshots with a 1,5GB test file on internal data drive :


Is this the first time you’ve seen this error? (Y/N): N

Steps to replicate it:

  1. Select “download” from three dots action button of a file, or from download button of a share link
  2. Wait for download to start…

HOWEVER :
If I disable (a2dismod) http2, the problem vanishes…

The output of your Nextcloud log in Admin > Logging:

None related (warnings from news app regarding empty streams, and other debug notifications from music app)

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => '---',
  'passwordsalt' => '---',
  'secret' => '---',
  'trusted_domains' => 
  array (
    0 => '---',
    1 => '---',
  ),
  'datadirectory' => '---',
  'dbtype' => 'mysql',
  'version' => '18.0.6.0',
  'overwrite.cli.url' => 'https://---',
  'dbname' => '---',
  'dbhost' => '---',
  'dbport' => '',
  'dbtableprefix' => '---',
  'dbuser' => '---',
  'dbpassword' => '---',
  'installed' => true,
  'maintenance' => false,
  'enable_previews' => true,
  'enabledPreviewProviders' => 
  array (
    0 => 'OC\\Preview\\PNG',
    1 => 'OC\\Preview\\JPEG',
    2 => 'OC\\Preview\\GIF',
    3 => 'OC\\Preview\\BMP',
    4 => 'OC\\Preview\\XBitmap',
    5 => 'OC\\Preview\\Movie',
    6 => 'OC\\Preview\\PDF',
    7 => 'OC\\Preview\\MP3',
    8 => 'OC\\Preview\\TXT',
    9 => 'OC\\Preview\\MarkDown',
    10 => 'OC\\Preview\\TIFF',
  ),
  'filesystem_check_changes' => 0,
  'filelocking.enabled' => 'true',
  'integrity.check.disabled' => false,
  'knowledgebaseenabled' => false,
  'logfile' => '/var/log/nextcloud.log',
  'loglevel' => 2,
  'logtimezone' => 'Europe/---',
  'log_rotate_size' => 104857600,
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'overwriteprotocol' => 'https',
  'preview_max_x' => 1024,
  'preview_max_y' => 768,
  'preview_max_scale_factor' => 1,
  'redis' => 
  array (
    'host' => '/var/run/redis/redis-server.sock',
    'port' => 0,
    'timeout' => 0.0,
 ),
  'quota_include_external_storage' => false,
  'share_folder' => '/',
  'skeletondirectory' => '',
  'theme' => '',
  'trashbin_retention_obligation' => 'auto, 7',
  'updater.release.channel' => 'stable',
  'mysql.utf8mb4' => true,
  'app_install_overwrite' => 
  array (
    0 => 'files_reader',
  ),
  'updater.secret' => '---',
);

The output of your Apache/nginx/system log in /var/log/____:

-IP- - - [10/Jun/2020:21:29:17 +0200] "PROPFIND /remote.php/dav/files/---/requested file HTTP/2.0" 207 4471 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko/20100101 Firefox$
-IP- - - [10/Jun/2020:21:29:17 +0200] "GET /ocs/v2.php/apps/text/workspace?path=%path-to-requested-file HTTP/2.0" 404 182 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko/20100101 Firefox/$
-IP- - - [10/Jun/2020:21:29:18 +0200] "GET /index.php/apps/files/ajax/getstoragestats.php?dir=path-to-requested-file HTTP/2.0" 200 285 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv$
-IP- - - [10/Jun/2020:21:30:36 +0200] "GET /remote.php/webdav/requested-file?downloadStartSecret=--- HTTP/2.0" 200 927091088 "-"$

This 1.4Go file actually took about 5 whole minutes before download actually started...

.htaccess file at nextcloud root :

<IfModule mod_headers.c>
  <IfModule mod_setenvif.c>
<IfModule mod_fcgid.c>
   SetEnvIfNoCase ^Authorization$ "(.+)" XAUTHORIZATION=$1
   RequestHeader set XAuthorization %{XAUTHORIZATION}e env=XAUTHORIZATION
</IfModule>
<IfModule mod_proxy_fcgi.c>
   SetEnvIfNoCase Authorization "(.+)" HTTP_AUTHORIZATION=$1
</IfModule>
  </IfModule>
 
  <IfModule mod_env.c>
Header onsuccess unset Referrer-Policy
Header always set Referrer-Policy "no-referrer"
 
Header onsuccess unset X-Content-Type-Options
Header always set X-Content-Type-Options "nosniff"
 
Header onsuccess unset X-Download-Options
Header always set X-Download-Options "noopen"
 
Header onsuccess unset X-Frame-Options
Header always set X-Frame-Options "SAMEORIGIN"
 
Header onsuccess unset X-Permitted-Cross-Domain-Policies
Header always set X-Permitted-Cross-Domain-Policies "none"
 
Header onsuccess unset X-Robots-Tag
Header always set X-Robots-Tag "none"
 
Header onsuccess unset X-XSS-Protection
Header always set X-XSS-Protection "1; mode=block"
 
SetEnv modHeadersAvailable true
  </IfModule>
 
  # Add cache control for static resources
  <FilesMatch "\.(css|js|svg|gif)$">
Header set Cache-Control "max-age=15778463"
  </FilesMatch>
 
  # Let browsers cache WOFF files for a week
  <FilesMatch "\.woff2?$">
Header set Cache-Control "max-age=604800"
  </FilesMatch>
</IfModule>
<IfModule mod_php7.c>
  php_value mbstring.func_overload 0
  php_value default_charset 'UTF-8'
  php_value output_buffering 0
  <IfModule mod_env.c>
SetEnv htaccessWorking true
  </IfModule>
</IfModule>
<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteCond %{HTTP_USER_AGENT} DavClnt
  RewriteRule ^$ /remote.php/webdav/ [L,R=302]
  RewriteRule .* - [env=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
  RewriteRule ^\.well-known/host-meta /public.php?service=host-meta [QSA,L]
  RewriteRule ^\.well-known/host-meta\.json /public.php?service=host-meta-json [QSA,L]
  RewriteRule ^\.well-known/webfinger /public.php?service=webfinger [QSA,L]
  RewriteRule ^\.well-known/nodeinfo /public.php?service=nodeinfo [QSA,L]
  RewriteRule ^\.well-known/carddav /remote.php/dav/ [R=301,L]
  RewriteRule ^\.well-known/caldav /remote.php/dav/ [R=301,L]
  RewriteRule ^remote/(.*) remote.php [QSA,L]
  RewriteRule ^(?:build|tests|config|lib|3rdparty|templates)/.* - [R=404,L]
  RewriteCond %{REQUEST_URI} !^/\.well-known/(acme-challenge|pki-validation)/.*
  RewriteRule ^(?:\.|autotest|occ|issue|indie|db_|console).* - [R=404,L]
</IfModule>
<IfModule mod_mime.c>
  AddType image/svg+xml svg svgz
  AddEncoding gzip svgz
</IfModule>
<IfModule mod_dir.c>
  DirectoryIndex index.php index.html
</IfModule>
AddDefaultCharset utf-8
Options -Indexes
<IfModule pagespeed_module>
  ModPagespeed Off
</IfModule>
 
ErrorDocument 403 //
ErrorDocument 404 //

NC Data drive, Internal Sata drive, 4To, Ext4 formatted, mounted with fstab as such :

UUID=0d39d317-6a21-4860-b1c4-5e317ebc561e /media/INT4To auto nosuid,nodev,nofail,x-gvfs-show,x-gvfs-name=INT4To 0 0

Would be very grateful to any enlighted soul to help me out on this one ! :wink:

When downloading multi-GB files on remote locations, I noticed it takes 1-2 minutes to reach the proper download speed (whatever it is). But the download starts right away, albeit slow. And this “transition” period does not really depend on the file size, it being 1GB or 5GB…

Are you sure it is actually doing nothing “about 5 whole minutes”…?
Have you tried different browsers, cleaning cache, etc…?

What does “the browser starts loading” mean?

Where is your NC instance located?

There is a “popular” problem when uploading big files - “Error when assembling chunks…”
That means big files are split in parts (chunks), uploaded in sequence and combined again.
If re-combining does not end within the allotted time - the error message…

Could it be that downloading big files follows the same “strategy”?

What is the CPU on your NC server?
Have a look at its utilization level when the system is “waiting” for the download to start…

What type of storage is used?

1 Like

Many thanks for your input, I will check and test cpu ratings while trying to download.
Great food for thought. Thank you. Will report here…

Ok so I do have a big increase in cpu load while browser is hanging waiting for download to start.
My datadog agent reports it’s not quite à 100% spike, but it still is a notable load.
Strangely the load only stops after the file has finished downloading.
In the datadog screenshot, you can see the cpu load at bottom left corner, and transfer rate at the top (scale of tranfer rate is not true, it is divided by 10)…
File was 1.3Go, took 3 minutes for download to start.
Using Firefox, there are these 3 dots moving in the tab name while waiting for download to start (same behavior as when waiting for a large page to load)…
Will test more tomorrow and try to be more precise ! :wink:

You can see below download started shortly after 22H00, while cpu spike started before (when download request was made around 21H57)

I must add that all those large files have been uploaded to NC with the flowupload app…
Reading your comment about the chuncks, makes me want to test if I have the same behavior with a large file I will internally copy to the NC storage (ext4 formatted HDD), and reveal with an occ scan command…

The high-load CPU time covers the “preparation” phase and download (3min + 3min)
image
While the I/O jump is only for the download phase…
image

Could it be storage related? Your data partition is not on an external USB drive by any chance?

1 Like

That’s correct (prep+dl).

No the hard drive is plugged to the mother board through a sata cable.
Disk is formatted in ext4. mounted through fstab with :
UUID=0d39d317-6a21-4860-b1c4-5e317ebc561e /media/INT4To auto nosuid,nodev,nofail,x-gvfs-show,x-gvfs-name=INT4To 0 0

Also…
Just did the test I wanted to check :

  • Copied my 1.3Go test file to the NC data drive through ssh
  • gave it proper ownership permissions
  • Revealed it with the occ files:scan command
  • requested a download of the file through NC WebUI

=> Same cpu spike, same 3 minutes to wait before download starts…

So we can rule out a problem with the flowupload app, and not so sure about “chuncks” problem anymore…

Could it be something in php ?

PHP config seems fine…
I have no idea where to look…

Hi,
And again thank you for your input.

I’ve conducted yet another test today that seems to show that my server copies the file to another location before serving it…

Here is the test :
I have an external storage (s3 type) configured in my NC instance.

  • I copied my test file (1.3Go) onto that external storage.
  • I requested a download of that file through my NC WebUI
  • It took again several minutes to start
  • My datadog agent shows the file was first downloaded to my server before serving it

Could it be that large files are systematically copied from storage to system drive before they are served ?
Where could this configuration reside ? Apache ? PHP ? (but there is no temp folder configured in php)

Datadog screenshot : top part is io, blue line is download, red line is upload… Bottom graph shows write activity on internal system drive (not NC data drive)…

Here is the detail of the .htaccess file in nextcloud dir :
Anything seem off ?

<IfModule mod_headers.c>
  <IfModule mod_setenvif.c>
<IfModule mod_fcgid.c>
   SetEnvIfNoCase ^Authorization$ "(.+)" XAUTHORIZATION=$1
   RequestHeader set XAuthorization %{XAUTHORIZATION}e env=XAUTHORIZATION
</IfModule>
<IfModule mod_proxy_fcgi.c>
   SetEnvIfNoCase Authorization "(.+)" HTTP_AUTHORIZATION=$1
</IfModule>
  </IfModule>
 
  <IfModule mod_env.c>
Header onsuccess unset Referrer-Policy
Header always set Referrer-Policy "no-referrer"
 
Header onsuccess unset X-Content-Type-Options
Header always set X-Content-Type-Options "nosniff"
 
Header onsuccess unset X-Download-Options
Header always set X-Download-Options "noopen"
 
Header onsuccess unset X-Frame-Options
Header always set X-Frame-Options "SAMEORIGIN"
 
Header onsuccess unset X-Permitted-Cross-Domain-Policies
Header always set X-Permitted-Cross-Domain-Policies "none"
 
Header onsuccess unset X-Robots-Tag
Header always set X-Robots-Tag "none"
 
Header onsuccess unset X-XSS-Protection
Header always set X-XSS-Protection "1; mode=block"
 
SetEnv modHeadersAvailable true
  </IfModule>
 
  # Add cache control for static resources
  <FilesMatch "\.(css|js|svg|gif)$">
Header set Cache-Control "max-age=15778463"
  </FilesMatch>
 
  # Let browsers cache WOFF files for a week
  <FilesMatch "\.woff2?$">
Header set Cache-Control "max-age=604800"
  </FilesMatch>
</IfModule>
<IfModule mod_php7.c>
  php_value mbstring.func_overload 0
  php_value default_charset 'UTF-8'
  php_value output_buffering 0
  <IfModule mod_env.c>
SetEnv htaccessWorking true
  </IfModule>
</IfModule>
<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteCond %{HTTP_USER_AGENT} DavClnt
  RewriteRule ^$ /remote.php/webdav/ [L,R=302]
  RewriteRule .* - [env=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
  RewriteRule ^\.well-known/host-meta /public.php?service=host-meta [QSA,L]
  RewriteRule ^\.well-known/host-meta\.json /public.php?service=host-meta-json [QSA,L]
  RewriteRule ^\.well-known/webfinger /public.php?service=webfinger [QSA,L]
  RewriteRule ^\.well-known/nodeinfo /public.php?service=nodeinfo [QSA,L]
  RewriteRule ^\.well-known/carddav /remote.php/dav/ [R=301,L]
  RewriteRule ^\.well-known/caldav /remote.php/dav/ [R=301,L]
  RewriteRule ^remote/(.*) remote.php [QSA,L]
  RewriteRule ^(?:build|tests|config|lib|3rdparty|templates)/.* - [R=404,L]
  RewriteCond %{REQUEST_URI} !^/\.well-known/(acme-challenge|pki-validation)/.*
  RewriteRule ^(?:\.|autotest|occ|issue|indie|db_|console).* - [R=404,L]
</IfModule>
<IfModule mod_mime.c>
  AddType image/svg+xml svg svgz
  AddEncoding gzip svgz
</IfModule>
<IfModule mod_dir.c>
  DirectoryIndex index.php index.html
</IfModule>
AddDefaultCharset utf-8
Options -Indexes
<IfModule pagespeed_module>
  ModPagespeed Off
</IfModule>
 
ErrorDocument 403 //
ErrorDocument 404 //

Well, that explains why the “waiting”/downloading times are close - it’s the same process done twice…

Could be.
Or maybe this is done with all files, just not noticeable on smaller ones…

I have never used S3 (type) storage but this workflow is not unheard of.

Consider 7zip.
If you open a (big) zipped file/folder and drag the file from the 7zip window to a non-C: location, it will first copy it into a temp folder on the system drive and after that move it to destination…

But I’d expect this to be configurable…

EDIT:
Do you have the S3 storage encrypted?
Then the first “waiting” step could be decryption…

This article (at the end) talks about /tmp folders when uploading

Thank you Henry.
This is indeed a very interesting documentation !

It doesn’t however fit my use case, my S3 external storage, is not -and will never be- my primary storage (also it is not encrypted). Primary storage will remain internal HDD. And since my problem is existing for this storage, solution will not come from the S3 external storage configuration.

I’m really very grateful for the time you have spent looking at my issue and configuration.

Problem unfortunately is still there, and my research of a solution ongoing !..

I think “primary storage” here means the location of the data, not the OS Nextcloud runs on.
I have multiple instances of NC and always run the OS+NC on a mirrored SSD (50-100GB) and the data folder on HDD. I believe this is a normal setup…

There are few old but ongoing “S3 is slow on Nextcloud” discussions on GitHub…

Yes I agreee with you.
My setup is :
sda/ is Ubuntuserver sysdrive, hosted NC instance in /var/www/nextcloud
sdb/ is NC data drive (4To)
sdc/ is a backup drive mirror of sdb/ copied everynight from sdb with rclone sync through a cron job.
S3 is extra external storage for archives.

I have no particular problems with S3.
My problem is a problem because I want to be able to send share links for specific files (from main data drive sdb/) to specific people, and these people of course don’t understand why their download takes so long to start… :slightly_smiling_face:
So right now I CAN’T share files because of this issue … :confused:

1 Like

@nextcloudCommunity !!
Anyone to take a little time to take a look at this issue please ? :slight_smile:
#Halp!

This is a long shot - not exactly same (mis)behavior - but might be worth looking into


Taken from here
2 Likes

Wow ! Long shot indeed. Thank you again Henry for looking into this.
I’m not running an arm architecture (not RK3399 cpu, not Rpi).
My machine is a Dell x86 machine with 82579LM chip for network.
But I’ll have a deeper (and very curious) look into this … Thanks :slight_smile: