No indication of download progress or ETA when donwloading big files from nextcloud AIO

Nextcloud version: Nextcloud AIO v9.3.0
Operating system and version: Debian GNU/Linux 12 (bookworm) x86_64
Apache or nginx version: nginx/1.26.1 as a reverse proxy, Apache in AIO docker

The issue I am facing:

When I download files from my nextcloud instance, no progressing bar is shown in the browser for the download. I see the progress bar going from left to right and back, not indicating the remaining percentage or time. At some point the download completes successfully.

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:

  1. Download big file from nextcloud via web interface in any browser
  2. See no progressing bar
  3. Be annoyed

Has anyone any idea why this might be the case?

Content-Length is passed successfully through the reverse proxy for files put in the AIO webroot, this is not the problem, but i suspect that no Content-Length is provided when downloading files via the web interface, though I don’t know how I would test that.

EDIT: I just found out using tcpdump that the filesize is passed correctly in the Content-Length field when downloading a file from nextcloud. Now I have no clue at all why download progress is not shown.

EDIT2:
I found out: The Content-Length field is (among others) swallowed by the nginx reverse-proxy. This is the reason no progress bar is shown progressing.
this is my active /etc/nginx/conf.d/default.conf:

server {
    listen       80;
    server_name  localhost;

    location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }
}

server {
    listen      443 ssl;

    server_name domain.ddns.net;
    location / {
        proxy_pass http://127.0.0.1:11000; #the host ip address, localhost I assume would just come back to this docker container
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        client_max_body_size 0;

        # Websocket
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
    ssl_certificate /etc/letsencrypt/live/domain.ddns.net/fullchain.pem; # managed by Certbot
 # managed by certbot on host machine
    ssl_certificate_key /etc/letsencrypt/live/domain.ddns.net/privkey.pem;
 # managed by Certbot
}

This is my /etc/nginx/nginx.conf

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;


    ##
    # Connection header for WebSocket reverse proxy
    ##
    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }

    include /etc/nginx/conf.d/*.conf;
}

How do i get nginx to pass (ideally, all) response fields that the nextcloud-AIO-apache gives it, including Content-Length?

Appreciate all and any help,

vin

1 Like

you could try one or both of these:
→ proxy_buffering ←
→ proxy_pass_request_headers ←
in your /etc/nginx/conf.d/default.conf:

        # Disable proxy buffering
        proxy_buffering off;

        # Ensure headers are passed correctly
        proxy_pass_request_headers on;

        # Websocket
        .. / etc. / ..

Then you can simply download a file from your cloud with curl:

curl -I https://domain.ddns.net/s/$token

and check the output to see if the Content-Length header is present.

hth


Much and good luck,
ernolf

1 Like

Thanks for the suggestions, but no luck at all.
I’ve tried:
proxy_buffering off;
proxy_pass_request_headers on;
proxy_pass_header Content-Length;

I’ve tried quoting and double-quoting “Content-Length” nothing changes.
I am sooo close to actually read the nginx source code, because the documentation is kind of garbage.

Hm… This might be a bug in apache: [Bug]: please send content-length header in http stream · Issue #47017 · nextcloud/server · GitHub

Hello szaimen, nice talking to you, I have come across many a post of yours on my nextcloud journey :slight_smile:

I don’t think this is the mentioned apache bug, as the Content-Lengt header field can be observed being sent correctly by the AIO apache when sniffing packets with tcpdump between the reverse proxying nginx and the AIO apache directly on the server:

X-Powered-By: PHP/8.2.21^M
Content-Security-Policy: default-src 'none';^M
Content-Type: application/octet-stream^M
Last-Modified: Sat, 20 Jul 2024 11:07:08 GMT^M
ETag: "3ca4f71c77f7b89fbc592a3d8afabf79"^M
Content-Length: 661651456^M
X-Request-Id: OsiAGB1U6or2N1uy10GN^M
OC-ETag: "3ca4f71c77f7b89fbc592a3d8afabf79"^M
X-Debug-Token: OsiAGB1U6or2N1uy10GN^M
Content-Disposition: attachment; filename*=UTF-8''debian-12.6.0-amd64-netinst.iso; filename="debian-12.6.0-amd64-netinst.iso"^M
X-Accel-Buffering: no^M
^M
ER^H^@^@^@<90><90>^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@3íú<8e>Õ¼^@|ûüf1Ûf1ÉfSfQ^FW<8e>Ý<8e>ÅR¾^@|¿^@^F¹^@^Aó¥êK^F^@^@R´A»ªU1É0öùÍ^Sr^V<81>ûUªu^P<83>á^At^KfÇ^Fó^F´Bë^Uë^B1ÉZQ´^HÍ^S[^O¶Æ@P<83>á?Q÷áSRP»^@|¹^D^@f¡°^GèD^@^O<82><80>^@f@<80>Ç^Bâòf<81>>@|ûÀxpu ú¼ì{êD|^@^@è<83>^@isolinux.bin missing or corrupt.^M
f`f1Òf^C^Fø{f^S^Vü{fRfP^FSj^Aj^P<89>æf÷6è{Àä^F<88>á<88>Å<92>ö6î{<88>Æ^HáA¸^A^B<8a>^Vò{Í^S<8d>d^PfaÃè^^^@Operating system load error...

When I sniff the same answer locally after going through nginx and directly before arriving in my browser it becomes:

HTTP/1.1 200 OK
Server: nginx/1.26.1
Date: Thu, 08 Aug 2024 22:31:09 GMT
Content-Type: application/octet-stream
Transfer-Encoding: chunked
Connection: keep-alive
Content-Disposition: attachment; filename*=UTF-8''debian-12.6.0-amd64-netinst.iso; filename="debian-12.6.0-amd64-netinst.iso"
Content-Security-Policy: default-src 'none';
Etag: "3ca4f71c77f7b89fbc592a3d8afabf79"
Last-Modified: Sat, 20 Jul 2024 11:07:08 GMT
Oc-Etag: "3ca4f71c77f7b89fbc592a3d8afabf79"
Referrer-Policy: no-referrer
Strict-Transport-Security: max-age=31536000;
X-Content-Type-Options: nosniff
X-Debug-Token: OsiAGB1U6or2N1uy10GN
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Request-Id: OsiAGB1U6or2N1uy10GN
X-Robots-Tag: noindex, nofollow
X-Xss-Protection: 1; mode=block

ce4
ER..............................3......|..f1.f1.fSfQ.W....R..|.........K...R.A..U1.0....r...U.u....t.f.....B....1.ZQ....[...@P..?Q..SRP..|...f....D.....f@.....f.>@|..xpu	...{.D|.....isolinux.bin missing or corrupt.
f`f1.f...{f...{fRfP.Sj.j...f.6.{.........6.{....A......{...d.fa....Operating system load error...

So, it seems apache does send the Content-Length correctly.

I have tried putting “SetEnvIf Request_URI “.php$” ap_trust_cgilike_cl” at the beginning of /var/lib/docker/volumes/nextcloud_aio_nextcloud/_data/.htaccess and restarting all containers, but this has not resolved the issue.

If you tell me where else to put “SetEnvIf Request_URI “.php$” ap_trust_cgilike_cl” I will try it, as I’m currently clinging to every straw that could fix this :-/

All right, it looks like this is then indeed an issue of your nginx reverse proxy…

I found this: Why does Nginx remove Content-Length header for chunked content? - Server Fault

So this looks like it should work at least in theory

        proxy_buffering off;
        proxy_pass_request_headers on;
        proxy_pass_header Content-Length;
        gzip off;
        proxy_request_buffering off;
        chunked_transfer_encoding off;

Nope. No luck. I think I will try posting in the nginx mailinglist next.

This is coming out of nginx, unlike it came in:

OK. Nginx support chunkin out of the box since 1.3.9+
It transfers only chunks, which makes it more perfomant but it “eats” the “Content-Length” Header.

These are some of my entry points:

It may be, that I misunderstood some points but as far as I understand, if it would create content-Length Headers for each chunk, it should enter the size of each chunk in it, which is as useles as no content-Length Header:

hth


Much and good luck,
ernolf

1 Like

You could try if you get what you want with

→ chunked_transfer_encoding off ←

just stumbling in the dark :wink:


ernolf

I’ve tried, and indeed it has the effect that

Transfer-Encoding: chunked

is not present in the answer anymore, but it sadly doesn’t make “Content-Length” appear.

OK. There’s updates. I have had a closer look at the tcpdump file that sniffed the traffic before it enters the proxy, and it turns out nextcloud actually starts transferring the file several times, and the one time where all the data of the file is actually transferred has THIS header:

HTTP/1.1 200 OK^M
Content-Disposition: attachment; filename*=UTF-8''debian-12.6.0-amd64-netinst.iso; filename="debian-12.6.0-amd64-netinst.iso"^M
Content-Security-Policy: default-src 'none';^M
Content-Type: application/octet-stream^M
Date: Thu, 08 Aug 2024 22:31:08 GMT^M
Etag: "3ca4f71c77f7b89fbc592a3d8afabf79"^M
Last-Modified: Sat, 20 Jul 2024 11:07:08 GMT^M
Oc-Etag: "3ca4f71c77f7b89fbc592a3d8afabf79"^M
Referrer-Policy: no-referrer^M
Strict-Transport-Security: max-age=31536000;^M
X-Accel-Buffering: no^M
X-Content-Type-Options: nosniff^M
X-Debug-Token: OsiAGB1U6or2N1uy10GN^M
X-Frame-Options: SAMEORIGIN^M
X-Permitted-Cross-Domain-Policies: none^M
X-Request-Id: OsiAGB1U6or2N1uy10GN^M
X-Robots-Tag: noindex, nofollow^M
X-Xss-Protection: 1; mode=block^M
Connection: close^M
Transfer-Encoding: chunked^M

I found out because the poor vim that has the tcpdump open hangs for a few seconds while searching through the whole debian image after finding this header.

So we are back to why the NC apache does not send Content-Length and uses Transfer-Encoding: chunked when serving the download.

Probably because of https://github.com/nextcloud/server/issues/47017#issuecomment-2267102392

Ah Szaimen already linked it above, sorry.

Although I don’t think this is a bug in apache, as this change was made on purpose to mitigate a security issue.

I found out. It works now. It indeed is as simple as editing the file
/var/lib/docker/volumes/nextcloud_aio_nextcloud/_data/.htaccess
to contain:

    <IfModule mod_proxy_fcgi.c>
       # added:
       SetEnv ap_trust_cgilike_cl
       # /added
       SetEnvIfNoCase Authorization "(.+)" HTTP_AUTHORIZATION=$1
    </IfModule>

As far as the /etc/nginx/conf.d/default.conf is concerned, nothing special is needed:

server {
    listen       80;
    server_name  localhost;

    location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }
}

server {
    listen      443 ssl;

    server_name my.domain.net;
    location / {
        proxy_pass http://127.0.0.1:11000; #the host ip address, localhost I assume would just come back to this docker container
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        # if we comment this out, uploads fail
        client_max_body_size 0;

        # Websocket
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
    ssl_certificate /etc/letsencrypt/live/my.domain.net/fullchain.pem; # managed by Certbot
 # managed by certbot on host machine
    ssl_certificate_key /etc/letsencrypt/live/my.domain.net/privkey.pem;
 # managed by Certbot
}

So it indeed was the apache bug szaimen mentioned.
DIACLAIMER: I don’t fully understand what kind of security vulnerabilities this apache option opens my server up to. It might. If you know, please post.

Thanks to everyone that tried to help solve the mystery!

1 Like

Thank you.
That is great. I was not aware of that bug:
I now noticed, that My Apache/2.4.62 does not send the header either. Adding

   SetEnv ap_trust_cgilike_cl

to the apache2.conf (not to <IfModule mod_proxy_fcgi.c> block but global) gives that header back.
Maybe important for others who are missing that header or functionalities depending on it.

Thank you for nailing this out to the extend. :+1:


ernolf

1 Like

Please don’t call it a bug. It was changed on purpose.

Excuse me! It is handled on bugzilla …

… resulting in misbehaviour; no progress indication from big downloads etc., collateral damage at least :wink:

It was a ‘decission’ but not well documented.
Do you know what exactly was the purpose?


ernolf

Well, yes, there’s a bug report that the missing content-length header is a breaking change that was not properly documented. And I agree, this was/is totally unexpected (I dug into the iOS code a few weeks back because I thought it was a bug on our side :confused: ). My point is just that the change itself is not due to a bug, but a decision made as a reaction to some security vulnerability. That’s why calling it a bug might be mis-leading.

2 Likes

I was only missing Content-Length for big downloads, and lots of transfers from the server weren’t omitting it in the first place.
I have just refined the apache .htaccess to contain:

SetEnvIf Request_URI "remote\.php\/dav\/files\/.*" ap_trust_cgilike_cl

in the

<IfModule mod_proxy_fcgi.c>

section.
That does the job too, and is more specific, which probably doesn’t matter much because this, btw, is the vulnerability:

### low: Apache HTTP Server: HTTP Response Splitting in multiple modules ([CVE-2024-24795](https://www.cve.org/CVERecord?id=CVE-2024-24795))

HTTP Response splitting in multiple modules in Apache HTTP Server allows an attacker that can inject malicious response headers into backend applications to cause an HTTP desynchronization attack.

As far as I understand that, it means there is low risk because essentially the attacker must already control a backend application that feeds into apache, in the case of me running a nextcloud instance on a dedicated machine meaning the attacker would have to have access to my server already. Maybe a nextcloud plugin could exploit this vulnerability… I don’t know about the security of these.

EDIT: Now that I think of it, I don’t see a good reason not to include the fix into the AIO per default, or is there?

EDIT, IMPORTANT:
the change to .htaccess has to be reapplied after nextcloud AIO update, as I have just noticed!

1 Like