Dual reverse proxy returns a blank page

Moving this discussion from github.

Issue

TL;DR
Running Nextcloud behind dual reverse proxy through a VPS returns a blank 200 OK page.

I followed the steps on the reverse proxy documentation and noticed:

If you need HTTPS between Nextcloud and the reverse proxy because it is running on a different server in the same network, simply add another reverse proxy to the chain that runs on the same server like AIO and takes care of HTTPS proxying (most likely via self-signed certificates). Another option would be to create a VPN between the server that runs AIO and the server that runs the reverse proxy which takes care of encrypting the connection.

Everytime i open my subdomain.domain through browser or curl -vk it returns 200 OK with content-length: 0. However, accessing the server with curl -vk http://10.8.0.1:12000 from the VPS works perfectly. I’ve attached the outputs below. My guess is probably headers from either reverse proxy since I’m not familiar with those or TLS issues from Caddy reverse proxy.

I have other services that run through the same DNS, VPS & Nginx, Wireguard connection, and Docker perfectly fine. Note that I didn’t need to use Caddy for other services. I use Caddy as an additional reverse proxy on my Nextcloud server or Nextcloud would not work.

Sorry if it’s a bunch of logs and configs all at once, but I figured it would be necessary. I’ve looked everywhere but could not find any solution & would gladly send any more logs/info if needed!

My setup

  • DNS: Cloudflare (no Cloudflare proxy, rocket loader off for subdomain).
  • VPS: DigitalOcean Ubuntu running Nginx v1.26.0 reverse proxy (UFW ports opened: 80, 443, 51820, SSH port). TLS from certbot.
  • Wireguard on port 51820:
    • VPS module v1.0.20210606 & tools v1.0.20210914 with IP: 10.8.0.1.
    • server v0.5.3 with IP 10.8.0.2.
  • Server: Nextcloud AIO v10.2.0 on Docker Desktop v4.37.1 (178610) Windows with Caddy v2.8.4 reverse Proxy as Windows service.

Nginx Config on VPS

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

server {
        if ($scheme = "http") {
                return 301 https://$host$request_uri;
        }
        if ($http_x_forwarded_proto = "http") {
                return 301 https://$host$request_uri;
        }

        server_name <redacted subdomain.domain>;
        listen 443 ssl;
        listen [::]:443 ssl;
        http2 on;

        client_max_body_size 10G;
        client_body_buffer_size 512k;
        proxy_read_timeout 86400s;

        # Debugging
        access_log  /var/log/nginx/files_access.log;
        error_log /var/log/nginx/files_error.log debug;

        location / {
                proxy_pass http://10.8.0.2:12000; # I also tried with https here

                proxy_ssl_verify off;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Port $server_port;
                proxy_set_header X-Forwarded-Scheme $scheme;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header Host $host;
                proxy_set_header Early-Data $ssl_early_data;

                chunked_transfer_encoding on;
                proxy_buffering off;
                proxy_request_buffering off;

                # Websocket
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection $connection_upgrade;
        }
        # SSL config
        ssl_certificate /etc/letsencrypt/live/<redacted subdomain.domain>/fullchain.pem; # managed by Certbot
        ssl_certificate_key /etc/letsencrypt/live/<redacted subdomain.domain>/privkey.pem; # managed by Certbot
        ssl_dhparam /etc/dhparam; # curl -L https://ssl-config.mozilla.org/ffdhe2048.txt -o /etc/dhparam
        ssl_early_data on;
        ssl_session_timeout 1d;
        ssl_session_cache shared:SSL:10m;
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ecdh_curve x25519:x448:secp521r1:secp384r1:secp256r1;
        ssl_prefer_server_ciphers on;
        ssl_conf_command Options PrioritizeChaCha;
        ssl_ciphers TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:>
}

Wireguard Config

on VPS

[Interface]
Address = 10.8.0.1/24
MTU = 1400
SaveConfig = true
ListenPort = 51820
PrivateKey = <Private Key>

[Peer]
PublicKey = <Public Key>
AllowedIPs = 10.8.0.0/24, fdbe:5d93:caca::/64
Endpoint = <Server public IP behind CGNAT>
PersistentKeepalive = 25

on Server

[Interface]
PrivateKey = <Private Key>
Address = 10.8.0.2/24, fdbe:5d93:caca::1/64
MTU = 1400
Table = off

[Peer]
PublicKey =  <Public Key>
AllowedIPs = 0.0.0.0/0, ::/0
Endpoint = <VPS public IP>
PersistentKeepalive = 25

Caddy Config

on windows service where docker AIO is also running

{
	email <Redacted e-mail>
	debug
}

# for local access
https://localhost:11000 {
	redir /.well-known/carddav /remote.php/dav/ 301
	redir /.well-known/caldav /remote.php/dav/ 301
	reverse_proxy http://127.0.0.1:11000 {
		# Pass the original Host header
		header_up Host {host}
		header_up X-Real-IP {remote}
	}
	tls internal
}

http://10.8.0.2:12000 { # I also tried with https here
	redir /.well-known/carddav /remote.php/dav/ 301
	redir /.well-known/caldav /remote.php/dav/ 301
	reverse_proxy http://127.0.0.1:11000 {
		# Pass the original Host header
		header_up Host {host}
		header_up X-Real-IP {remote}
	}
}

AIO Docker-compose

services:
  nextcloud-aio-mastercontainer:
    image: nextcloud/all-in-one:latest
    init: true
    restart: always
    container_name: nextcloud-aio-mastercontainer
    volumes:
      - nextcloud_aio_mastercontainer:/mnt/docker-aio-config
      - /var/run/docker.sock:/var/run/docker.sock:ro
    network_mode: bridge
    ports:
      - 8080:8080
    environment:
      APACHE_PORT: 11000
      APACHE_IP_BINDING: 127.0.0.1
      NEXTCLOUD_DATADIR: /run/desktop/mnt/host/d/ncdata
      NEXTCLOUD_UPLOAD_LIMIT: 50G
      NEXTCLOUD_MAX_TIME: 360000
      NEXTCLOUD_MEMORY_LIMIT: 1024M
      NEXTCLOUD_STARTUP_APPS: deck twofactor_totp tasks calendar contacts note
      NEXTCLOUD_ADDITIONAL_APKS: imagemagick tesseract-ocr tesseract-ocr-data-eng ffmpeg libva-utils libva-vdpau-driver mesa-va-gallium
      NEXTCLOUD_ADDITIONAL_PHP_EXTENSIONS: imagick
      NEXTCLOUD_ENABLE_DRI_DEVICE: true
      ENABLE_NVIDIA_GPU: true
      SKIP_DOMAIN_VALIDATION: true
      AIO_COMMUNITY_CONTAINERS: pi-hole jellyfin local-ai   

Nextcloud Config

'trusted_proxies' => ['127.0.0.1', '::1', '<Docker WSL2 Subnet>', 'fdbe:5d93:caca::/64', '10.8.0.1', '10.8.0.2', '<Redacted VPS public IP>'],
'overwritehost' => '<Redacted subdomain.domain>',
'overwriteprotocol' => 'https',
'overwrite.cli.url' => 'https://<Redacted subdomain.domain>/'
'trusted_domains' => '[host.docker.internal, <Redacted subdomain.domain>]'

Debug Logs

Nginx files_error.log
Caddy.log
Nextcloud-aio-apache.log
Nextcloud-aio-mastercontainer.log
Curl from VPS.log
Wireguard on VPS.log

Thank you in advance! Would really appreciate any tips or advice.

There was a similar topic Probably DNS help with NC Docker + Collabora + Wireguard tunnel please check if you can learn there. This one runs a “normal” Docker which is IMHO easier to understand… but should similar.

you are welcome :handshake: better more logs and config than less.

I’m still having hard time to understand your config. would be great you create a drawing to explain such complex setup.

what I get is (correct me it I’m wrong)

  • your docker host has 10.8.0.2
    • on this host you run AiO and Caddy (what is purpose for Caddy?)
  • your VPS has Wireguard tunnel with docker host

I understand the Caddy config like this

  • listen on http://10.8.0.2:12000 → reverse_proxy http://127.0.0.1:11000
  • listen on https://localhost:11000 → reverse_proxy http://127.0.0.1:11000
  • this makes me think there is an address conflict with Caddy and AiO Apache on port 11000

I have the feeling Caddy is more or less useless in this setup isn’t it? I would setup VPS Nginx to connect with AiO Apache on :11000

2 Likes

Hi!
First of all thank you so much for helping. It works now! I’ll explain in detail so whoever stumbles upon this problem might find it helpful.

TL;DR
I’m 90% certain the problem is #2 and #3. Things that changed:

  1. Updated Nextcloud v10.2.0 → v10.3.0.
  2. Issues with Caddy & Nextcloud conflict on :11000 port binding.
  3. APACHE_IP_BINDING to 0.0.0.0. I have tried this in the past with and without Caddy, maybe it wasn’t applied properly.

Here’s a diagram I created on my faulty setup:


In theory should it work though? I am just curious what’s with the content-length: 0.

To answer your questions:

  • Caddy reverse proxy:
    1. is used for local access mainly. I want to reach my own server when I’m at home without reaching the internet.
    2. Secondly, my VPS responds with connection refused. That’s why I thought Caddy was necessary. Please see my other diagram below with Caddy from external access removed.

The issue I think is Apache is not binding to the right IP or Caddy port :11000 conflict. After changing the APACHE_IP_BINDING env to 0.0.0.0 again and removing that part from Caddy works flawlessly.

Once again, thank you for your help!

1 Like

very cool, thank you for detailed drawing, I think this really helps others :handshake:

1 Like

This topic was automatically closed 8 days after the last reply. New replies are no longer allowed.