Bad Gateway 502 with Nginx in Docker, used as reverse proxy in front of Nextcloud in Docker

I configured my Nginx docker instance to pass all requests made to https://my.public.domain/nextcloud to the nextcloud docker instance in the background.

When I call the url described above (of course by using the real domain name), I get an 502 Bad Gateway status.

The logs of nextcloud, retrieved using docker logs nextcloud-server, show nothing at all that seems to be related to the request. It seems like nextcloud does not react at all.

The logs of nginx, retrieved using docker logs nginx show these lines:

2019/12/26 19:05:59 [error] 6#6: *80 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xxx.xxx.xx, server: my.public.domain, request: "GET /nextcloud/ HTTP/2.0", upstream: "http://127.0.0.1:9001/", host: "my.public.domain"
xx.xxx.xxx.xx - - [26/Dec/2019:19:05:59 +0000] "GET /nextcloud/ HTTP/2.0" 502 552 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/79.0.3945.79 Chrome/79.0.3945.79 Safari/537.36" "-"

Expected Behaviour: Be able to reach the web interface of nextcloud. (See login page etc.)


Technical Details:

Nextcloud version: 17.0.2.1
Operating system and version: Ubuntu Server 18.04.3 LTS
Apache or nginx version: latest
PHP version: PHP/7.3.13

Is this the first time you’ve seen this error? : Yes

Steps to replicate it:

  1. Install Nextcloud as described in “Installation” section further below.
  2. Set up Nginx as docker container on same host as nextcloud with valid SSL certificates, using the configuration file provided further below in “Nginx-Configuration”.
  3. Configure Nextcloud as shown in “config.php”.

The output of your Nextcloud log in Admin > Logging: (currently not accessible, will be provided as soon as needed)

The output of your config.php file in /path/to/nextcloud

<?php
$CONFIG = array (
  'htaccess.RewriteBase' => '/',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'apps_paths' => 
  array (
    0 => 
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 => 
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'instanceid' => 'whatever',
  'passwordsalt' => 'whatever',
  'secret' => 'whatever',
  'trusted_domains' => 
  array (
    0 => '192.168.xxx.xxx',
    1 => '127.0.0.1',
    2 => 'my.public.domain',
  ),
  'trusted_proxies' => ['127.0.0.1', 'localhost'],
  'datadirectory' => '/var/www/html/data',
  'dbtype' => 'pgsql',
  'version' => '17.0.2.1',
  'overwrite.cli.url' => 'http://127.0.0.1:9001',
  'overwritewebroot' => '/',
  //'overwriteprotocol' => 'https',
  //'overwritehost' => 'my.public.domain',
  'dbname' => 'nextcloud',
  'dbhost' => 'database',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'oc_ncadmin',
  'dbpassword' => 'whatever',
  'installed' => true,
);

Nginx-Configuration:

server {
    listen      80;
    listen [::]:80;
    server_name my.public.domain;

    location / {
        rewrite ^ https://$host$request_uri? permanent;
    }

    #for certbot challenges (renewal process)
    location ~ /.well-known/acme-challenge {
        allow all;
        root /data/letsencrypt;
    }
}

#https://my.public.domain
server {
    server_name my.public.domain;
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_tokens off;

    ssl_buffer_size 8k;
    ssl_dhparam /etc/ssl/certs/dhparam-2048.pem;

    ssl_protocols TLSv1.2 TLSv1.1 TLSv1;
    ssl_prefer_server_ciphers on;
    ssl_ciphers ECDH+AESGCM:ECDH+AES256:ECDH+AES128:DH+3DES:!ADH:!AECDH:!MD5;

    ssl_ecdh_curve secp384r1;
    ssl_session_tickets off;

    # OCSP stapling
    ssl_stapling on;
    ssl_stapling_verify on;
    resolver 8.8.8.8 8.8.4.4;

    ssl_certificate /etc/letsencrypt/live/my.public.domain/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/my.public.domain/privkey.pem;

    root /usr/share/nginx/html;
    index index.html;

    location /nextcloud/ {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://127.0.0.1:9001/;
    }
}

Installation:

I have Nextcloud running in a docker container with PostgreSQL database (in a seperate container). For installation I followed the guide provided on the docker hub page of Nextcloud.

For nextcloud I created an own Dockerfile, which installs some depedencies. This Dockerfile is used in the docker-compose.yml. The created docker image is tagged “xxxx/nextcloud”.

I set up my nextcloud server using ssh while being within the LAN of the server, which is located at my home.


Docker setup:

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                      NAMES
6c41eacea755        nginx:latest        "nginx -g 'daemon of…"   About an hour ago   Up 59 minutes       0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   nginx
108f037f2f47        xxxx/nextcloud     "/entrypoint.sh apac…"   About an hour ago   Up About an hour    127.0.0.1:9001->80/tcp                     nextcloud-server
992102960b40        postgres            "docker-entrypoint.s…"   About an hour ago   Up About an hour    5432/tcp                                   nextcloud-database

what happens when you curl -v http://127.0.0.1:9001/?

that’s not related to your problem, but use the internal docker network to connect to your nextcloud container. you don’t need to expose port 9001:80. just use proxy_pass http://nextcloud-server/;.
same applies to the connection to the database.

and you have to set trusted_proxy in the config.php. you’ll get a warning.
https://docs.nextcloud.com/server/17/admin_manual/configuration_server/reverse_proxy_configuration.html?highlight=trusted_proxy

Thank you very much for the quick reply @Reiner_Nippes. :slight_smile:

Output of curl -v http://127.0.0.1:9001/:

user@myserver:~$ curl -v http://127.0.0.1:9001/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 9001 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:9001
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 302 Found
< Date: Fri, 27 Dec 2019 12:15:17 GMT
< Server: Apache/2.4.38 (Debian)
< X-Powered-By: PHP/7.3.13
< Set-Cookie: ocrk8uu7f39e=whatever; path=/; HttpOnly
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Set-Cookie: oc_sessionPassphrase=whatever; path=/; HttpOnly
< Content-Security-Policy: default-src 'self'; script-src 'self' 'nonce-endDa2MzdzBxOHpLa1E3eEJCbkM0TzFNZTc2OTZPMUZGU2g5Y1hxOHkzZz06OTFmSUFDOU13WWk4NkhyR2QzeUhoNHM5RXBYZTBJb3JMVTQrSXhmT2sxTT0='; style-src 'self' 'unsafe-inline'; frame-src *; img-src * data: blob:; font-src 'self' data:; media-src *; connect-src *; object-src 'none'; base-uri 'self';
< Referrer-Policy: no-referrer
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Frame-Options: SAMEORIGIN
< X-Permitted-Cross-Domain-Policies: none
< X-Robots-Tag: none
< X-XSS-Protection: 1; mode=block
< Set-Cookie: nc_sameSiteCookielax=true; path=/; httponly;expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax
< Set-Cookie: nc_sameSiteCookiestrict=true; path=/; httponly;expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict
< Location: http://127.0.0.1:9001/login
< Content-Length: 0
< Content-Type: text/html; charset=UTF-8
< 
* Connection #0 to host 127.0.0.1 left intact

When curl-ing the redirected page I receive HTML code, which is probably the desired result.


Secondly, regarding the internal docker network. To be clear about what you mean: In my docker-compose.yml of my nextcloud-server/database setup I remove the following part:

    ports:
      - 127.0.0.1:9001:80

Will this expose port 80 of the nextcloud server only within the docker network, which makes it accessible to nginx using the internal hostname “nextcloud-server”?

Because right now, when I edit the nginx config to be like this:

proxy_pass http://nextcloud-server/;

and also edit the config.php of Nextcloud to be like this:

  'trusted_domains' =>
  array (
    0 => '192.168.xxx.xxx',
    1 => '127.0.0.1',
    2 => 'my.public.domain',
    3 => 'nginx',
  ),
  'trusted_proxies' => ['127.0.0.1', 'localhost', 'nginx'],
  'datadirectory' => '/var/www/html/data',
  'dbtype' => 'pgsql',
  'version' => '17.0.2.1',
  'overwrite.cli.url' => 'http://127.0.0.1:9001',
  'overwritewebroot' => '/',
  //'overwriteprotocol' => 'https',
  //'overwritehost' => 'my.public.domain',

and also edit the docker-compose.yml of nextcloud like suggested above, I can’t even get a connection to the nextcloud server, not even a 502 Bad Gateway result.

Also when starting nginx with the new configuration, I get these two lines in it’s logs:

2019/12/27 12:30:24 [emerg] 1#1: host not found in upstream "nextcloud-server" in /etc/nginx/conf.d/default.conf:55
nginx: [emerg] host not found in upstream "nextcloud-server" in /etc/nginx/conf.d/default.conf:55

So using the internal network of docker does not seem to work at this moment. I don’t know why this happens. I have tried this before to make sure this is not the cause of the problem but it also did not work then.


Thirdly, regarding trusted_proxy:

I had set the following properties in my config.php before:

  'trusted_proxies' => ['127.0.0.1', 'localhost', 'nginx'],

I tried using the singular form trusted_proxy besides that like this:

  'trusted_proxies' => ['127.0.0.1', 'localhost', 'nginx'],
  'trusted_proxy' => ['127.0.0.1', 'localhost', 'nginx'],

Using this configuration, the problem with Bad Gateway 502 still persists. To be clear: Because the internal-docker-network-thing did even work worse at the moment, I reverted those changes back to the 127.0.0.1:9001 set up, to test this change in config.php.


One more thing to mention. When starting the nextcloud-server docker instance, I get this in the logs:

AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.22.0.3. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.22.0.3. Set the 'ServerName' directive globally to suppress this message
[Fri Dec 27 12:39:45.762012 2019] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.38 (Debian) PHP/7.3.13 configured -- resuming normal operations
[Fri Dec 27 12:39:45.762360 2019] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'

Don’t know if this might be helpful. Maybe the first two lines are reson for the problem?


Thank you again for trying to help. :slight_smile:

that’s true. so nextcloud is up&running.

if you define this in your nginx container. you’ll send the request to port 9001 of your nginx container. inside a container 127.0.0.1 is not the host 127.0.0.1.

here again. you force the browser to access port 9001 on the machine where the browser is running and bypass the nginx container. so you won’t be able to access nextcloud from any other machine. and since you don’t expose port 80 on port 9001 anymore you won’t get an answer. if that’s to confusing and my english too bad search youtube for “docker networking 101”. it’s perfect explained there.

to test this you can enter the nginx container via docker exec -it nginx /bin/sh and try to ping or curl the nextcloud-server.

do you define any networks in your docker compose files?

the 502 is not related to trusted_proxies. later when you successfully logged into nextcloud you’ll get a warning in the settings page.

here are my setting for the trusted_xxx variables.
the trusted_domains are the server fqdn and name of the nginx container. i’m using the nginx-fpm nextcloud container. you are using the apache one. in my case nginx is the web-server not the proxy.
the trusted proxies is only the backend (docker internal) ip address of the reverse proxy. (in my case a traefik container. in your case the nginx container.)

no. that’s the “normal output” of the apache nextcloud container. nothing to worry about.

Thank you very much @Reiner_Nippes for your awesome support.

The hint with the docker networking really helped.

I managed to get the setup working by creating a docker network with a certain name (e.g. “proxy-network”)
docker network create -d bridge proxy-network

This allowed me to add the following lines to the docker-compose-files of both nginx and nextcloud:

networks:
  reverse-proxy-network:
    external:
      name: proxy-network

which then allowed me to add the newly defined network reverse-proxy-network to the defined services in those files.

Here is an example for the nextcloud docker-compose.yml file:

services:
  database:
    image: ...
    volumes:
      ...
    container_name: nextcloud-database
    networks:
      - internalnetwork #allows this service to be discovered by nextcloud-service
  nextcloud:
    image: ...
    links:
      ...
    volumes:
      ...
    container_name: nextcloud-server
    networks:
      - reverse-proxy-network # needed to discover nginx reverse proxy
      - internalnetwork # needed to discover database-server

networks:
  internalnetwork: # needed for the services in this docker-compose to communicate
    driver: bridge
  reverse-proxy-network: # needed for communication with reverse proxy
    external:
      name: proxy-network

The docker-compose.yml file for nginx would loke similar:

services:
  nginx:
    image: ...
    volumes:
      ...
    networks:
      - reverse-proxy-network # needed to discover nextcloud-server

networks:
  reverse-proxy-network:
    external:
      name: proxy-network

Using this setup, I can now use this line in the configuration file of nginx:

proxy_pass http://nextcloud-server/;

At this point the name of the nextcloud-container as defined in the docker-compose.yml of nextcloud must be used.

Using this setup removed the 502 Bad Gateway error.


Unfortunately I now have another issue, because my nextcloud instance is set to be available under https://my.public.domain/nextcloud. Using this URL I can reach the server, but loading any resources (javascript and css, …) fails, because they are requested from https://my.public.domain/ (without path). But this is another issue and I will first try to find answers on the web and if I am not successful, I will come back to here.

The Bad Gateway Issue is solved! :slight_smile:

Thank you again for your help @Reiner_Nippes.

maybe the overwrite parameters are helpful.
https://docs.nextcloud.com/server/17/admin_manual/configuration_server/reverse_proxy_configuration.html?highlight=overwrite#overwrite-parameters

Thanks again @Reiner_Nippes, using overwritewebroot fixed the problem. :slight_smile: