Docker installation with macvlan and cloudflare = Bad gateway 502

aaaha · September 18, 2023, 7:06am

Hi there,

I am completely new to Nextcloud an Cloudflare but managed to install Nextcloud within docker, using the Nextcloud installation instructions for a reverse proxy setup., meticulously. Only difference is: I am using the macvlan network architecture in docker. So every container receives (should receive) its own dedicated IP.

Now I am running in to an 502 bad gateway error. These are my logs

nextcloud-aio-nextcloud Container:

Activating Collabora config…
An unhandled exception has been thrown:
Error: Call to a member function xpath() on bool in /var/www/html/custom_apps/richdocuments/lib/WOPI/Parser.php:51
Stack trace:
#0 /var/www/html/custom_apps …

cloudflare tunnel container:
2023-09-18T06:31:54Z ERR Request failed error=„Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: dial tcp <IP-of-nextcloud-master-container>:11000: connect: connection refused“ connIndex=1 dest=https://<my-sub-and-tld>/hosting/capabilities event=0 ip=<probably-my-IP> type=http

All the other containers seem to be fine (master, collabora, database, redis, imaginary, notify-push, apache).

My docker-compose looks like this:

version: ‚3‘
services:
  nextcloud-aio-mastercontainer:
    image: nextcloud/all-in-one:latest
    init: true
    restart: unless-stopped
    container_name: nextcloud-aio-mastercontainer 
    volumes:
      - nextcloud_aio_mastercontainer:/mnt/docker-aio-config
      - /var/run/docker.sock:/var/run/docker.sock:ro
    ports:
      #- 80:80 
      - 8080:8080
      #- 8443:8443 
    environment: 
      - APACHE_PORT=11000
      #- APACHE_IP_BINDING=  // I tried activating this, tried with localhost, 127.0.0.1, 0.0.0.0, IP of cloudflare container, IP of nextcloud master container. All with no luck.
      - SKIP_DOMAIN_VALIDATION=true
      - NEXTCLOUD_DATADIR=/volume1/docker/nextcloud/data
    networks:
      default:
        ipv4_address: <ip-of-nextcloud>

volumes:
  nextcloud_aio_mastercontainer:
    name: nextcloud_aio_mastercontainer

networks:
  default:
    name: eth_macvlan
    external: true

The hostname settings within cloudflare for my sub- and tld are http.
I tried https with enabling no TLS verification but withouth luck.

My concern is that I am using macvlan structure and all those created containers cant talk to each other since they are not running on localhost.

What else can I check or try out?

szaimen · September 18, 2023, 7:07am

Hi, can you follow https://github.com/nextcloud/all-in-one/blob/main/reverse-proxy.md#6-how-to-debug-things?

aaaha · September 18, 2023, 7:48am

Make sure to exactly follow the whole reverse proxy documentation step-for-step from top to bottom!

Done.

Make sure that you used the docker run command that is described in this reverse proxy documentation. Hint: make sure that you have set the APACHE_PORT via e.g. --env APACHE_PORT=11000 during the docker run command!

Apache Port is 11000, done.

Make sure to set the APACHE_IP_BINDING variable correctly. If in doubt, set it to --env APACHE_IP_BINDING=0.0.0.0

I tried activating this, tried with localhost, 127.0.0.1, 0.0.0.0, IP of cloudflare container, IP of nextcloud master container. All with no luck.

Make sure that all ports to which your reverse proxy is pointing match the chosen APACHE_PORT.

Set within cloudflares UI, is 11000.

Make sure that the reverse proxy is running on the host OS or if running in a container, connected to the host network. If that is not possible (e.g. on Windows or if the reverse proxy is running on a different host), substitute localhost or 127.0.0.1 in the default configurations by the private ip-address of the host that is running the docker daemon. If you are not sure how to retrieve that, you can run: ip a | grep "scope global" | head -1 | awk '{print $2}' | sed 's|/.*||'. If the command returns a public ip-address, use ip a | grep "scope global" | grep docker0 | awk '{print $2}' | sed 's|/.*||' instead (the commands only work on Linux)

Set to the IP of nextclouds mastercontainer, which is the IP I set in my docker-compose. This works for an other service I host and access via cloudflare also. Done.

Make sure that the mastercontainer is able to spawn other containers. You can do so by checking that the mastercontainer indeed has access to the Docker socket which might not be positioned in one of the suggested directories like /var/run/docker.sock but in a different directory, based on your OS and the way how you installed Docker. The mastercontainer logs should help figuring this out. You can have a look at them by running sudo docker logs nextcloud-aio-mastercontainer after the container is started the first time.

It can. I can see them all in portainer, also the installation page from nextcloud showed them all “green”. Done.

Check if after the mastercontainer was started, the reverse proxy if running inside a container, can reach the provided apache port. You can test this by running nc -z localhost 11000; echo $? from inside the reverse proxy container. If the output is 0, everything works. Alternatively you can of course use instead of localhost the ip-address of the host here for the test.

I cant access the cloudflare container since I dont know where the execute command for bash is located.
sudo docker exec -it cloudflare bash
sudo docker exec -it cloudflare /bin/bash
sudo docker exec -it cloudflare /bin/sh
sudo docker exec -it cloudflare sh
sudo docker exec -it cloudflare bin
If I obtain access I will check using the mastercontainers IP. This is my last straw.

Make sure that you are not behind CGNAT. If that is the case, you will not be able to open ports properly. In that case you might use a Cloudflare Tunnel.

I am using cloudflare, done.

If you use Cloudflare, you might need to skip the domain validation anyways since it is known that Cloudflare might block the validation attempts. In that case, see the last option below.

Skipping, done.

If your reverse proxy is configured to use the host network (as recommended in the above docs) or running on the host, make sure that you’ve configured your firewall to open port 443 and 80.

Do I need to open those two ports on the nextcloud mastercontainer?

Check if you have a public IPv4- and public IPv6-address. If you only have a public IPv6-address (e.g. due to DS-Lite), make sure to enable IPv6 in Docker and your whole networking infrastructure (e.g. also by adding an AAAA DNS-entry to your domain).

only IPv4, check.

Try to configure everything from scratch if it still does not work by following GitHub - nextcloud/all-in-one: The official Nextcloud installation method. Provides easy deployment and maintenance with most features included in this one Nextcloud instance..

Done twice.

As last resort, you may disable the domain validation by adding --env SKIP_DOMAIN_VALIDATION=true to the docker run command. But only use this if you are completely sure that you’ve correctly configured everything!

Done.

szaimen · September 18, 2023, 8:04am

This cannot work.

Is the cloudflare tunnel on the host? Then you need to point it at http://localhost:11000

aaaha · September 18, 2023, 12:08pm

Thanks for your help!

Well it is and it isn’t.
The cloudflare tunnel has its own container with its own IP. So has next cloud. Both are running on the same host though.
Also I read in 3. of the documentation that

This is only recommended to be set if you use localhost in your reverse proxy config to connect to your AIO instance.

which I’m not. I use the nextcloud IP to address the nextcloud instance within cloudflare.

szaimen · September 18, 2023, 12:55pm

All right. Then you could try to point it at http://nextcloud-aio-apache:11000

aaaha · September 18, 2023, 2:09pm

Ah ok, so should cloudflare contact the apache instance and not the nextcloud-aio-mastercontainer? I just tried the following clients within cloudflare:

http://nextcloud-aio-apache:11000

2023-09-18T13:32:24Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: dial tcp: lookup nextcloud-aio-apache on 127.0.0.11:53: no such host" connIndex=1 dest=https://my.domain.tld/favicon.ico event=0 ip=123.123.123.123 type=http

http://172.19.0.10:11000 #which is the IP docker assigned to apache automatically

2023-09-18T13:37:04Z ERR error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: dial tcp 172.19.0.10:11000: connect: network is unreachable" cfRay=8089fb439d177a3c-DUS event=1 ingressRule=1 originService=https://172.19.0.10:11000

Both http and https did not work.

szaimen · September 18, 2023, 2:18pm

Is the cloudflare container in the nextcloud-aio network?

aaaha · September 18, 2023, 2:25pm

Also yes and no:
Cloudflare: 192.168.x.x
Nextcloud Master: 192.168.x.x
All other nextcloud containers spawned by the master: 172.18.x.x

szaimen · September 18, 2023, 2:57pm

Adding cloudflare to the nextcloud-aio network should make the above work.

Otherwise you would need to point cloudflare at http://private.ip.of.the.server:11000

aaaha · September 18, 2023, 3:29pm

Is it possible to give the apache server its own dedicated IP when or after starting/spawning?

szaimen · September 18, 2023, 3:31pm

No it isnt

aaaha · September 26, 2023, 1:15pm

It is actually:

remove the existing network connection (one can do so nicely in portainer, at the very end of the container page)
stop the container
sudo docker network connect name_of_the_macvlan_network_adapter --ip=desired_IP_address name-of-container
start the container

Downside: I had to do so for every of those 8 containers manually, but now I am finally presented with the login screen (not beeing able to log in, but thats an other issue).

szaimen · September 26, 2023, 2:11pm

It might but not permanently

system · September 23, 2024, 1:41pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.