Nextcloud container has no connection to internet (probably dns problem)

ℹ️ Support
,
1

Hello, folks. Latest stable nextcloud apache in docker
installation. Right after installation I found that the
countainer has no connection to the outside world. I can curl
localhost, db, but not any external service. Can’t apt update
as well:

Err:1 http://deb.debian.org/debian bullseye InRelease
  Temporary failure resolving 'deb.debian.org'
Err:2 http://deb.debian.org/debian-security bullseye-security InRelease
  Temporary failure resolving 'deb.debian.org'
Err:3 http://deb.debian.org/debian bullseye-updates InRelease
  Temporary failure resolving 'deb.debian.org'
Reading package lists... Done    
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
W: Failed to fetch http://deb.debian.org/debian/dists/bullseye/InRelease  Temporary failure resolving 'deb.debian.org'
W: Failed to fetch http://deb.debian.org/debian-security/dists/bullseye-security/InRelease  Temporary failure resolving 'deb.debian.org'
W: Failed to fetch http://deb.debian.org/debian/dists/bullseye-updates/InRelease  Temporary failure resolving 'deb.debian.org'
W: Some index files failed to download. They have been ignored, or old ones used instead.

/etc/resolv.conf of the container:

nameserver 127.0.0.11
options edns0 trust-ad ndots:0

curl -I 192.168.77.30
HTTP/1.1 301 Moved Permanently
Server: openresty
Date: Mon, 10 Oct 2022 17:12:36 GMT
Content-Type: text/html
Content-Length: 166
Connection: keep-alive
Location: http://192.168.77.30:81

Here I curled host machine lan ip (nginx-proxy-manager runs on this port).

Host machine is running debian 11. docker-compose file:

services:
  db:
    image: mariadb:10.5
    container_name: nextcloud-db
    hostname: nas
    restart: always
    command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW 
    volumes:
      - /mnt/media/docker/containers/nextcloud-db:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=
      - MYSQL_PASSWORD=
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
  app:
    image: nextcloud:stable
    container_name: nextcloud
    hostname: nas
    restart: always
    ports:
      - "51005:80"
    links:
      - db
    depends_on:
      - db
    volumes:
      - /mnt/media/docker/containers/nextcloud:/var/www/html
    environment:
      - MYSQL_PASSWORD=
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
      - MYSQL_HOST=db
      - NEXTCLOUD_ADMIN_USER=admin
      - NEXTCLOUD_ADMIN_PASSWORD=
    dns:
      - 1.1.1.1
      - 1.0.0.1
      - 9.9.9.9

networks:
  default:
    name: dockerpub0

Tryed to manualy change nameservers in /etc/resolv.conf to

nameserver 1.1.1.1
nameserver 8.8.8.8

Got no effect. In admin pannel on Logging page I see a lot of:

Error: dns_get_record(): A temporary server error occurred. at /var/www/html/lib/private/Http/Client/DnsPinMiddleware.php#83

I have a dozen containers with the same setup and they all work
fine, all have connection and resolves any address without
problems. Any ideas what could go wrong? As I said it’s a fresh
new container without any legacy data.

2

Network on the host is presumably working fine? Does it have a firewall?

4

Network on the host is fine. I use nftables:

table inet filter {
        chain input {
                type filter hook input priority filter; policy drop;
                iif "lo" accept
                ct state established,related accept
                icmp type { echo-reply, echo-request } accept
                tcp dport 22122 accept
                tcp dport 80 accept
                tcp dport 443 accept
                tcp dport 81 accept
                tcp dport 9091 accept
                tcp dport 8200 accept
                udp dport 1900 accept
                tcp dport 445 accept
                tcp dport { 51001, 51002 } accept
                tcp dport { 51003, 51004 } accept
                tcp dport 51005 accept
                tcp dport 51006 accept
                iifname "dockerpub0" accept
                oifname "dockerpub0" accept
        }

        chain forward {
                type filter hook forward priority filter; policy accept;
                ct state established,related accept
                iifname "dockerpub0" accept
                oifname "dockerpub0" accept
        }

        chain output {
                type filter hook output priority filter; policy accept;
        }
}
table ip nat {
        chain prerouting {
                type nat hook prerouting priority dstnat; policy accept;
                ct state invalid drop
        }

        chain postrouting {
                type nat hook postrouting priority srcnat; policy accept;
                ct state invalid drop
                iifname "dockerpub0" oifname { "wg0", "eno1" } snat to 192.168.77.30
        }
}

sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1

I’ve turned off iptables for docker in systemd service.

5

Only thing that comes to mind is that maybe the firewall won’t let it through. Is it the case that the container can connect to the host but not anything outside it?

Is dockerpub0 the interface the container is connected to?

6

That seems reasonable from one hand, but all my other containers with the same network setup work fine. Also I tried to flush nft ruleset and it didn’t help. So I suppose firewall is not a problem or not an obvious one (at least for me).

Yes, dockerpub0 is a bridge interface for all docker containers with 172.32.0.0/24 network. 192.168.77.30 is LAN server ip.

Is it the case that the container can connect to the host but not anything outside it?

That’s the case for nextcloud container only. As I was saying any other container successfully connect with other containers/lan clients and internet resources.

7

Meaning that you have the same dockerpub0 network defined in other compose files on the same host? Or other docker hosts on the same network?

Docker compose renames some things after the compose file name. In that case it may be that the actual interface name is different. You might run ip link show to verify the actual interface name is correct.

I think it’s safe to say the problem isn’t the container since it’s fresh, so the problem must be something with the compose setup or the firewall, or something else on the host.

8

Exactly. All my containers except nginx that is in host network mode run in dockerpub0 network. Every compose file on the same server have this part:

networks:
  default:
    name: dockerpub0

ip link show shows a lot of br-* and veth* interfaces.

5: br-5a79f95f074c: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default 
    link/ether 02:42:5f:29:62:e0 brd ff:ff:ff:ff:ff:ff
6: dockerpub0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default 
    link/ether 02:42:10:3e:f0:a2 brd ff:ff:ff:ff:ff:ff
7: br-ca84b244450d: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default 
    link/ether 02:42:76:19:80:d4 brd ff:ff:ff:ff:ff:ff
8: br-c398e092c476: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default 
    link/ether 02:42:31:5c:f3:15 brd ff:ff:ff:ff:ff:ff
9: br-fe7cebb0e215: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default 
    link/ether 02:42:c2:06:be:de brd ff:ff:ff:ff:ff:ff
10: br-57b1cbafb9a1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:18:6a:ab:48 brd ff:ff:ff:ff:ff:ff
15: br-506d01bb09b9: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:e2:43:69:73 brd ff:ff:ff:ff:ff:ff
18: br-2b88b0c6c2f2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:b9:d6:7b:33 brd ff:ff:ff:ff:ff:ff
38: vethe593439@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-57b1cbafb9a1 state UP mode DEFAULT group default 
    link/ether 6e:96:6e:81:7f:47 brd ff:ff:ff:ff:ff:ff link-netnsid 0
40: veth862a83f@if39: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-57b1cbafb9a1 state UP mode DEFAULT group default 
    link/ether be:47:d2:c1:63:90 brd ff:ff:ff:ff:ff:ff link-netnsid 1
46: veth38c6670@if45: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-57b1cbafb9a1 state UP mode DEFAULT group default 
    link/ether c6:7b:94:02:8c:72 brd ff:ff:ff:ff:ff:ff link-netnsid 2
59: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/none 
115: veth433b7c1@if114: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-57b1cbafb9a1 state UP mode DEFAULT group default 
    link/ether d6:3b:f3:4d:83:dc brd ff:ff:ff:ff:ff:ff link-netnsid 3
117: vethcf0fbdd@if116: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-57b1cbafb9a1 state UP mode DEFAULT group default 
    link/ether 36:25:54:34:0b:20 brd ff:ff:ff:ff:ff:ff link-netnsid 4

AFAIU they are dynamic and doesn’t affect firewall rules when it comes to network interaction. If I turn on iptables and leave docker to manage rules I don’t see br-* and veth* interfaces in iptables rules. And again even if docker somehow user this br-* and veth* interfaces for real network interaction, it means that other containers also somehow related to these interfaces. Other containers work with the very same network setup including firewall rules. Therefore I see two possible ways: 1) magically all other containers made a hole in firewall to work as expected, 2) nextcloud container has some feature that makes impossible to work in my setup. What feature - that’s the questions. If I can fix it from firewall side or docker side or any other side I’m gladly hear any suggestions.

9

Well, I made a little experiment. I removed all custom settings. Everything by default setup. Iptables, default docker0 interface, no dns in compose file. No success. Nextcloud container just doesn’t see anything outside my lan. Every other container with this setup work as expected. Seems like I was right about it’s not a firewall. Every iptables rule and route was created by docker itself. Also I tried to use resolve service instead of systemd-resolved and manually set dns servers in /etc/resolv.conf of container. Still no connection. BTW it’s weird that for such a complex software as nextcloud that is all about network interaction there is no ping utility preinstalled.

10

I wouldn’t make that assumption because I have previously helped people who turned on UFW (iptables) and found out that it needed rules added for the docker interface.

It would rarely be needed or used.

All I can tell you is the problem isn’t some design flaw in Nextcloud’s container image because it’s working fine for everyone else. It must have something to do with your system.

Maybe you can try your compose file on another system temporarily just to see if it works?

Also, I think the compose file reference says to use external if the compose file is connecting to a network that already exists.

11

Yeah, you’re right, thanks.

Just tried the same compose file on VPS (ubuntu server 22.04) and on my server mirror (slackware 15) - same results.

I’m not against this version, just want already to find the problem and fix it…

AFAIS you’re experienced in networking, docker and nextcloud. Could you please tell me are there any problems in my nftables setup and compose configuration? Anything that could lead to my problem?

12

This thread reminds me of GitHub - nextcloud/all-in-one: Nextcloud AIO stands for Nextcloud All In One and provides easy deployment and maintenance with most features included in this one Nextcloud instance.

13

Any ideas?

14

This is likely the best workound for you:

15

That doesn’t work. Any workarounds with specifying dns like in compose file (see my first post here), manually changing dns inside the docker container, changing dns in daemon.json file just don’t work. Nextcloud-aio doesn’t work too with exactly the same problem.

It could be not just dns issue. I tried to curl a couple of internet resources by ip including my vps with apache running and I didn’t get a response. More looks like nextcloud container unable to connect to internet resources. Curious.

16

maybe back to start. two of my NC24 images have this:

cat /etc/resolv.conf

nameserver 127.0.0.11
options ndots:0

which is different from your file:

according to resolv.conf(5) - Linux manual page this doesn’t sound like a problem… but who knows?

17

Ok. I’ve managed to solve^W avoid this by installing linuxserver/nextcloud container. Works like a charm out of the box. If someone wants to investigate this non-typical issue I’ll save the container.