AIO instance not reachable - apache container did not (re)start

ℹ️ Support 📦 Appliances (Docker, Snappy, VM, NCP, AIO)
1

Since tonight, I guess after the daily backup, the AIO apache container did not work anymore. Checking its state it tells “created”.

docker inspect nextcloud-aio-apache
[
    {
        "Id": "32317764e2d8dc681c0ec65ee5a00a417f1a82a2e22ef32b137d5e0e50c16fc4",
        "Created": "2023-08-22T02:45:28.072232759Z",
        "Path": "/start.sh",
        "Args": [
            "/usr/bin/supervisord",
            "-c",
            "/supervisord.conf"
        ],
        "State": {
            "Status": "created",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 128,
            "Error": "driver failed programming external connectivity on endpoint nextcloud-aio-apache (5aef47796940b8ef58acd84eb2170a8819cb3936045db996aa63dcdc8823792d):  (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 443 -j DNAT --to-destination 172.19.0.8:443 ! -i br-09118e3649c1: iptables: No chain/target/match by that name.\n (exit status 1))",
            "StartedAt": "0001-01-01T00:00:00Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:a77cf3dec4110a506b2511515cea91b3c87e16c682098e437d2503890be49d1f",
        "ResolvConfPath": "/var/lib/docker/containers/32317764e2d8dc681c0ec65ee5a00a417f1a82a2e22ef32b137d5e0e50c16fc4/resolv.conf",
        "HostnamePath": "",
        "HostsPath": "/var/lib/docker/containers/32317764e2d8dc681c0ec65ee5a00a417f1a82a2e22ef32b137d5e0e50c16fc4/hosts",
        "LogPath": "",
        "Name": "/nextcloud-aio-apache",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "nextcloud_aio_nextcloud:/var/www/html:ro",
                "nextcloud_aio_apache:/mnt/data:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {
                    "max-file": "5",
                    "max-size": "10m"
                }
            },
            "NetworkMode": "nextcloud-aio",
            "PortBindings": {
                "443/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "443"
                    }
                ],
                "443/udp": [
                    {
                        "HostIp": "",
                        "HostPort": "443"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "unless-stopped",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                0,
                0
            ],
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "host",
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": true,
            "SecurityOpt": null,
            "Tmpfs": {
                "/home/www-data": "",
                "/tmp": "",
                "/usr/local/apache2/logs": "",
                "/var/log/supervisord": "",
                "/var/run/supervisord": ""
            },
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/c02286e8327e6d94d18320c49f70d021dd00bddbda574729f2ec4d9df59a50a8-init/diff:/var/lib/docker/overlay2/1fa2acb45de1364fb852f255ee302ae678fea778a8dde6fe80c2e1090bc68d61/diff:/var/lib/docker/overlay2/cceab8191bfa400ea0fd4ec65a82cea2851d65ed6c11f2a1eb800f74b0ae7f2f/diff:/var/lib/docker/overlay2/b7e29d9f12e5336d483837a4ba157a8d9b9f985acb95ea6a3903c2da0276f869/diff:/var/lib/docker/overlay2/94a7600386c7b9e88c9c1355806d8c310cae3e2595a3fda3851dbd127bb01c03/diff:/var/lib/docker/overlay2/26b52bdeff8d783d04e0e6c8c14bef89984646bd7c7457db64a7870ec4aab95f/diff:/var/lib/docker/overlay2/c43d739c0fc10b891b56404049c77c18f66430fdaf972b88c2d19694f0751df5/diff:/var/lib/docker/overlay2/452a7a33031fc225b104f6a0360fc4e2ee2995ab348069a2af093e291f76a8e9/diff:/var/lib/docker/overlay2/c4c23983689d359b949bd92ec96e38bde11eb4cc8dc9a0574eb911ca8e8bd21e/diff:/var/lib/docker/overlay2/b24cd21df5940bd32420bdbb6afc85a7aed1b9e6883a8186e426a5c0623c2a4a/diff:/var/lib/docker/overlay2/996d817ff601979ff0a8c02a0fae724830e51a8623225814b795eb79924e2f74/diff:/var/lib/docker/overlay2/c8435876e7dbf4644513705d933d1b5c0a722709e903c74e141d3edcf723faff/diff:/var/lib/docker/overlay2/701370f641636873fd59f7fcbaedff6d055107decf2461a0f69837c05e9ac533/diff:/var/lib/docker/overlay2/76b1b46a34807eb176e0a892e1b2f250390faa91ffd20a5c09a60233bbd3dc41/diff",
                "MergedDir": "/var/lib/docker/overlay2/c02286e8327e6d94d18320c49f70d021dd00bddbda574729f2ec4d9df59a50a8/merged",
                "UpperDir": "/var/lib/docker/overlay2/c02286e8327e6d94d18320c49f70d021dd00bddbda574729f2ec4d9df59a50a8/diff",
                "WorkDir": "/var/lib/docker/overlay2/c02286e8327e6d94d18320c49f70d021dd00bddbda574729f2ec4d9df59a50a8/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "volume",
                "Name": "nextcloud_aio_nextcloud",
                "Source": "/var/lib/docker/volumes/nextcloud_aio_nextcloud/_data",
                "Destination": "/var/www/html",
                "Driver": "local",
                "Mode": "ro",
                "RW": false,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "nextcloud_aio_apache",
                "Source": "/var/lib/docker/volumes/nextcloud_aio_apache/_data",
                "Destination": "/mnt/data",
                "Driver": "local",
                "Mode": "rw",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "32317764e2d8",
            "Domainname": "",
            "User": "www-data",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "443/tcp": {},
                "443/udp": {},
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "NC_DOMAIN=yxc.selfhost.eu",
                "NEXTCLOUD_HOST=nextcloud-aio-nextcloud",
                "COLLABORA_HOST=nextcloud-aio-collabora",
                "TALK_HOST=nextcloud-aio-talk",
                "APACHE_PORT=443",
                "ONLYOFFICE_HOST=nextcloud-aio-onlyoffice",
                "TZ=Etc/UTC",
                "APACHE_MAX_SIZE=10737418240",
                "APACHE_MAX_TIME=3600",
                "NOTIFY_PUSH_HOST=nextcloud-aio-notify-push",
                "PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "HTTPD_PREFIX=/usr/local/apache2",
                "HTTPD_VERSION=2.4.57",
                "HTTPD_SHA256=dbccb84aee95e095edfbb81e5eb926ccd24e6ada55dcd83caecb262e5cf94d2a",
                "HTTPD_PATCHES=rewrite-windows-testchar-h.patch 1d5620574fa03b483262dc5b9a66a6906553389952ab5d3070a02f887cc20193"
            ],
            "Cmd": [
                "/usr/bin/supervisord",
                "-c",
                "/supervisord.conf"
            ],
            "Healthcheck": {
                "Test": [
                    "CMD-SHELL",
                    "/healthcheck.sh"
                ]
            },
            "Image": "nextcloud/aio-apache:latest",
            "Volumes": {
                "/mnt/data": {}
            },
            "WorkingDir": "/usr/local/apache2",
            "Entrypoint": [
                "/start.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "com.centurylinklabs.watchtower.enable": "false"
            },
            "StopSignal": "SIGWINCH"
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "a153e4c6a2ce612421400490e8a4103232ea9ce484292d55dc386bace71a6409",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/a153e4c6a2ce",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "nextcloud-aio": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "32317764e2d8"
                    ],
                    "NetworkID": "09118e3649c160f7c59fd501b2d65c4d3390b8c0f79bc1c3d58cd3930062113f",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]
3

Hi, I’ve found this: port - Running docker container : iptables: No chain/target/match by that name - Stack Overflow

4

You are quick! :slight_smile:

ss -tulpn
Netid           State            Recv-Q           Send-Q                                       Local Address:Port                     Peer Address:Port          Process
udp             UNCONN           0                0                                             0.0.0.0%eth0:68                            0.0.0.0:*              users:(("wickedd-dhcp4",pid=1146,fd=8))
udp             UNCONN           0                0                                                127.0.0.1:323                           0.0.0.0:*              users:(("chronyd",pid=1913,fd=5))
udp             UNCONN           0                0                                                    [::1]:323                              [::]:*              users:(("chronyd",pid=1913,fd=6))
udp             UNCONN           0                0                          [fe80::208:9bff:fefa:dc4e]%eth0:546                              [::]:*              users:(("wickedd-dhcp6",pid=1147,fd=8))
tcp             LISTEN           0                100                                              127.0.0.1:25                            0.0.0.0:*              users:(("master",pid=2050,fd=13))
tcp             LISTEN           0                4096                                               0.0.0.0:8443                          0.0.0.0:*              users:(("docker-proxy",pid=9391,fd=4))
tcp             LISTEN           0                4096                                               0.0.0.0:8000                          0.0.0.0:*              users:(("docker-proxy",pid=7536,fd=4))
tcp             LISTEN           0                4096                                               0.0.0.0:9443                          0.0.0.0:*              users:(("docker-proxy",pid=7347,fd=4))
tcp             LISTEN           0                4096                                               0.0.0.0:80                            0.0.0.0:*              users:(("docker-proxy",pid=9660,fd=4))
tcp             LISTEN           0                4096                                               0.0.0.0:8080                          0.0.0.0:*              users:(("docker-proxy",pid=9555,fd=4))
tcp             LISTEN           0                128                                                0.0.0.0:22                            0.0.0.0:*              users:(("sshd",pid=1911,fd=3))
tcp             LISTEN           0                100                                                  [::1]:25                               [::]:*              users:(("master",pid=2050,fd=14))
tcp             LISTEN           0                4096                                                  [::]:8443                             [::]:*              users:(("docker-proxy",pid=9414,fd=4))
tcp             LISTEN           0                4096                                                  [::]:8000                             [::]:*              users:(("docker-proxy",pid=7548,fd=4))
tcp             LISTEN           0                4096                                                  [::]:9443                             [::]:*              users:(("docker-proxy",pid=7451,fd=4))
tcp             LISTEN           0                4096                                                  [::]:80                               [::]:*              users:(("docker-proxy",pid=9675,fd=4))
tcp             LISTEN           0                4096                                                  [::]:8080                             [::]:*              users:(("docker-proxy",pid=9574,fd=4))
tcp             LISTEN           0                128                                                   [::]:22                               [::]:*              users:(("sshd",pid=1911,fd=4))

I did already reboot the server but the result is the same. But I will check the link for hints that restarting/rebooting does not help. The firewall is disabled anyway.

5

You mean iptables is not running? That could explain the problem…

6

firewalld was disabled. But I tried with it enabled and it does not work either.

iptables -t filter -F

iptables -t filter -X

systemctl restart docker

Did not help either. I will search for more info regarding the messages.

systemctl status firewalld.service
â—Ź firewalld.service - firewalld - dynamic firewall daemon
     Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: disabled)
     Active: active (running) since Tue 2023-08-22 17:47:27 CEST; 7min ago
       Docs: man:firewalld(1)
   Main PID: 26975 (firewalld)
      Tasks: 2 (limit: 4915)
     CGroup: /system.slice/firewalld.service
             └─ 26975 /usr/bin/python3 /usr/sbin/firewalld --nofork --nopid

Aug 22 17:52:37 wolke firewalld[26975]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER' failed: iptables: No chain/target/match by that name.
Aug 22 17:52:37 wolke firewalld[26975]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION-STAGE-1' failed: iptables: No chain/target/match by that name.
Aug 22 17:52:37 wolke firewalld[26975]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION-STAGE-1' failed: iptables: No chain/target/match by that name.
Aug 22 17:52:37 wolke firewalld[26975]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION-STAGE-2' failed: iptables: No chain/target/match by that name.
Aug 22 17:52:37 wolke firewalld[26975]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION-STAGE-2' failed: iptables: No chain/target/match by that name.
Aug 22 17:52:37 wolke firewalld[26975]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
Aug 22 17:52:37 wolke firewalld[26975]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
Aug 22 17:52:37 wolke firewalld[26975]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i br-09118e3649c1 -o br-09118e3649c1 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that>
Aug 22 17:52:38 wolke firewalld[26975]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Aug 22 17:52:39 wolke firewalld[26975]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
7

Yeah, looks like something is broken on your insall with iptables and/or firewalld…

8

I found the cause but not the reason - yet. I added an openhab docker image yesterday. Nextcloud worked fine after that. Yet both containers want to use port 8443 and 8080. Thus I let openhab use 8081 and 8444.

After I removed that container. The apache container could start.

1 Like
9

I changed the openhab ports and at least a complete docker restart does not lead to the same problem again.

docker run   --name openhab   --net=host   -v /etc/localtime:/etc/localtime:ro   -v /opt/openhab/addons:/openhab/addons   -v /opt/openhab/conf:/openhab/conf   -v /opt/openhab/userdata:/openhab/userdata   -e "CRYPTO_POLICY=unlimited"   -e "EXTRA_JAVA_OPTS=-Duser.timezone=Europe/Berlin"   -e OPENHAB_HTTP_PORT=8082 -e OPENHAB_HTTPS_PORT=8445 -e USER_ID=1001 -e GROUP_ID=9001 openhab/openhab:3.4.5
1 Like