AIO: waiting for Database container to start - but it looks like it started, no errors

šŸš§ Installation
, ,
28

unfortunately, itā€™s the same outcome.

everything is ā€œStartingā€. Database is up, but Nextcloud is waiting for it.

the bridge network does not say internal on it.
the nextcloud-aio network does say internal.

same again here:

sudo docker exec -it nextcloud-aio-nextcloud bash
bash-5.1$ nc -z nextcloud-aio-database 5432; echo $?
1
29

Hmā€¦ What is the output of sudo network inspect nextcloud-aio?

30 
sudo docker network inspect nextcloud-aio
[
    {
        "Name": "nextcloud-aio",
        "Id": "0c88d69e1c2424284c9525b5790b234b497a049ce1bb1405e1810ba99ff9de6e",
        "Created": "2022-05-20T15:48:19.684403856+01:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.19.0.0/16",
                    "Gateway": "172.19.0.1"
                }
            ]
        },
        "Internal": true,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "04be0dee1f444ada7e38b5f293ee0faa87ae94aeeca4681696fdbad16f1a6bb1": {
                "Name": "nextcloud-aio-collabora",
                "EndpointID": "dd22c7863ed9682b0743450d557b1c4c4908d12e39171eb4d262741926ffa45d",
                "MacAddress": "02:42:ac:13:00:03",
                "IPv4Address": "172.19.0.3/16",
                "IPv6Address": ""
            },
            "22518b94cc95163d9a548c5fd530c8d6803107e12307715c1042abc68aef58a2": {
                "Name": "nextcloud-aio-talk",
                "EndpointID": "3e4683b1a90e11a0826f040718423b8c965a44ec0128494c37d8311789ba1541",
                "MacAddress": "02:42:ac:13:00:05",
                "IPv4Address": "172.19.0.5/16",
                "IPv6Address": ""
            },
            "298f11b2ddfce8dc082eca350030ac669b335f79211fffa5e90e1cd3c3d237b9": {
                "Name": "nextcloud-aio-apache",
                "EndpointID": "ce022e8741539b44e173227246ef0a6fc7321a3f0507178efa92311a77243b2f",
                "MacAddress": "02:42:ac:13:00:09",
                "IPv4Address": "172.19.0.9/16",
                "IPv6Address": ""
            },
            "a468d02164dd1b2a4c8945189d124f668e49f0907a18325982542f2e3f04ab92": {
                "Name": "nextcloud-aio-mastercontainer",
                "EndpointID": "1010344cb8bf4893ea752aa1a6e66f2e8b6a51d90cb6b2fa9bcc5bcbbcec5e83",
                "MacAddress": "02:42:ac:13:00:02",
                "IPv4Address": "172.19.0.2/16",
                "IPv6Address": ""
            },
            "c2665a9220c93d23973598ed3de85a251092c19f7296f4b0e3e6b6b7311b9396": {
                "Name": "nextcloud-aio-clamav",
                "EndpointID": "7b8c00c7a42d9eec1af2a356982932b4eaea2ec509b0af2aa4b3098a48663096",
                "MacAddress": "02:42:ac:13:00:04",
                "IPv4Address": "172.19.0.4/16",
                "IPv6Address": ""
            },
            "d0efe4fb7613e8ed7c9432f786e1405b4d0adfb0359bf9e9d97465a40ee3f7c2": {
                "Name": "nextcloud-aio-redis",
                "EndpointID": "621fe1eb4fe78c28940ee372f950d10ba3bb02ca9f0b90bf9017eed5c55951f0",
                "MacAddress": "02:42:ac:13:00:07",
                "IPv4Address": "172.19.0.7/16",
                "IPv6Address": ""
            },
            "d548bc6a83bc846c99d31cd2fa1923235a7e61467068280835b6ebff8197bc11": {
                "Name": "nextcloud-aio-nextcloud",
                "EndpointID": "0d90cb069463e856d36fde66ea1ae2c77c4d67d228d2a379e9761b8a0347453d",
                "MacAddress": "02:42:ac:13:00:08",
                "IPv4Address": "172.19.0.8/16",
                "IPv6Address": ""
            },
            "ebb7764b3694b105293ca4e7bf8ada1ed3d764d0226a7c0e7bba935d3621c913": {
                "Name": "nextcloud-aio-database",
                "EndpointID": "a1d843824cd7f77721a25cb1d2e37499218b8c9480087dfc4ac8b062ca4865f7",
                "MacAddress": "02:42:ac:13:00:06",
                "IPv4Address": "172.19.0.6/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.enable_icc": "true"
        },
        "Labels": {}
    }
]
31

Hmā€¦ this looks goodā€¦

Too bad! I thought we would have found the issue and a fix for itā€¦

I still think that it must be something with the internal networking but for me it works correctlyā€¦

32

looking back through our conversation, the nextcloud-aio was originally Internal: true
so if the bridge does not need to be internal, then I guess we were looking at the wrong bit.

33

do you have ufw enabled on your server?

34

no, I donā€™t. is that a ubuntu thing?
iā€™m using Fedora, it has firewalld. Which is enabled, and I believe docker is configuring the ports with it directly.

if i disable firewalld, the docker container wonā€™t start.

35

All right, then I donā€™t have more ideas what we could try unfortunately :confused:

As I said, it looks like a networking/firewall issue.

My last idea would be trying to run it manually: all-in-one/latest.yml at main Ā· nextcloud/all-in-one Ā· GitHub And then see which network configs are differently (assuming running it manually works on your serverā€¦)

36

Sorry, I sent the wrong link. The link to the documentation on running it manually is this one: all-in-one/manual-install at main Ā· nextcloud/all-in-one Ā· GitHub

1 Like
37

Commenting because this is the issue Iā€™m having on a similar OS (Fedora 35, though not in a VM).

I followed through the same steps here and still have the same issue with the database log saying it has started but the nextcloud log repeating that itā€™s waiting for the DB.

Can I offer any files or outputs?

38

Hello, I fear it simply does not work on your server currently. Thus your only option seems to be AIO: waiting for Database container to start - but it looks like it started, no errors - #35 by szaimen currentlyā€¦

39

Worth a shot, youā€™ve answered my questions twice, once in the wrong thread. Thank you for all the assistance and patience!

I think I might try a VM of something other than Fedora to get around my OS.

40

Docker currently does not cooperate with the nftables backend. It currently side-steps firewalld by injecting its own rules in iptables ahead of firewalldā€™s rules. However, with the nftables backend firewalldā€™s rule will still be evaluated. Netfilter in the kernel will call iptables, then nftables for the same packet. This means firewalld/nftables is likely to drop the packet even if docker has iptables rules to ACCEPT.

https://fedoraproject.org/wiki/Changes/firewalld_default_to_nftables

Maybe this is worth a try?

41

Yes, looks promising!

@Rilr Feel free to try and report back if it works! We can then add it to the documentation :slight_smile:

1 Like
42

Hereā€™s the firewalld log from yesterday:

May 20 13:33:28 cirrus systemd[1]: firewalld.service: Consumed 1.426s CPU time.
May 20 13:33:28 cirrus systemd[1]: Stopped firewalld.service - firewalld - dynamic firewall daemon.
May 20 13:33:28 cirrus systemd[1]: firewalld.service: Deactivated successfully.
May 20 13:33:28 cirrus systemd[1]: Stopping firewalld.service - firewalld - dynamic firewall daemon...
May 20 12:34:15 cirrus firewalld[13472]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i br-55959a46b>
May 20 12:13:17 cirrus firewalld[13472]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i br-a20b3536d>
May 20 12:13:17 cirrus firewalld[13472]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o d>
May 20 12:13:16 cirrus systemd[1]: Started firewalld.service - firewalld - dynamic firewall daemon.
May 20 12:13:16 cirrus systemd[1]: Starting firewalld.service - firewalld - dynamic firewall daemon...
May 20 12:09:44 cirrus systemd[1]: firewalld.service: Consumed 1.215s CPU time.
May 20 12:09:44 cirrus systemd[1]: Stopped firewalld.service - firewalld - dynamic firewall daemon.
May 20 12:09:44 cirrus systemd[1]: firewalld.service: Deactivated successfully.
May 20 12:09:43 cirrus systemd[1]: Stopping firewalld.service - firewalld - dynamic firewall daemon...
May 20 11:54:07 cirrus firewalld[714]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i br-a20b3536d08>
May 20 11:44:18 cirrus firewalld[714]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o doc>
May 20 11:44:17 cirrus firewalld[714]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATIO>
May 20 11:44:17 cirrus firewalld[714]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATIO>
May 20 11:44:17 cirrus firewalld[714]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATIO>
May 20 11:44:17 cirrus firewalld[714]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATIO>
May 20 11:44:17 cirrus firewalld[714]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATIO>
May 20 11:44:17 cirrus firewalld[714]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATIO>
May 20 11:44:17 cirrus firewalld[714]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER' failed:>
May 20 11:44:17 cirrus firewalld[714]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER' failed:>
May 20 11:44:17 cirrus firewalld[714]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -X DOCKER' failed: ip>
May 20 11:44:17 cirrus firewalld[714]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -F DOCKER' failed: ip>
May 20 11:44:17 cirrus firewalld[714]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT' failed: ip>
May 20 11:44:17 cirrus firewalld[714]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D PREROUTING' failed>
May 20 11:44:17 cirrus firewalld[714]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT -m addrtype>
                                       Try `iptables -h' or 'iptables --help' for more information.
May 20 11:44:17 cirrus firewalld[714]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT -m addrtype>
                                       Try `iptables -h' or 'iptables --help' for more information.
May 20 11:44:17 cirrus firewalld[714]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D PREROUTING -m addr>
                                       Try `iptables -h' or 'iptables --help' for more information.
May 20 11:06:31 fedora systemd[1]: Started firewalld.service - firewalld - dynamic firewall daemon.
May 20 11:06:31 fedora systemd[1]: Starting firewalld.service - firewalld - dynamic firewall daemon...

Today, I followed that link, and got this from the comments:

firewall-cmd --permanent --zone=trusted --add-interface=docker0
firewall-cmd --reload

but since doing that (after cleaning it all down to start from fresh). I canā€™t get past the domain name part of the AIO install.

Domain does not point to this server or reverse proxy not configured correctly.

looking in the firewall logs, they now look worse:

ay 21 10:11:49 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i br-111a657892b>
May 21 10:11:49 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i br-cef227f4784>
May 21 10:11:49 cirrus firewalld[747]: ERROR: ZONE_CONFLICT: 'docker0' already bound to 'trusted'
May 21 10:11:49 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o doc>
May 21 10:11:49 cirrus firewalld[747]: ERROR: ZONE_CONFLICT: 'docker0' already bound to 'trusted'
May 21 10:11:48 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o doc>
May 21 10:11:05 cirrus firewalld[747]: WARNING: ALREADY_ENABLED: docker0
May 21 09:56:47 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i br-111a657892b>
May 21 09:55:12 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i br-cef227f4784>
May 21 09:55:06 cirrus firewalld[747]: ERROR: ZONE_CONFLICT: 'docker0' already bound to 'trusted'
May 21 09:55:06 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o doc>
May 21 09:55:05 cirrus firewalld[747]: ERROR: ZONE_CONFLICT: 'docker0' already bound to 'trusted'
May 21 09:55:05 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o doc>
May 21 09:54:04 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o doc>
May 21 09:54:03 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o doc>
May 21 09:54:03 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATIO>
May 21 09:54:03 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATIO>
May 21 09:54:03 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATIO>
May 21 09:54:03 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATIO>
May 21 09:54:03 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATIO>
May 21 09:54:03 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATIO>
May 21 09:54:03 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER' failed:>
May 21 09:54:03 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER' failed:>
May 21 09:54:03 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -X DOCKER' failed: ip>
May 21 09:54:03 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -F DOCKER' failed: ip>
May 21 09:54:03 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT' failed: ip>
May 21 09:54:03 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D PREROUTING' failed>
May 21 09:54:03 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT -m addrtype>
                                       Try `iptables -h' or 'iptables --help' for more information.
May 21 09:54:03 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT -m addrtype>
                                       Try `iptables -h' or 'iptables --help' for more information.
May 21 09:54:03 cirrus firewalld[747]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D PREROUTING -m addr>
                                       Try `iptables -h' or 'iptables --help' for more information.
May 21 09:54:01 fedora systemd[1]: Started firewalld.service - firewalld - dynamic firewall daemon.
May 21 09:54:01 fedora systemd[1]: Starting firewalld.service - firewalld - dynamic firewall daemon...
43

Now iā€™m trying what the author suggested.

sudo sed -i 's/FirewallBackend=nftables/FirewallBackend=iptables/g' /etc/firewalld/firewalld.conf
sudo systemctl restart firewalld docker

I found that docker wouldnā€™t start. the two changes i had applied are seemingly incompatible. so if you tried adding docker0 to trusted, then remove it now:

sudo firewall-cmd --permanent --zone=trusted --remove-interface=docker0
sudo firewall-cmd --reload

iā€™m now in the long wait again to see this works as I download all the packages/containers etc.

but my firewalld logs still look less than good:

May 21 10:25:08 cirrus firewalld[748]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o doc>
May 21 10:25:08 cirrus firewalld[748]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i br-cef227f4784>
May 21 10:25:07 cirrus firewalld[748]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o doc>
May 21 10:25:07 cirrus firewalld[748]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i br-111a657892b>
May 21 10:25:07 cirrus firewalld[748]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATIO>
May 21 10:25:07 cirrus firewalld[748]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATIO>
May 21 10:25:07 cirrus firewalld[748]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATIO>
May 21 10:25:07 cirrus firewalld[748]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATIO>
May 21 10:25:07 cirrus firewalld[748]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATIO>
May 21 10:25:07 cirrus firewalld[748]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATIO>
May 21 10:25:07 cirrus firewalld[748]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER' failed:>
May 21 10:25:07 cirrus firewalld[748]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER' failed:>
May 21 10:25:07 cirrus firewalld[748]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -X DOCKER' failed: ip>
May 21 10:25:07 cirrus firewalld[748]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -F DOCKER' failed: ip>
May 21 10:25:07 cirrus firewalld[748]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT' failed: ip>
May 21 10:25:07 cirrus firewalld[748]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D PREROUTING' failed>
May 21 10:25:07 cirrus firewalld[748]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT -m addrtype>
                                       Try `iptables -h' or 'iptables --help' for more information.
May 21 10:25:07 cirrus firewalld[748]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT -m addrtype>
                                       Try `iptables -h' or 'iptables --help' for more information.
May 21 10:25:07 cirrus firewalld[748]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D PREROUTING -m addr>
                                       Try `iptables -h' or 'iptables --help' for more information.
44

Bingo.

Containers
 Apache (Running)
 Database (Running)
 Nextcloud (Running)
 Redis (Running)
 Collabora (Running)
 Talk (Running)
 ClamAV (Running)

I donā€™t know if this is a long term option, if fedora and itā€™s updates will be happy.
but itā€™s working, so thank you @szaimen and @SysKeeper

1 Like
45

@szaimen That was the trick! Learned a ton through this troubleshooting process, thanks. :slight_smile:

46

Great that you were able to fix it! :slight_smile:

I googled a bit and it seems like the issue is so well known that it is even mentioned on the official firewalld website https://firewalld.org/:
image

1 Like
47

Iā€™ve added this to our documentation now: GitHub - nextcloud/all-in-one: Nextcloud AIO stands for Nextcloud All In One and provides easy deployment and maintenance with most features included in this one Nextcloud instance.