Nextcloud running on docker GONE 404

axo · July 7, 2021, 11:51pm

Hello. I have been using my nextcloud instance running on a VPS with docker.
Today i woke up and my cloud is not there anymore… just 404… no access via web, mobile or desktop sync application.
I haven’t touched the server since running a couple updates with apt update like 3 weeks ago… so i have no clue where to start looking.

I am a noob with docker and barely understand how it works. So please could you guide me with the first obvious places where i should start looking for problems?

I got this:

root@vmi489807:/var/log# docker container ls -a
CONTAINER ID   IMAGE                                  COMMAND                  CREATED        STATUS                 PORTS                                      NAMES
ffe5ec01c9d5   collabora/code:latest                  "/bin/sh -c 'bash st…"   23 hours ago   Up 9 hours             9980/tcp                                   collabora_online
1b91597298b7   nextcloud:fpm-alpine                   "/entrypoint.sh php-…"   23 hours ago   Up 9 hours             9000/tcp                                   nextcloud
290effaca15d   redis:alpine                           "docker-entrypoint.s…"   23 hours ago   Up 9 hours (healthy)   6379/tcp                                   redis
57c005edc8e7   portainer/portainer-ce:latest          "/portainer --host u…"   11 days ago    Up 9 hours             8000/tcp, 9000/tcp                         portainer
949d37692f05   mariadb:latest                         "docker-entrypoint.s…"   12 days ago    Up 9 hours (healthy)   3306/tcp                                   nextcloud-db
e949deee693e   traefik:latest                         "/entrypoint.sh trae…"   2 weeks ago    Up 9 hours             0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   traefik
1824b5446fe9   tecnativa/docker-socket-proxy:latest   "/docker-entrypoint.…"   5 months ago   Up 9 hours             2375/tcp                                   dockerproxy
a8b7247157e6   containrrr/watchtower:latest           "/watchtower --clean…"   6 months ago   Up 9 hours             8080/tcp                                   watchtower
root@vmi489807:/var/log# docker exec 1b91597298b7 ls
3rdparty
AUTHORS
COPYING
apps
config
console.php
core
cron.php
custom_apps
data
index.html
index.php
lib
occ
ocm-provider
ocs
ocs-provider
public.php
remote.php
resources
robots.txt
status.php
themes
version.php
root@vmi489807:/var/log# docker exec -it 1b91597298b7 /bin/bash
OCI runtime exec failed: exec failed: container_linux.go:370: starting container process caused: exec: "/bin/bash": stat /bin/bash: no such file or directory: unknown
root@vmi489807:/var/log# docker exec -i -t 1b91597298b7 /bin/bash
OCI runtime exec failed: exec failed: container_linux.go:370: starting container process caused: exec: "/bin/bash": stat /bin/bash: no such file or directory: unknown
root@vmi489807:/var/log#

I am reading a few guides and i am trying to navigate the directories but i get the error shown here at the end of the code…

The last time i checked i was running this configuration:

Nextcloud 21.0.1
4CPU
8GB Ram
Ubuntu 20.04.2
PHP 8.0.7
PostgreSQL 13.3
Nginx server

Thanks for any help.

axo · July 9, 2021, 2:48am

Ok. I am able to browse the docker container now…
i see the nextcloud files are there…

root@vmi489807:~# docker exec -it nextcloud /bin/sh
/var/www/html # ls
3rdparty      COPYING       config        core          custom_apps   index.html    lib           ocm-provider  ocs-provider  remote.php    robots.txt    themes
AUTHORS       apps          console.php   cron.php      data          index.php     occ           ocs           public.php    resources     status.php    version.php

I did a test renaming the index files in the nextcloud folder and created a new index.html file with a hello world test message and i can not reach it. So i am assuming there may be a problem with the traefik image? As far as i understand, this image is like a router that directs the traffic to the appropriate port and server image right?

denNorske · July 9, 2021, 2:13pm

Can you show output of what
sudo docker logs traefik gives you?

Can you locate any error messages ?

if there is a lot of output, you can limit it with the following syntax;
sudo docker logs --follow --since 1m30s traefik

This will spit out logs from 1m30s ago, and follow the log output if new entries are appended as you check it.

It’s highly likely there’s something with traefik causing this yes, as it acts as a hub to access the nextcloud directories.

axo · July 9, 2021, 4:29pm

Thank you for the help!!

Here is the output for the traefik logs… i shaved it a bit as there are many similar error lines…

root@vmi284507:~# docker logs e949deee693e
time="2021-06-23T00:00:19Z" level=info msg="Configuration loaded from file: /traefik.yaml"
2021/06/23 03:10:55 reverseproxy.go:476: httputil: ReverseProxy read error during body copy: unexpected EOF
2021/06/23 03:10:55 reverseproxy.go:476: httputil: ReverseProxy read error during body copy: unexpected EOF
2021/06/23 03:10:55 reverseproxy.go:476: httputil: ReverseProxy read error during body copy: unexpected EOF
.........

unexpected EOF
2021/07/06 11:29:20 reverseproxy.go:476: httputil: ReverseProxy read error during body copy: unexpected EOF
2021/07/06 13:37:39 reverseproxy.go:476: httputil: ReverseProxy read error during body copy: unexpected EOF
time="2021-07-07T00:02:32Z" level=error msg="service \"nginx\" error: unable to find the IP address for the container \"/nginx\": the server is ignored" providerName=docker container=nginx-0e1433223a0648bf5419adb72b608a96172bf890ff8facea2a3f1e0f5c78f691
time="2021-07-07T00:02:34Z" level=error msg="service \"nginx\" error: unable to find the IP address for the container \"/nginx\": the server is ignored" providerName=docker container=nginx-0e1433223a0648bf5419adb72b608a96172bf890ff8facea2a3f1e0f5c78f691
time="2021-07-07T00:02:36Z" level=error msg="service \"nginx\" error: unable to find the IP address for the container \"/nginx\": the server is ignored" providerName=docker container=nginx-0e1433223a0648bf5419adb72b608a96172bf890ff8facea2a3f1e0f5c78f691
time="2021-07-07T00:02:36Z" level=error msg="service \"nginx\" error: unable to find the IP address for the container \"/nginx\": the server is ignored" providerName=docker container=nginx-0e1433223a0648bf5419adb72b608a96172bf890ff8facea2a3f1e0f5c78f691
time="2021-07-07T00:02:37Z" level=error msg="service \"nginx\" error: unable to find the IP address for the container \"/nginx\": the server is ignored" providerName=docker container=nginx-0e1433223a0648bf5419adb72b608a96172bf890ff8facea2a3f1e0f5c78f691
time="2021-07-07T00:02:37Z" level=error msg="service \"nginx\" error: unable to find the IP address for the container \"/nginx\": the server is ignored" providerName=docker container=nginx-0e1433223a0648bf5419adb72b608a96172bf890ff8facea2a3f1e0f5c78f691
time="2021-07-07T00:02:38Z" level=error msg="service \"nginx\" error: unable to find the IP address for the container \"/nginx\": the server is ignored" providerName=docker container=nginx-0e1433223a0648bf5419adb72b608a96172bf890ff8facea2a3f1e0f5c78f691
time="2021-07-07T00:02:38Z" level=error msg="service \"nginx\" error: unable to find the IP address for the container \"/nginx\": the server is ignored" providerName=docker container=nginx-0e1433223a0648bf5419adb72b608a96172bf890ff8facea2a3f1e0f5c78f691
time="2021-07-07T00:02:41Z" level=error msg="service \"nginx\" error: unable to find the IP address for the container \"/nginx\": the server is ignored" providerName=docker container=nginx-0e1433223a0648bf5419adb72b608a96172bf890ff8facea2a3f1e0f5c78f691
time="2021-07-07T00:02:41Z" level=error msg="service \"nginx\" error: unable to find the IP address for the container \"/nginx\": the server is ignored" providerName=docker container=nginx-0e1433223a0648bf5419adb72b608a96172bf890ff8facea2a3f1e0f5c78f691
time="2021-07-07T13:52:04Z" level=error msg="accept tcp [::]:443: use of closed network connection" entryPointName=web-secure
time="2021-07-07T13:52:04Z" level=error msg="accept tcp [::]:80: use of closed network connection" entryPointName=web
time="2021-07-07T13:52:04Z" level=error msg="Error while starting server: http: Server closed" entryPointName=web
time="2021-07-07T13:52:04Z" level=error msg="Error while starting server: http: Server closed" entryPointName=web-secure
time="2021-07-07T13:52:04Z" level=error msg="Error while starting server: http: Server closed" entryPointName=web-secure
time="2021-07-07T13:52:04Z" level=error msg="close tcp [::]:80: use of closed network connection" entryPointName=web
time="2021-07-07T13:52:04Z" level=error msg="close tcp [::]:443: use of closed network connection" entryPointName=web-secure
time="2021-07-07T13:52:04Z" level=error msg="Error while starting server: http: Server closed" entryPointName=web
time="2021-07-07T13:52:04Z" level=error msg="Failed to list containers for docker, error error during connect: Get \"http://dockerproxy:2375/v1.24/containers/json?limit=0\": context canceled" providerName=docker
time="2021-07-07T13:52:57Z" level=info msg="Configuration loaded from file: /traefik.yaml"
time="2021-07-07T14:28:22Z" level=error msg="accept tcp [::]:443: use of closed network connection" entryPointName=web-secure
time="2021-07-07T14:28:22Z" level=error msg="Error while starting server: http: Server closed" entryPointName=web-secure
time="2021-07-07T14:28:22Z" level=error msg="accept tcp [::]:80: use of closed network connection" entryPointName=web
time="2021-07-07T14:28:22Z" level=error msg="Error while starting server: http: Server closed" entryPointName=web-secure
time="2021-07-07T14:28:22Z" level=error msg="Error while starting server: http: Server closed" entryPointName=web
time="2021-07-07T14:28:22Z" level=error msg="Error while starting server: http: Server closed" entryPointName=web
time="2021-07-07T14:28:22Z" level=error msg="close tcp [::]:80: use of closed network connection" entryPointName=web
time="2021-07-07T14:28:22Z" level=error msg="close tcp [::]:443: use of closed network connection" entryPointName=web-secure
time="2021-07-07T14:28:22Z" level=error msg="Failed to list containers for docker, error Cannot connect to the Docker daemon at tcp://dockerproxy:2375. Is the docker daemon running?" providerName=docker
time="2021-07-07T14:28:22Z" level=error msg="Failed to list containers for docker, error Cannot connect to the Docker daemon at tcp://dockerproxy:2375. Is the docker daemon running?" providerName=docker
time="2021-07-07T14:28:22Z" level=error msg="Failed to list containers for docker, error Cannot connect to the Docker daemon at tcp://dockerproxy:2375. Is the docker daemon running?" providerName=docker
time="2021-07-07T14:28:22Z" level=error msg="Failed to list containers for docker, error Cannot connect to the Docker daemon at tcp://dockerproxy:2375. Is the docker daemon running?" providerName=docker
time="2021-07-07T14:30:57Z" level=info msg="Configuration loaded from file: /traefik.yaml"
time="2021-07-08T18:14:17Z" level=error msg="accept tcp [::]:80: use of closed network connection" entryPointName=web
time="2021-07-08T18:14:17Z" level=error msg="Error while starting server: http: Server closed" entryPointName=web
time="2021-07-08T18:14:17Z" level=error msg="accept tcp [::]:443: use of closed network connection" entryPointName=web-secure
time="2021-07-08T18:14:17Z" level=error msg="Error while starting server: http: Server closed" entryPointName=web-secure
time="2021-07-08T18:14:17Z" level=error msg="Error while starting server: http: Server closed" entryPointName=web
time="2021-07-08T18:14:17Z" level=error msg="Error while starting server: http: Server closed" entryPointName=web-secure
time="2021-07-08T18:14:17Z" level=error msg="close tcp [::]:80: use of closed network connection" entryPointName=web
time="2021-07-08T18:14:17Z" level=error msg="close tcp [::]:443: use of closed network connection" entryPointName=web-secure
time="2021-07-08T18:14:40Z" level=info msg="Configuration loaded from file: /traefik.yaml"
time="2021-07-08T18:14:48Z" level=error msg="accept tcp [::]:443: use of closed network connection" entryPointName=web-secure
time="2021-07-08T18:14:48Z" level=error msg="Error while starting server: accept tcp [::]:443: use of closed network connection" entryPointName=web-secure
time="2021-07-08T18:14:48Z" level=error msg="Error while starting server: http: Server closed" entryPointName=web-secure
time="2021-07-08T18:14:48Z" level=error msg="accept tcp [::]:80: use of closed network connection" entryPointName=web
time="2021-07-08T18:14:48Z" level=error msg="Error while starting server: accept tcp [::]:80: use of closed network connection" entryPointName=web
time="2021-07-08T18:14:48Z" level=error msg="Error while starting server: http: Server closed" entryPointName=web
time="2021-07-08T18:15:02Z" level=info msg="Configuration loaded from file: /traefik.yaml"
time="2021-07-08T18:26:58Z" level=error msg="accept tcp [::]:443: use of closed network connection" entryPointName=web-secure
time="2021-07-08T18:26:58Z" level=error msg="accept tcp [::]:80: use of closed network connection" entryPointName=web
time="2021-07-08T18:26:58Z" level=error msg="Error while starting server: http: Server closed" entryPointName=web
time="2021-07-08T18:26:58Z" level=error msg="Error while starting server: http: Server closed" entryPointName=web-secure
time="2021-07-08T18:26:58Z" level=error msg="Error while starting server: http: Server closed" entryPointName=web
time="2021-07-08T18:26:58Z" level=error msg="Error while starting server: http: Server closed" entryPointName=web-secure
time="2021-07-08T18:26:58Z" level=error msg="close tcp [::]:80: use of closed network connection" entryPointName=web
time="2021-07-08T18:26:58Z" level=error msg="close tcp [::]:443: use of closed network connection" entryPointName=web-secure
time="2021-07-08T18:27:48Z" level=info msg="Configuration loaded from file: /traefik.yaml"
time="2021-07-09T00:31:59Z" level=error msg="accept tcp [::]:443: use of closed network connection" entryPointName=web-secure
time="2021-07-09T00:31:59Z" level=error msg="accept tcp [::]:80: use of closed network connection" entryPointName=web
time="2021-07-09T00:31:59Z" level=error msg="Error while starting server: http: Server closed" entryPointName=web
time="2021-07-09T00:31:59Z" level=error msg="Error while starting server: http: Server closed" entryPointName=web
time="2021-07-09T00:31:59Z" level=error msg="close tcp [::]:80: use of closed network connection" entryPointName=web
time="2021-07-09T00:31:59Z" level=error msg="Error while starting server: http: Server closed" entryPointName=web-secure
time="2021-07-09T00:31:59Z" level=error msg="Error while starting server: http: Server closed" entryPointName=web-secure
time="2021-07-09T00:31:59Z" level=error msg="close tcp [::]:443: use of closed network connection" entryPointName=web-secure
time="2021-07-09T00:32:20Z" level=info msg="Configuration loaded from file: /traefik.yaml"

So i guess the nginx server is not being seen?

then here is the traefik.yaml file:

root@vmi284507:~# docker exec -it e949deee693e /bin/sh
/ # ls
acme.json      dev            entrypoint.sh  home           media          opt            root           sbin           sys            traefik.yaml   var
bin            dynamic.yaml   etc            lib            mnt            proc           run            srv            tmp            usr

/ # cat traefik.yaml
## Static Configuration

api:
  dashboard: false

entryPoints:
  web:
    address: ':80'
    http:
      redirections:
        entryPoint:
          to: web-secure

  web-secure:
    address: ':443'
    http:
      middlewares:
        - secureHeaders@file
      tls:
        certResolver: letsencrypt

providers:
  docker:
    exposedByDefault: false
    endpoint: 'tcp://dockerproxy:2375'
    network: 'dockerproxy'
  file:
    watch: true
    filename: 'dynamic.yaml'

log:
  level: error
  
certificatesResolvers:
  letsencrypt:
    acme:
      email: "axo@nextcloudserver.com"
      storage: 'acme.json'
      tlsChallenge: {}
/ #

denNorske · July 9, 2021, 4:36pm

So it might look like that you were supposed to have a nginx container running, and it is not currently running. I might be wrong (I haven’t used traefik actively yet). But it looks like you are using traefik to route the traffic between the container to the host.

Maybe nginx is not in a container, maybe it was running on your server directly
(in so case, check status with sudo service nginx status )

Try to see if you can locate either the container or service, and try to get it started

axo · July 9, 2021, 11:37pm

Ok now i am puzzled…

this is from another post:

root@vmi284507:/etc# docker images
REPOSITORY                      TAG          IMAGE ID       CREATED        SIZE
nginx                           alpine       b9e2356ea1be   2 days ago     22.8MB
nextcloud                       fpm-alpine   236141c6113c   2 days ago     568MB
collabora/code                  latest       9a428589679f   2 days ago     1.39GB
redis                           alpine       500703a12fa4   2 days ago     32.3MB
portainer/portainer-ce          latest       8bd64518b976   2 weeks ago    210MB
mariadb                         latest       6d5c5ed114ad   2 weeks ago    408MB
traefik                         latest       99f7809fa346   2 weeks ago    92MB
tecnativa/docker-socket-proxy   latest       e419047c5d43   5 months ago   21.1MB
adminer                         latest       234dd75608e9   5 months ago   91.3MB
containrrr/watchtower           latest       2ddeb26d6d30   6 months ago   15.2MB

Notice the nginx server image showing inside the docker container ???

Well… now it is not showing anymore:

root@vmi284507:~# docker ps -a
CONTAINER ID   IMAGE                                  COMMAND                  CREATED        STATUS                 PORTS                                      NAMES
ffe5ec01c9d5   collabora/code:latest                  "/bin/sh -c 'bash st…"   2 days ago     Up 7 hours             9980/tcp                                   collabora_online
1b91597298b7   nextcloud:fpm-alpine                   "/entrypoint.sh php-…"   2 days ago     Up 7 hours             9000/tcp                                   nextcloud
290effaca15d   redis:alpine                           "docker-entrypoint.s…"   2 days ago     Up 7 hours (healthy)   6379/tcp                                   redis
57c005edc8e7   portainer/portainer-ce:latest          "/portainer --host u…"   13 days ago    Up 7 hours             8000/tcp, 9000/tcp                         portainer
949d37692f05   mariadb:latest                         "docker-entrypoint.s…"   2 weeks ago    Up 7 hours (healthy)   3306/tcp                                   nextcloud-db
e949deee693e   traefik:latest                         "/entrypoint.sh trae…"   2 weeks ago    Up 7 hours             0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   traefik
1824b5446fe9   tecnativa/docker-socket-proxy:latest   "/docker-entrypoint.…"   5 months ago   Up 7 hours             2375/tcp                                   dockerproxy
a8b7247157e6   containrrr/watchtower:latest           "/watchtower --clean…"   6 months ago   Up 7 hours             8080/tcp                                   watchtower

I remember this morning i was browsing the docker container and nginx was not there… and then it showed up, but now it is gone again…

I just took a look at the commands history… here:

docker exec -it nginx /bin/sh

I remember i was not able to browse it… i got an error “no such container”… the same happens if i try to browse collabora, mariadb or portainer… (showing off my ignorance with docker here)

When nginx was showing up in the list, the server was not working anyways… same issue

To answer your question… nginx is not registered as a main service in the server… it was loaded inside the docker during the installation.

So my noob question here…
What happens if i just try to pull the nginx image to the container from the docker servers? as far as i understand traefik should interconnect everything automatically… but i do not want to make a mistake and lose this setup, i would have to update links i shared with many people…
Isn’t docker like a soup?.. you just add more ingredients to it and they will just blend together? Or i will need to internally setup things for the new nginx image inside the container?

Thanks for the enlightenment

denNorske · July 10, 2021, 8:35am

Ah I see!
Now it would be nice to know how you have sat everything up to run, in terms of docker commands. I can see that you are not running it in docker-compose based on the naming of the containers (but still possible).

So, how are all the containers run by you? Are you using docker run [...] for instance?

If yes, a possible way to find the nginx configuration you have used before, is to check your bash history file on the host.
try
history | grep nginx

Docker is essentially running pre-defined images in containers. And you can modify their behavior by passing in environment variables or by mounting in relevant configuration files.

In your case i would expect that nextcloud is exposing a volume, or is using a host-mounted volume for its files. In that case, nginx needs to know where these files are, so that it can (as a webserver) serve those files to the user

Let me know what you find!

axo · July 10, 2021, 4:33pm

I used an ansible playbook for the whole setup: https://github.com/ReinerNippes/nextcloud_on_docker

I am navigating all the files there and i am completely lost, like reading chinese… Im not even sure which file is the list of instructions or commands that docker executes.

axo · July 10, 2021, 4:43pm

In my noobish attempt i am trying to “start” nginx without luck:

root@vmi284507:~# docker start nginx
Error response from daemon: No such container: nginx
Error: failed to start containers: nginx
root@vmi284507:~# docker images ls
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE
root@vmi284507:~# docker images 
REPOSITORY                      TAG          IMAGE ID       CREATED        SIZE
collabora/code                  latest       9c323ee534bb   22 hours ago   1.39GB
nginx                           alpine       b9e2356ea1be   3 days ago     22.8MB
nextcloud                       fpm-alpine   236141c6113c   3 days ago     568MB
redis                           alpine       500703a12fa4   3 days ago     32.3MB
portainer/portainer-ce          latest       8bd64518b976   2 weeks ago    210MB
mariadb                         latest       6d5c5ed114ad   2 weeks ago    408MB
traefik                         latest       99f7809fa346   2 weeks ago    92MB
tecnativa/docker-socket-proxy   latest       e419047c5d43   5 months ago   21.1MB
adminer                         latest       234dd75608e9   5 months ago   91.3MB
containrrr/watchtower           latest       2ddeb26d6d30   6 months ago   15.2MB
root@vmi284507:~# docker stop nextcloud
nextcloud
root@vmi284507:~# docker ps
CONTAINER ID   IMAGE                                  COMMAND                  CREATED        STATUS                  PORTS                                      NAMES
0bdd4e53e31c   collabora/code:latest                  "/bin/sh -c 'bash st…"   17 hours ago   Up 17 hours             9980/tcp                                   collabora_online
57c005edc8e7   portainer/portainer-ce:latest          "/portainer --host u…"   2 weeks ago    Up 17 hours             8000/tcp, 9000/tcp                         portainer
949d37692f05   mariadb:latest                         "docker-entrypoint.s…"   2 weeks ago    Up 17 hours (healthy)   3306/tcp                                   nextcloud-db
e949deee693e   traefik:latest                         "/entrypoint.sh trae…"   2 weeks ago    Up 17 hours             0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   traefik
1824b5446fe9   tecnativa/docker-socket-proxy:latest   "/docker-entrypoint.…"   5 months ago   Up 17 hours             2375/tcp                                   dockerproxy
a8b7247157e6   containrrr/watchtower:latest           "/watchtower --clean…"   6 months ago   Up 17 hours             8080/tcp                                   watchtower
root@vmi284507:~# docker start nextcloud
nextcloud
root@vmi284507:~# docker stop nginx
Error response from daemon: No such container: nginx
[quote="axo, post:8, topic:119806, full:true"]
I used an ansible playbook for the whole setup: [https://github.com/ReinerNippes/nextcloud_on_docker](https://github.com/ReinerNippes/nextcloud_on_docker)

I am navigating all the files there and i am completely lost, like reading chinese.. Im not even sure which file is the list of instructions or commands that docker executes.

denNorske · July 11, 2021, 2:57pm

I have never used Ansible before, but it sure looks like Ansible is responsible for putting everything online when you start the configuration.

Are you able to take all the stuff offline, and then just restart with the configuration script?

Or if you can find the playbook for nginx.
in the repo, I find it under:

I am not sure, but this command seem to be related; https://docs.ansible.com/ansible/latest/cli/ansible-playbook.html#ansible-playbook

I hope that someone with knowledge with Ansible can help out here though.