Nextcloud down and daily backup stuck

Today my Nextcloud instance went down but I don’t see errors in the logs.
All containers are down except the aio-mastercontainer. The logs and the AIO dashboard show that the daily backup is in progress but it usually doesn’t happen at this time. The daily backup should have happened 9 hours ago.

AIO dashboard:

Logs of the master container:

Daily backup script has started
grep: write error: Broken pipe
Connection to nextcloud-aio-apache (172.18.0.10) 11000 port [tcp/*] succeeded!
Starting mastercontainer update...
(The script might get exited due to that. In order to update all the other containers correctly, you need to run this script with the same settings a second time.)
Could not get digest of container nextcloud/aio-watchtower:latest cURL error 28: SSL connection timeout (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://registry-1.docker.io/v2/nextcloud/aio-watchtower/manifests/latest
Not pulling the image for the nextcloud/aio-watchtower container because docker hub does not seem to be reachable.
Waiting for watchtower to stop
Waiting for watchtower to stop
Creating daily backup...
Deleting duplicate sessions
NOTICE: PHP message: 404 Not Found
Type: Slim\Exception\HttpNotFoundException

4
Message: Not found.
File: /var/www/docker-aio/php/vendor/slim/slim/Slim/Middleware/RoutingMiddleware.php
Line: 76
Trace: #0 /var/www/docker-aio/php/vendor/slim/slim/Slim/Routing/RouteRunner.php(56): Slim\Middleware\RoutingMiddleware->performRouting(Object(GuzzleHttp\Psr7\ServerRequest))
#1 /var/www/docker-aio/php/vendor/slim/csrf/src/Guard.php(482): Slim\Routing\RouteRunner->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#2 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(168): Slim\Csrf\Guard->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Slim\Routing\RouteRunner))
#3 /var/www/docker-aio/php/vendor/slim/twig-view/src/TwigMiddleware.php(115): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#4 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(121): Slim\Views\TwigMiddleware->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#5 /var/www/docker-aio/php/src/Middleware/AuthMiddleware.php(38): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#6 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(269): AIO\Middleware\AuthMiddleware->__invoke(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#7 /var/www/docker-aio/php/vendor/slim/slim/Slim/Middleware/ErrorMiddleware.php(76): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#8 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(121): Slim\Middleware\ErrorMiddleware->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#9 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(65): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#10 /var/www/docker-aio/php/vendor/slim/slim/Slim/App.php(199): Slim\MiddlewareDispatcher->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#11 /var/www/docker-aio/php/vendor/slim/slim/Slim/App.php(183): Slim\App->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#12 /var/www/docker-aio/php/public/index.php(185): Slim\App->run()
#13 {main}
Tips: To display error details in HTTP response set "displayErrorDetails" to true in the ErrorHandler constructor.

Logs of watchtower:

time="2024-05-30T13:47:43Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
time="2024-05-30T13:47:44Z" level=debug msg="Making sure everything is sane before starting"
time="2024-05-30T13:47:44Z" level=info msg="Watchtower 1.7.1"
time="2024-05-30T13:47:44Z" level=info msg="Using no notifications"
time="2024-05-30T13:47:44Z" level=info msg="Only checking containers which name matches \"nextcloud-aio-mastercontainer\""
time="2024-05-30T13:47:44Z" level=info msg="Running a one time update."
time="2024-05-30T13:47:44Z" level=debug msg="Checking containers for updated images"
time="2024-05-30T13:47:44Z" level=debug msg="Retrieving running containers"
time="2024-05-30T13:47:44Z" level=debug msg="Trying to load authentication credentials." container=/nextcloud-aio-mastercontainer image="nextcloud/all-in-one:latest"
time="2024-05-30T13:47:44Z" level=debug msg="No credentials for index.docker.io found" config_file=/config.json
time="2024-05-30T13:47:44Z" level=debug msg="Got image name: nextcloud/all-in-one:latest"
time="2024-05-30T13:47:44Z" level=debug msg="Checking if pull is needed" container=/nextcloud-aio-mastercontainer image="nextcloud/all-in-one:latest"
time="2024-05-30T13:47:44Z" level=debug msg="Built challenge URL" URL="https://index.docker.io/v2/"
time="2024-05-30T13:47:45Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
time="2024-05-30T13:47:45Z" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
time="2024-05-30T13:47:45Z" level=debug msg="Setting scope for auth token" image=docker.io/nextcloud/all-in-one scope="repository:nextcloud/all-in-one:pull"
time="2024-05-30T13:47:45Z" level=debug msg="No credentials found."
time="2024-05-30T13:47:46Z" level=debug msg="Parsing image ref" host=index.docker.io image=nextcloud/all-in-one normalized=docker.io/nextcloud/all-in-one tag=latest
time="2024-05-30T13:47:46Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/nextcloud/all-in-one/manifests/latest"
time="2024-05-30T13:47:46Z" level=debug msg="Found a remote digest to compare with" remote="sha256:9cbff5762d12bda2fd1d45150872212f64131483a2a0f77311d6da8b3fbefbdc"
time="2024-05-30T13:47:46Z" level=debug msg=Comparing local="sha256:9cbff5762d12bda2fd1d45150872212f64131483a2a0f77311d6da8b3fbefbdc" remote="sha256:9cbff5762d12bda2fd1d45150872212f64131483a2a0f77311d6da8b3fbefbdc"
time="2024-05-30T13:47:46Z" level=debug msg="Found a match"
time="2024-05-30T13:47:46Z" level=debug msg="No pull needed. Skipping image."
time="2024-05-30T13:47:46Z" level=debug msg="No new images found for /nextcloud-aio-mastercontainer"
time="2024-05-30T13:47:46Z" level=info msg="Session done" Failed=0 Scanned=1 Updated=0 notify=no
time="2024-05-30T13:47:46Z" level=info msg="Waiting for the notification goroutine to finish" notify=no

Hi, can you open a bug report here?

For tracking

1 Like

Sure, I’ll do it. Thanks for the reply.

Created this issue: Nexcloud down and daily backup stuck · Issue #4736 · nextcloud/all-in-one · GitHub

1 Like

Do you know how to cancel an ongoing backup? Or anything to unblock this situation? My Nextcloud is unusable…

yes, you need to remove the daily_backup_running file in the mastercontainer volume inside the data folder in there. Then reloading should show the controls again.

1 Like

Thank you! I could see again the controls in the AIO dashboard. Then I restarted the containers and all seems good now.
The backup log says this:


The backup only took a minute to complete, but for some reason something got stuck and Nextcloud went offline for hours…