Can't stop nextcloud-aio-nextcloud container

ℹ️ Support 📦 Appliances (Docker, Snappy, VM, NCP, AIO)
, ,
1

system information
Nextcloud version : 28
Nextcloud AIO version : 8.2.0 latest
Operating system and version : Ubuntu 22.04.4 LTS
Docker (compose)

The issue you are facing:

I have been facing issues that my NC-AIO instance has been dying whenever it tried to pull a backup via the integrated borg backup service. Docker shows the nextcloud-aio-nextcloud container as unhealthy, and when trying to restart it, I get an error message (via command line tool, I get a generic error, and via portainer, I get a 500 error).

The solution has been to restart the system, which seems to resolve the issue. When checking the logs, it seems like the mastercontainer is unable to stop the nextcloud container, so I pulled a manual backup and tried to recreate the containers (via command line, via portainer and by deleting the containers and recreating them via the AIO interface).

After recreation, the containers started up manually, so I tried to gracefully stop them via the AIO interface, and the nextcloud-aio-nextcloud container is once again unhealthy and the AIO interface is stuck in a loading screen.

I am using a stock configuration of the compose file with the only difference being that I moved the nextcloud data path to my NAS. This has been working for about a year, so I can’t believe that the issue lies there. All my containers are up to date according to the AIO interface and I am still running NC 28, with the AIO version having recently been upgraded to 8.2.0.

Log of the nextcloud-aio-nextcloud container

This will probably be the most useful, as all the other containers shut down gracefully:

Connection to nextcloud-aio-database (172.21.0.5) 5432 port [tcp/postgresql] succeeded!
             now              
------------------------------
 2024-04-24 19:15:26.12831+02
(1 row)
+ '[' -f /dev-dri-group-was-added ']'
++ find /dev -maxdepth 1 -mindepth 1 -name dri
+ '[' -n '' ']'
+ set +x
Enabling Imagick...
WARNING: opening from cache https://dl-cdn.alpinelinux.org/alpine/v3.19/main: No such file or directory
WARNING: opening from cache https://dl-cdn.alpinelinux.org/alpine/v3.19/community: No such file or directory
Applying one-click-instance settings...
System config value one-click-instance set to boolean true
System config value one-click-instance.user-limit set to integer 100
System config value one-click-instance.link set to string https://nextcloud.com/all-in-one/
support already enabled
Adjusting log files...
System config value upgrade.cli-upgrade-link set to string https://github.com/nextcloud/all-in-one/discussions/2726
System config value logfile set to string /var/www/html/data/nextcloud.log
Config value logfile for app admin_audit set to /var/www/html/data/audit.log
System config value updatedirectory set to string /nc-updater
Applying network settings...
System config value allow_local_remote_servers set to boolean true
System config value davstorage.request_timeout set to integer 3600
System config value trusted_domains => 1 set to string cloud.art-lab.dev
System config value overwrite.cli.url set to string https://cloud.art-lab.dev/
System config value htaccess.RewriteBase set to string /
.htaccess has been updated
System config value dbpersistent set to boolean false
System config value auth.bruteforce.protection.enabled set to boolean true
System config value ratelimit.protection.enabled set to boolean true
System config value files_external_allow_create_new_local set to boolean false
System config value trusted_proxies => 0 set to string 127.0.0.1
System config value trusted_proxies => 1 set to string ::1
Config value base_endpoint for app notify_push set to https://cloud.art-lab.dev/push
+ echo nextcloud-aio-collabora
+ grep -q 'nextcloud-.*-collabora'
+ COLLABORA_HOST=cloud.art-lab.dev
+ set +x
Config value wopi_url for app richdocuments set to https://cloud.art-lab.dev/
No ipv6-address found for cloud.art-lab.dev.
Config value wopi_allowlist for app richdocuments set to 85.0.1.95,127.0.0.1/8,192.168.0.0/16,172.16.0.0/12,10.0.0.0/8,fd00::/8,::1,82.174.115.29,188.88.179.30
+ '[' -z '' ']'
+ TALK_HOST=cloud.art-lab.dev
+ HPB_PATH=/standalone-signaling/
+ '[' -z '' ']'
+ TURN_DOMAIN=cloud.art-lab.dev
+ set +x
Connection to nextcloud-aio-talk-recording (172.21.0.9) 1234 port [tcp/*] succeeded!
Config value recording_servers for app spreed set to {"servers":[{"server":"http://nextcloud-aio-talk-recording:1234/","verify":true}],"secret":"a9a1c218892c4ccf183a6efeeb33002cb126a2bd06d8edfb"}
Connection to nextcloud-aio-clamav (172.21.0.7) 3310 port [tcp/*] succeeded!
Config value av_mode for app files_antivirus set to daemon
Config value av_port for app files_antivirus set to 3310
Config value av_host for app files_antivirus set to nextcloud-aio-clamav
Config value av_stream_max_length for app files_antivirus set to 104857600
Config value av_max_file_size for app files_antivirus set to 104857600
Config value av_infected_action for app files_antivirus set to only_log
System config value enabledPreviewProviders => 0 set to string OC\Preview\Imaginary
System config value preview_imaginary_url set to string http://nextcloud-aio-imaginary:9000
System config value preview_imaginary_key set to string 87c84aa38a4b2005766c0cecd8b7ed29f3631806b9b0e27b
Connection to nextcloud-aio-fulltextsearch (172.21.0.8) 9200 port [tcp/*] succeeded!
{
    "search_platform": "OCA\\FullTextSearch_Elasticsearch\\Platform\\ElasticSearchPlatform",
    "app_navigation": "0",
    "provider_indexed": "{\"test_provider\":\"0\",\"deck\":\"1\",\"files\":\"1\"}",
    "cron_err_reset": "1713774254",
    "tick_ttl": "1800",
    "collection_indexing_list": "50",
    "migration_24": "1",
    "collection_internal": "local"
}
{
    "elastic_host": "http:\/\/elastic:7009ffddeaeec13c3af759d868634f5221c9bb7e978e8574@nextcloud-aio-fulltextsearch:9200",
    "elastic_index": "nextcloud-aio",
    "fields_limit": "10000",
    "es_ver_below66": "0",
    "elastic_logger_enabled": "1",
    "analyzer_tokenizer": "standard",
    "allow_self_signed_cert": "false"
}
{
    "files_local": "1",
    "files_external": "0",
    "files_group_folders": "0",
    "files_encrypted": "0",
    "files_federated": "0",
    "files_size": "20",
    "files_pdf": "1",
    "files_office": "1",
    "files_image": "0",
    "files_audio": "0",
    "files_chunk_size": "2"
}
+ '[' true = true ']'
+ '[' 11000 = 443 ']'
+ '[' 127.0.0.1 = 127.0.0.1 ']'
++ dig nextcloud-aio-apache A +short +search
++ grep '^[0-9.]\+$'
++ sort
++ head -n1
+ IPv4_ADDRESS_APACHE=172.21.0.13
++ dig nextcloud-aio-apache AAAA +short +search
++ grep '^[0-9a-f:]\+$'
++ sort
++ head -n1
+ IPv6_ADDRESS_APACHE=
++ dig nextcloud-aio-mastercontainer A +short +search
++ grep '^[0-9.]\+$'
++ sort
++ head -n1
+ IPv4_ADDRESS_MASTERCONTAINER=172.21.0.4
++ dig nextcloud-aio-mastercontainer AAAA +short +search
++ grep '^[0-9a-f:]\+$'
++ sort
++ head -n1
+ IPv6_ADDRESS_MASTERCONTAINER=
+ sed -i 's|^;listen.allowed_clients|listen.allowed_clients|' /usr/local/etc/php-fpm.d/www.conf
+ sed -i 's|listen.allowed_clients.*|listen.allowed_clients = 127.0.0.1,::1,172.21.0.13,,172.21.0.4,|' /usr/local/etc/php-fpm.d/www.conf
+ sed -i '/^listen.allowed_clients/s/,,/,/g' /usr/local/etc/php-fpm.d/www.conf
+ sed -i '/^listen.allowed_clients/s/,$//' /usr/local/etc/php-fpm.d/www.conf
+ grep listen.allowed_clients /usr/local/etc/php-fpm.d/www.conf
listen.allowed_clients = 127.0.0.1,::1,172.21.0.13,172.21.0.4
+ set +x
[24-Apr-2024 19:16:12] NOTICE: fpm is running, pid 545
[24-Apr-2024 19:16:12] NOTICE: ready to handle connections
Activating Collabora config...
✓ Reset callback url autodetect
Checking configuration
🛈 Configured WOPI URL: https://cloud.art-lab.dev/
🛈 Configured public WOPI URL: https://cloud.art-lab.dev
🛈 Configured callback URL: 
✓ Fetched /hosting/discovery endpoint
✓ Valid mimetype response
✓ Valid capabilities entry
✓ Fetched /hosting/capabilities endpoint
✓ Detected WOPI server: Collabora Online Development Edition 23.05.10.1
Collabora URL (used for Nextcloud to contact the Collabora server):
  https://cloud.art-lab.dev/
Collabora public URL (used in the browser to open Collabora):
  https://cloud.art-lab.dev
Callback URL (used by Collabora to connect back to Nextcloud):
  autodetected (will use the same URL as your user for browsing Nextcloud)
[24-Apr-2024 19:16:50] WARNING: [pool www] child 573 exited on signal 9 (SIGKILL) after 16.460371 seconds from start
[24-Apr-2024 19:16:50] NOTICE: [pool www] child 575 started
[24-Apr-2024 19:16:57] WARNING: [pool www] child 572 exited on signal 9 (SIGKILL) after 23.008593 seconds from start
[24-Apr-2024 19:16:57] NOTICE: [pool www] child 586 started
[24-Apr-2024 19:16:57] WARNING: [pool www] child 574 exited on signal 9 (SIGKILL) after 23.197863 seconds from start
[24-Apr-2024 19:16:57] NOTICE: [pool www] child 587 started
[24-Apr-2024 19:16:57] WARNING: [pool www] child 550 exited on signal 9 (SIGKILL) after 43.659201 seconds from start
[24-Apr-2024 19:16:57] NOTICE: [pool www] child 588 started
[24-Apr-2024 19:16:57] WARNING: [pool www] child 585 exited on signal 9 (SIGKILL) after 0.885860 seconds from start
[24-Apr-2024 19:16:57] NOTICE: [pool www] child 589 started
[24-Apr-2024 19:16:57] WARNING: [pool www] child 575 exited on signal 9 (SIGKILL) after 7.290069 seconds from start
[24-Apr-2024 19:16:57] NOTICE: [pool www] child 590 started
[24-Apr-2024 19:16:58] WARNING: [pool www] child 586 exited on signal 9 (SIGKILL) after 0.903060 seconds from start
[24-Apr-2024 19:16:58] NOTICE: [pool www] child 591 started
[24-Apr-2024 19:16:58] WARNING: [pool www] child 587 exited on signal 9 (SIGKILL) after 0.787958 seconds from start
[24-Apr-2024 19:16:58] NOTICE: [pool www] child 592 started
[24-Apr-2024 19:16:58] WARNING: [pool www] child 588 exited on signal 9 (SIGKILL) after 0.655938 seconds from start
[24-Apr-2024 19:16:58] NOTICE: [pool www] child 593 started
[24-Apr-2024 19:16:58] WARNING: [pool www] child 589 exited on signal 9 (SIGKILL) after 0.511838 seconds from start
[24-Apr-2024 19:16:58] NOTICE: [pool www] child 594 started
[24-Apr-2024 19:16:58] WARNING: [pool www] child 591 exited on signal 9 (SIGKILL) after 0.210153 seconds from start
[24-Apr-2024 19:16:58] NOTICE: [pool www] child 595 started
[24-Apr-2024 19:16:58] WARNING: [pool www] child 592 exited on signal 9 (SIGKILL) after 0.193775 seconds from start
[24-Apr-2024 19:16:58] NOTICE: [pool www] child 596 started
[24-Apr-2024 19:30:13] NOTICE: Terminating ...
+++ pgrep -f /var/www/html/cron.php
++ '[' -n '685
793' ']'
++ echo 'Waiting for cron to stop...'
++ sleep 5
Waiting for cron to stop...
+++ pgrep -f /var/www/html/cron.php
Waiting for cron to stop...
++ '[' -n '685
793' ']'
++ echo 'Waiting for cron to stop...'
++ sleep 5
+++ pgrep -f /var/www/html/cron.php
2

Hi, as you can see from the logs, the nextcloud container is waiting for the cronjob to stop. This can take up to 10min. This is why it feels like the aio interface gets stuck but it is actually waiting for the cronjob.

3

Hi szaimen!

Thanks so much for your answer, you are absolutely right. The issue was that the container stopped, but could no longer be restarted (e.g. nightly backups stopped the container, and then couldn’t restart them anymore). After some research, it seems like somehow, my docker-ce installation was corrupted. That might have happened due to some system update.

I have now reinstalled my docker-ce instance and am currently rebuilding my AIO from my last backup. Wish me luck :slight_smile:

1 Like