Well, still having the nextcloud-aio-apache container is in a Created state crash every 4 - 5 days. I finally realized it was inevitable and likely something I would just have to live with. So, I figured out how to restart everything via bash. Those commands are here. The docker ps -a are just so I can see the progress. The docker exec at the end will restart all the NextCloud AIO sub-containers.
cd /var/lib/docker
docker ps -a
docker container prune -f
docker ps -a
docker compose down
docker ps -a
docker rm -f $(docker container ls -q --filter name=nextcloud*)
docker ps -a
service docker restart
docker compose up -d --pull always
docker ps -a
docker exec -it nextcloud-aio-mastercontainer sudo -u www-data php /var/www/docker-aio/php/src/Cron/StartAndUpdateContainers.php
docker ps -a
From there I started thinking about how to automate this so I wasn’t having to manually do it every time the site crashed. It could be hours between when it crashed and when I noticed or someone messaged me. I figured a website checking script would probably do what I needed. I searched and found one and then adopted it in to a bash script.
Here is that checking script that also does the restart when it encounters a 503. It also sends me an email and text (Verizon email-to-text). I am using mutt/msmtp for this on Ubuntu sending through Gmail. I am not covering that setup, but can if someone really needs it.
Storage.Website.Errors.sh
#!/bin/bash
#Purpose = Checker to see if Storage site is up. If 502 error, restart docker.
#START
# Variables
RECIPIENT="DanGilbertTX@aol.com"
TEXTRECIPIENT="8888675309@vtext.com"
SUBJECT="Forced to reset 502 for Storage on $(date +%A", "%B" "%d", "%Y)"
MESSAGE="The script found a 502 error and executed the reset script on Storage for docker. Please check to make sure it worked."
NON502SUBJECT="The website is down for Storage for an unknwon reason on $(date +%A", "%B" "%d", "%Y)"
NON502MESSAGE="The website is down for Storage for some unknown reason. Please check it."
response=$(/usr/bin/curl -s -o /dev/null -w "%{http_code}" https://storage.myredactedsite.org/s/Shared)
if [ "${response}" == "200" ]; then
# Everything looks good. No need for a notification.
echo "Website is up and working!"
elif [ "${response}" == "502" ]; then
echo "502 error. Resetting docker."
cd /var/lib/docker
/usr/bin/docker container prune -f
/usr/bin/docker compose down
/usr/bin/docker rm -f $(docker container ls -q --filter name=nextcloud*)
/usr/sbin/service docker restart
/usr/bin/docker compose up -d --pull always
/usr/bin/docker exec -it nextcloud-aio-mastercontainer sudo -u www-data php /var/www/docker-aio/php/src/Cron/StartAndUpdateContainers.php
echo $MESSAGE | /usr/bin/mutt -F "/root/.muttrc" -s "$SUBJECT" "$RECIPIENT"
echo $MESSAGE | /usr/bin/mutt -F "/root/.muttrc" -s "$SUBJECT" "$TEXTRECIPIENT"
else
echo "Website is down!"
echo $NON502MESSAGE | /usr/bin/mutt -F "/root/.muttrc" -s "$NON502SUBJECT" "$RECIPIENT"
fi
After testing it multiple times to make sure it works and sends email/text, I added it to cron to run every 15 minutes. I figure that it as a very acceptable RTO.
Also, make sure to chmod 755 Storage.Website.Errors.sh so that bash can execute the script.
*/15 * * * * /root/Storage.Website.Errors.sh
NOTE: I am running it as root and that is “bad”, however I know the risks and decided on it anyway. YMMV. You do you boo.
Finally, I would LOVE to fix the root cause, but I can live with this solution for a long time. Especially since it is now “automated” to fix itself for the standard 502 crash I am experiencing. I also learned a bit about scripting for docker and NC AIO during this process, so that is a bonus.