I am using this script on my test nextcloud instance. I hope others might get good use for it. In production I initiates manually whenever my daily verification of test instance is as desired. So this script exists on both test and production instance.
To clarify a few important things before just implementing this script as is:
I am running scheduled ZFS snapshotting each day with an automatic retention of 14 days, which is the ONLY reason I use the --no-backup parameter. However if I did not use the no-backup, then also because of ZFS, the backup procedure would be okay fast, but the cleanup of temporary files (the backup created by NC updater) would take forever, because delete operations of large bulks of data, is a pain on transactional filesystems.
#! /bin/bash
rm tmp.txt
SQLFILE="/path/to/dumps/nc-$(date +"%Y-%m-%d_%M-%H")"
mysqldump --opt --user='NCUSER' --password='NCUSERPASS' 'NCDATABASE' > "$SQLFILE.sql"
gzip -c "$SQLFILE.sql" > "$SQLFILE.gz"
#sudo -u www-data php8.1 /var/www/nextcloud/occ maintenance:mode --on
#gzip -r /var/www/nextcloud/data/* "ncbackup-$(date +"%Y-%m-%d_%M-%H").gz"
#If any JSON returned from status url, server is running and returns no internal error
#Add appcode and basic auth to CURL and recieve proper status to use for more advanced stuff.
curl https://cloud.yourdomain.ltd/ocs/v2.php/apps/serverinfo/api/v1/info?format=json > tmp.txt
if grep -q '{"ocs":{"meta":{"status":"failure","statuscode":401,"message":null},"data":{"message":"Unauthorized"}}}' "tmp.txt" ; then
rm tmp.txt
apt-get update -y
apt-get upgrade -y
PHPCOMMAND="sudo -u www-data php8.1" #Keep PHP version up to date when you migrates to newer PHP version!!
commands[0]="/var/www/nextcloud/updater/updater.phar --no-backup --no-interaction"
commands[1]="/var/www/nextcloud/occ db:add-missing-indices"
commands[2]="/var/www/nextcloud/occ maintenance:mode --off"
commands[3]="/var/www/nextcloud/occ maintenance:data-fingerprint"
commands[4]="/var/www/nextcloud/occ app:update --all"
for i in "${commands[@]}"; do eval $PHPCOMMAND $i; done
systemctl restart php8.1-fpm #Keep PHP version up to date when you migrates to newer PHP version!!
systemctl restart apache2
else
rm tmp.txt
echo 'server not running' ;
fi
This is only if you restore from a backup (with files that might be older than the files present on the client to trigger a redownload of this older version).
Like this, you would also apply major upgrades once they are online, isn’t it? I wouldn’t do that automatically, there is just a too high risk to need to clean up things or apps that are not available yet, and then it is deployed when you don’t have the time to handle all this.
There is a feature request to avoid major upgrades.
I usually don’t comment on scripts because I see them just as a kind of snapshot that can be further optimized. But since this was filed in the How to section and has not received any further optimization since, I would like to make an exception here:
One should create a variable for this, so that one don’t have to go through all scripts to search for hardcoded php versions after an update.
The PHP_VERSION can also be passed as an argument to the script.
I prefer in such cases a mechanism that detects which php version the web server uses to connect to php (fpm socket), since the PHP_VERSION in this script is less important for the php call itself but for restarting the right fpm service.
Is there a reason, why the date is formed like this with the minutes before the hour?
It would be more common and easier to sort if the hour (here %H, i.e. in 24 hour format) came BEFORE the minutes.
In addition, one can create a timestamp once the script is called, which then can be used again for all steps (there is one more commented out) so that one can see better how they belong to each other.
The --opt option is enabled by default. Therefore I miss the settings for host and utf8mb4
What was the idea behind piping it like that? This way there remain two files, one uncompressed “$SQLFILE.sql” and one compressed “$SQLFILE.gz”. In my understanding, that contradicts the purpose of compressing (saving space).
gzip (without “-c”) produces a “$SQLFILE.sql.gz” or was the aim to duplicate the file?
It is bad practice to use generic names for temporary files in a script. That can open up a security hole and should be avoided if possible. It must be ensured that they are in no way shared with other scripts and, if possible, are stored in the /tmp directory.
Use:
mytempfile=$(mktemp)
instead
That can be done by simply reading the response code. No need of a temp file at all.
The use of eval in this context is considered unsafe and should be avoided.
Without going further into the sense of the individual steps, purely from the point of view of bash syntax, I would design the script more like this:
#!/bin/bash
# Restrictions:
# Only on Debian like Operating Systems because of the use of 'apt-get'
# Only for MySQL/MariaDB
# R U ROOT?
(( $(id -u) )) && { echo "You must be root"; exit 1; }
declare PHP_VERSION="${1:-8.1}" # Keep PHP version up to date when you migrates to newer PHP version!!
declare WEBSERVER='apache2' # Change this to nginx or whatever webserver is runing
declare NEXTCLOUD_USER='www-data'
declare NEXTCLOUD_DIR='/var/www/nextcloud'
declare SERVER_URL="https://cloud.bar.baz"
declare DB_HOST='dbhost' # mostly 'localhost' (from config/config.php)
declare DB_USER='dbuser' # from config/config.php
declare DB_PASS='dbpassword' # from config/config.php
declare DB_NAME='dbname' # mostly 'nextcloud' (from config/config.php)
declare DB_UTF8MB4='mysql.utf8mb4' # 'true' or keep empty for 'false' (from config/config.php)
declare TIMESTAMP="$(date +"%Y-%m-%d_%H%M")" # yyyy-mm-dd_hh-mm
declare DB_DUMPFILE="/path/to/dumps/nc-$TIMESTAMP" # Point this to the directory where you store your dumps.
declare -i i
# declare and populate the array with the commands to be executed by phpcommand
declare -a commands=(
"updater/updater.phar --no-backup --no-interaction"
"occ db:add-missing-indices"
"occ maintenance:mode --off"
# "occ maintenance:data-fingerprint" # see https://help.nextcloud.com/t/cron-job-to-maintain-nextcloud/152833/2
"occ app:update --all"
)
# 'phpcommand' as a function is much more stable and robust than when it is composed as a variable and then executed with eval
phpcommand(){
sudo -u $NEXTCLOUD_USER /usr/bin/php$PHP_VERSION -f $*
}
# dcs=Default Character Set
declare dcs=''
${DB_UTF8MB4:-false} && dcs='--default-character-set=utf8mb4'
mysqldump "$dcs" --host="$DB_HOST" --user="$DB_USER" --password="$DB_PASS" "$DB_NAME" > "$DB_DUMPFILE.sql"
# This will produce a file "$DB_DUMPFILE.sql.gz":
gzip "$DB_DUMPFILE.sql"
#phpcommand $NEXTCLOUD_DIR/occ maintenance:mode --on
#gzip -r $NEXTCLOUD_DIR/data/* "ncbackup-$TIMESTAMP.sql.gz"
if (( $(curl -sIo /dev/null -w %{http_code} $SERVER_URL/status.php) == 200 )); then
apt-get update -y && apt-get upgrade -y
for i in "${!commands[@]}"; do
phpcommand $NEXTCLOUD_DIR/${commands[i]}
done
systemctl restart $WEBSERVER php$PHP_VERSION-fpm
else
echo 'server not running'
fi
exit 0
Now you can call it like this:
sudo /path/to/scriptname.sh 8.2
that would run it with php8.2 version, even though the default (fallback php-version) is 8.1
Disclaimer:
I just wrote this lines of code as it stands here from my head, it certainly still contains some inaccuracies or errors. So be warned, it is not intended to be used as is but only to show how one or the other can be made a little more robust and easier to maintain.
Regarding the content or goal of the script, I can only advise against updating the server using a cron job.
This is not so sensitive for test instances, but why one has test instances if not to “TEST” exactly every step and not to let it run at some point at night automated by a cronjob?
In my opinion the script is simply too thin to update a productive server. It is missing some very important parts, such as the undone mechanism - automatic or not, which ensures that if the update gets stuck somewhere or fails in some other way, everything returns to the previous state to guarantee a running system at all times.
That’s the way how I’ve always set it up for my needs.
Simply rattling off a few commands one after the other is nice, but that doesn’t mean you’ve built a solution, and certainly not one that can be run unattended in the cron of a third-party user.
So my urgent advice to everyone who is looking for script parts: Use them wherever they serve you, let them inspire you, expand them according to your needs - but ONLY if you understand exactly what is going on in the code. So you should be able to recognise vulnarable or malwritten codeparts. Don’t expect miracles and realise, you are on your own, when things go wrong!
Don’t auto-update your Nextcloud instance, unless you made sure, it doesn’t auto-update to new major releases!
…and as @tflidd pointed out, maintenance:data-fingerprint is probably not something you need or should run on a regular basis. I can’t remember ever using it, and my current production instance is running since 2017.
But that doesn’t help at all in this context. Then you can just as easily use php (without specifying the version). That’s what this query does: it asks which version is linked via the generic link /usr/bin/php.
It may well be that the web server is using a socket from a completely different PHP version. And since the script - in addition to the php-cli calls for the occ and updater.phar processes, also restarts the fpm service which is not addressed via a link but has to be called directly with the version, that is why you need to detect which socket from which php version is connected to which web server.
Detecting that is not as mundane as simply using:
dpkg helper scripts can give you more information. List all installed php versions with:
phpquery -V
If that one shows you more than one version, then you must use other mechanisms to detect the used php Version.
That’s why it’s completely okay if the php version is defined in a variable if you don’t feel like writing 10 lines of code to detect it in all possible situations.
You can see exactly how that is done for all eventualities, including cases where the connection completely goes beyond the normal php, apache2 and or nginx configuration routines, in the php-updater script when calling
php-updater --list-installed
That lists exactly which sockets are listening, which modules are loaded and which socket is connected to which web server.
Here an example output:
==============================================================================
actual PHP related packages installed and managed by dpkg
X Package Version Status
------------------------------------------------------------------------------
SAPI libapache2-mod-php8.1 8.1.27-1+ubuntu22.04.1+deb.sury.org+1 im
\apache2handler /usr/lib/apache2/modules/libphp8.1.so <<INACTIVE
- newrelic-php5 10.17.0.7 ia
\included modules: newrelic-20151012 newrelic-20160303 newrelic-20170718
newrelic-20180731 newrelic-20190902 newrelic-20200930 newrelic-20210902
newrelic-20220829 newrelic-20230831
- newrelic-php5-common 10.17.0.7 im
SAPI php8.1-cli 8.1.27-1+ubuntu22.04.1+deb.sury.org+1 im
\php binary /usr/bin/php8.1 <<INACTIVE not linked
- php8.1-common 8.1.27-1+ubuntu22.04.1+deb.sury.org+1 ia
\included modules: calendar ctype exif ffi fileinfo ftp gettext iconv pdo
phar posix shmop sockets sysvmsg sysvsem sysvshm tokenizer
SAPI php8.1-fpm 8.1.27-1+ubuntu22.04.1+deb.sury.org+1 im
\listening on unix socket /run/php/php8.1-fpm.sock <<NOT CONNECTED
- php8.1-opcache 8.1.27-1+ubuntu22.04.1+deb.sury.org+1 ia
\included module: opcache
- php8.1-readline 8.1.27-1+ubuntu22.04.1+deb.sury.org+1 ia
\included module: readline
- php8.2-apcu 5.1.23-1+ubuntu22.04.1+deb.sury.org+1 im
\included module: apcu
- php8.2-bcmath 8.2.15-1+ubuntu22.04.1+deb.sury.org+1 im
\included module: bcmath
- php8.2-bz2 8.2.15-1+ubuntu22.04.1+deb.sury.org+1 im
\included module: bz2
SAPI php8.2-cli 8.2.15-1+ubuntu22.04.1+deb.sury.org+1 im
\php binary /usr/bin/php8.2 <ACTIVE> linked to generic /usr/bin/php
- php8.2-common 8.2.15-1+ubuntu22.04.1+deb.sury.org+1 im
\included modules: calendar ctype exif ffi fileinfo ftp gettext iconv pdo
phar posix shmop sockets sysvmsg sysvsem sysvshm tokenizer
- php8.2-curl 8.2.15-1+ubuntu22.04.1+deb.sury.org+1 im
\included module: curl
- php8.2-dev 8.2.15-1+ubuntu22.04.1+deb.sury.org+1 im
SAPI php8.2-fpm 8.2.15-1+ubuntu22.04.1+deb.sury.org+1 im
\listening on unix socket /run/php/php8.2-fpm.sock <CONNECTED> apache2
- php8.2-gd 8.2.15-1+ubuntu22.04.1+deb.sury.org+1 im
\included module: gd
- php8.2-gmp 8.2.15-1+ubuntu22.04.1+deb.sury.org+1 im
\included module: gmp
- php8.2-igbinary 3.2.15-1+ubuntu22.04.1+deb.sury.org+1 im
\included module: igbinary
- php8.2-imagick 3.7.0-4+ubuntu22.04.1+deb.sury.org+2 im
\included module: imagick
- php8.2-intl 8.2.15-1+ubuntu22.04.1+deb.sury.org+1 im
\included module: intl
- php8.2-mbstring 8.2.15-1+ubuntu22.04.1+deb.sury.org+1 im
\included module: mbstring
- php8.2-mysql 8.2.15-1+ubuntu22.04.1+deb.sury.org+1 im
\included modules: mysqli mysqlnd pdo_mysql
- php8.2-opcache 8.2.15-1+ubuntu22.04.1+deb.sury.org+1 im
\included module: opcache
SAPI php8.2-phpdbg 8.2.15-1+ubuntu22.04.1+deb.sury.org+1 im
\phpdbg binary /usr/bin/phpdbg8.2 <ACTIVE> linked to generic /usr/bin/phpdbg
- php8.2-readline 8.2.15-1+ubuntu22.04.1+deb.sury.org+1 im
\included module: readline
- php8.2-redis 6.0.2-1+ubuntu22.04.1+deb.sury.org+1 im
\included module: redis
- php8.2-xml 8.2.15-1+ubuntu22.04.1+deb.sury.org+1 im
\included modules: dom simplexml xml xmlreader xmlwriter xsl
- php8.2-zip 8.2.15-1+ubuntu22.04.1+deb.sury.org+1 im
\included module: zip
- php8.2-zmq 1.1.3-24+ubuntu22.04.1+deb.sury.org+2 im
\included module: zmq
- php-common 2:94+ubuntu22.04.1+deb.sury.org+2 ia
- php-getallheaders 3.0.3-2 ia
- php-guzzlehttp-psr7 1.8.3-1 im
- php-pear 1:1.10.12+submodules+notgz+20210212-1ubuntu3 im
- php-psr-http-message 1.0.1-2 ia
- pkg-php-tools 1.42build1 ia
------------------------------------------------------------------------------
actual PHP extension NOT installed and managed by dpkg
------------------------------------------------------------------------------
- php8.2
\module installed by admin: lua
------------------------------------------------------------------------------
Status: - First letter (Desired Action): - Second letter (Package Status):
u=Unknown - i=Install - h=Hold a=Automatic - m=Manual
------------------------------------------------------------------------------
You can see here, that there is an apache2handler php module for php8.1 which is not loaded and there are two listening fpm sockets but only one of them (8.2) is connected.
At the same time, this listing shows, that the /usr/bin/php is as well linked to the version 8.2 but that is not allways the case.
Thank you for a very good and inspiring optimized script example.
I will certainly look into this in my production NC.
I am not, and has never been, any particular good at shell scripts. This is the script I made that still today, is used as CRON in my test NC.
I ALWAYS runs production updates manually. The reason is that I tests how the new versions plays out in my auto updated test lab first.
The GZIP commands is because I got another script that transfers the zipped file out of the server to an off-location and I got a test script for testing the dump in a test database. So I need it in both .sql and in GZIP. It probably could be done smarter I admit, but it works for my needs.
As I used snapshot backup utilizing that I use openZFS, and the database server is on another machine entirely, then when I do a snapshot rollback, I need to run this to adapt the database fingerprints to current files (I could move this one to the rollback script) but as this is my test server script I am lazy.
It has never given me any issues though.
I really like your feedback. It has given me a lot of inspiration to work with to actually semi automatic update prod. Never as a cron, but at least to do it with a single command. Thank you.
You’re of course right, I hadn’t thought of that. However, since I only have a single version of PHP installed on my Nextcloud server, it would probably work for the things I would use it for (not for auto-update scripts), or phpquery -V which I didn’t know, or maybe grepping it from dpkg -I…
Of course it’s fine, but then I either would have to change the variable in the script everytime I upgrade PHP or pass the variable every time I run the script.
And my problem would be less the additional 10 lines of code as such. It’s more that I would not have thought of everything and even if I had, I would not have managed to code it in the same sophisticated way as you. If I did something similiar it would probably be more like 50 lines, and it still wouldn’t properly cover every situation.
That being said. I really appreciate your detailed examples and explanations. It’s indeed very inspiring.
