Is this the first time you’ve seen this error? (Yes / No): Yes
When did this problem seem to first start? Last week
Installation method: From scratch, Ubuntu VM
Are you using Cloudflare, mod_security, or similar? No
Summary of the issue you are facing:
The Updater (both web and CLI) hangs at step 3, the backup. I can restart the process by deleting the step file, but that doesn’t solve it. Calling the updater with -vvv does not produce any log output for the backup step.
This worked flawlessly for this exact instance for years, I did not change anything on the machine since the last nextcloud update.
Steps to replicate it (hint: details matter!):
Doesn’t apply - maybe it’s an issue with 30.0.1’s updater…
Log entries
Nothing in nextcloud.log, no log output from the CLI updater either. The web updater times out as it used to (I have always used the CLI updater because of this)
The updater.log shows that creating the backup at least started, and checking the backup directory I can see that indeed, some part of the backup creation worked:
It may not be hanging, but simply taking a long time.
How large is your active installation folder? i.e. /var/www/html or whatever
Alternatively, is your datadirectory located within your Nextcloud installation folder (probably not since I see /mnt/data in your output, unless you have Nextcloud installed in /mnt which I guess is possible).
Try running the Updater with --no-backup to skip it.
You are right, my data directory is not in the installation folder but at /mnt/data, an NFS mount of a few hundred GB.
You may want to use the updatedirectory config value to keep the Updater backups / work files from having to move so many files around over NFS. It will likely speed updates up tremendously.
As something obviously wasn’t right I was careful and didn’t dare to try it without backup in case something breaks. Would you still recommend it?
I always suggest keeping regular backups. The “backups”, however, that the built-in Updater creates are for it’s own internal purposes are neither complete (i.e. do not cover your database nor files) nor intended for administrative restores. Those are the “backups” impacted by the --no-backup option.
For real backups, I suggest following the instructions in the Maintenance: Backups chapter of the official Admin Manual.
EDIT: of the 4.3GB in the apps folder, 2.8 GB are only from the “recognize” app…
FYI: The recognize did (does?) uses a symbolic link within its installation folder. There is a known bug in the Updater that results in multiple copies of a large file that is part of the recognize app (that has a couple symlinks to it) during the Updater’s backup stage. The impact is extra space consumed by the Updater’s internal backup (and a lengthier backup). It’ll hopefully get addressed soon.
I changed the update directory to the local disk and re-run the process, didn’t help. What I found out though, which is surely part of the problem: The updater seems to cause a reboot of the whole machine! Here’s the excerpt of the journalctl output:
Dec 09 17:38:45 nextcloud sudo[1463]: root : TTY=pts/1 ; PWD=/root ; USER=www-data ; COMMAND=/usr/bin/php /var/www/nextcloud/updater/updater.phar
Dec 09 17:38:45 nextcloud sudo[1463]: pam_unix(sudo:session): session opened for user www-data by root(uid=0)
Dec 09 17:39:01 nextcloud CRON[1474]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 09 17:39:01 nextcloud CRON[1475]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Dec 09 17:39:01 nextcloud CRON[1474]: pam_unix(cron:session): session closed for user root
Dec 09 17:39:13 nextcloud systemd[1]: Starting Clean php session files...
-- Subject: A start job for unit phpsessionclean.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit phpsessionclean.service has begun execution.
--
-- The job identifier is 947.
Dec 09 17:39:13 nextcloud sessionclean[1500]: PHP Warning: Failed to set memory limit to 1024 bytes (Current memory usage is 2097152 bytes) in Unknown on line 0
Dec 09 17:39:13 nextcloud sessionclean[1500]: Cannot load Zend OPcache - it was already loaded
Dec 09 17:39:14 nextcloud systemd[1]: phpsessionclean.service: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit phpsessionclean.service has successfully entered the 'dead' state.
Dec 09 17:39:14 nextcloud systemd[1]: Finished Clean php session files.
-- Subject: A start job for unit phpsessionclean.service has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit phpsessionclean.service has finished successfully.
--
-- The job identifier is 947.
-- Reboot --
Dec 09 17:39:37 nextcloud kernel: Linux version 5.4.0-200-generic (buildd@lcy02-amd64-023) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)) #220-Ubuntu SMP Fri Sep 27 13:19:16 UTC 2024 (Ubuntu 5.4.0-200.220-gen>
Dec 09 17:39:37 nextcloud kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-5.4.0-200-generic root=UUID=60eb452d-89c1-46c2-bd9d-8cc4396be8c8 ro maybe-ubiquity
Dec 09 17:39:37 nextcloud kernel: KERNEL supported cpus:
Dec 09 17:39:37 nextcloud kernel: Intel GenuineIntel
Dec 09 17:39:37 nextcloud kernel: AMD AuthenticAMD
Dec 09 17:39:37 nextcloud kernel: Hygon HygonGenuine
Dec 09 17:39:37 nextcloud kernel: Centaur CentaurHauls
Dec 09 17:39:37 nextcloud kernel: zhaoxin Shanghai
Dec 09 17:39:37 nextcloud kernel: BIOS-provided physical RAM map:
I’m not sure what’s happening as there’s no major errors, panics or such, but it looks bad.
I gave it another try, and after setting the memory limit to a higher value it worked. I’m not certain this is the actual fix for the problem, but at least it’s working again for me
