28.0.3: Nextcloud update to 28.0.5 failed in “Create backup” step

🚧 Installation
, ,
1

I am trying to upgrade from 28.0.3 to 28.0.5 (and further on to 29), but the command-line updater fails as follows:

# sudo -u wwwrun /usr/bin/php /data/ownCloud/updater/updater.phar --no-interaction
Nextcloud Updater - version: v28.0.2rc2-1-gba2e50f dirty

Current version is 28.0.3.

Update to Nextcloud 28.0.5 available. (channel: "stable")
Following file will be downloaded automatically: https://download.nextcloud.com/server/releases/nextcloud-28.0.5.zip
Open changelog ↗

Updater run in non-interactive mode.

Start update

Info: Pressing Ctrl-C will finish the currently running step and then stops the updater.

[✔] Check for expected files
[✔] Check for write permissions
[✘] Create backup failed
Could not copy "/data/ownCloud/composer.lock" to "/data/ownCloud/data/updater-oc06twnzrsh4/backups/nextcloud-28.0.3.2-1714549902/composer.lock"

Update failed. To resume or retry just execute the updater again.

Needless to say that “executing the updater again” does not help. Usually, problems like this are due to file permission issues. However, this seems not to be the case here.

The Nextcloud updater creates the backup directory hierarchy /data/updater-oc06twnzrsh4/backups/nextcloud-28.0.3.2-1714549902 successfully as well as the target file composer.lock within that directory. However, that file has a size of zero.

As the wwwrun user, I was able to copy /data/ownCloud/composer.lock to /data/ownCloud/data/updater-oc06twnzrsh4/backups/nextcloud-28.0.3.2-1714549902/composer.lock without problems and with the destination file having the same contents as the source file.

Here’s the relevant portion of the updater.log:

2024-05-01T07:51:42+0000 tPg8PQWvoI [info] end of checkForUpdate() Update to Nextcloud 28.0.5 available. (channel: "stable")<br /><span class="light">Following file will be downloaded automatically:</span> <code class="light">https://download.nextcloud.com/server/releases/nextcloud-28.0.5.zip</code><br /><a class="external_link" href="https://nextcloud.com/changelog/#28-0-5" target="_blank" rel="noreferrer noopener">Open changelog ↗</a>
2024-05-01T07:51:42+0000 tPg8PQWvoI [info] updater run in non-interactive mode
2024-05-01T07:51:42+0000 tPg8PQWvoI [info] updater started
2024-05-01T07:51:42+0000 tPg8PQWvoI [info] executeStep request for step "1"
2024-05-01T07:51:42+0000 tPg8PQWvoI [info] startStep("1")
2024-05-01T07:51:42+0000 tPg8PQWvoI [info] checkForExpectedFilesAndFolders()
2024-05-01T07:51:42+0000 tPg8PQWvoI [info] end of checkForExpectedFilesAndFolders()
2024-05-01T07:51:42+0000 tPg8PQWvoI [info] endStep("1")
2024-05-01T07:51:42+0000 tPg8PQWvoI [info] executeStep request for step "2"
2024-05-01T07:51:42+0000 tPg8PQWvoI [info] startStep("2")
2024-05-01T07:51:42+0000 tPg8PQWvoI [info] checkWritePermissions()
2024-05-01T07:51:42+0000 tPg8PQWvoI [info] end of checkWritePermissions()
2024-05-01T07:51:42+0000 tPg8PQWvoI [info] endStep("2")
2024-05-01T07:51:42+0000 tPg8PQWvoI [info] executeStep request for step "3"
2024-05-01T07:51:42+0000 tPg8PQWvoI [info] startStep("3")
2024-05-01T07:51:42+0000 tPg8PQWvoI [info] createBackup()
2024-05-01T07:51:42+0000 tPg8PQWvoI [info] backup folder location: /data/ownCloud/data/updater-oc06twnzrsh4/backups/nextcloud-28.0.3.2-1714549902/
2024-05-01T07:51:42+0000 tPg8PQWvoI [error] executeStep request failed with other exception
2024-05-01T07:51:42+0000 tPg8PQWvoI [error] Exception: Exception
Message: Could not copy "/data/ownCloud/composer.lock" to "/data/ownCloud/data/updater-oc06twnzrsh4/backups/nextcloud-28.0.3.2-1714549902/composer.lock"
Code:0
Trace:
#0 phar:///data/ownCloud/updater/updater.phar/lib/UpdateCommand.php(374): NC\Updater\Updater->createBackup()
#1 phar:///data/ownCloud/updater/updater.phar/lib/UpdateCommand.php(237): NC\Updater\UpdateCommand->executeStep()
#2 phar:///data/ownCloud/updater/updater.phar/vendor/symfony/console/Command/Command.php(255): NC\Updater\UpdateCommand->execute()
#3 phar:///data/ownCloud/updater/updater.phar/vendor/symfony/console/Application.php(1021): Symfony\Component\Console\Command\Command->run()
#4 phar:///data/ownCloud/updater/updater.phar/vendor/symfony/console/Application.php(275): Symfony\Component\Console\Application->doRunCommand()
#5 phar:///data/ownCloud/updater/updater.phar/vendor/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun()
#6 phar:///data/ownCloud/updater/updater.phar/updater.php(10): Symfony\Component\Console\Application->run()
#7 /data/ownCloud/updater/updater.phar(14): require('...')
#8 {main}
File:phar:///data/ownCloud/updater/updater.phar/lib/Updater.php
Line:456

2024-05-01T07:51:42+0000 tPg8PQWvoI [info] rollbackChanges("3")
2024-05-01T07:51:42+0000 tPg8PQWvoI [info] unlink .step
2024-05-01T07:51:42+0000 tPg8PQWvoI [info] end of  rollbackChanges()
2

Not sure if this helps, but just in case: I had some issues at this stage, but not the exact same one as you. In my case, I had some files/directories under the …/updater-…/backups/* with the wrong ownership preventing copying of files. I don’t believe this was a Nextcloud issue - I have just moved my data directory to an NFS share attached to the same location and think this was related to this. Setting the correct owner:group with chown down through this directory tree resolved the problem for me.

3

Unfortunately not – all file permissions and ownerships are find. I even deleted the whole updater-oc06twnzrsh4 directory to make sure. The updater then (re-)created the updater-oc06twnzrsh4/backups/nextcloud-28.0.3.2-1714549902/ directory hierarchy as expected – before giving up.

4

Is there anything unusual about /data/ownCloud/data? i.e.

  • is it a symbolic link elsewhere?
  • is it a distinct mount mount at the OS level from /data/ownCloud?

While I’m not sure what’s going on you have two things that may help you workaround it:

  • there is a parameter that can be used with the Updater (updater.phar) in command line mode to disable backups: --no-backup
  • you can try using a different directory entirely (outside of your installation and data folders) for the backup/update work files by using the updatedirectory parameter, but whether that’ll actually help you will depend on what the underlying cause is

It’s odd that you’re getting an exception without an exception explanation message in that area of code. :thinking:

5

There’s nothing special about the /data/ownCloud directory whatsoever. Meanwhile, we did a manual update to Nextcloud 29, but will again try the CLI updater next time and, if still needed, try your suggestions.

6

The code that is failing is PHP’s standard copy(). It’s returning false in your environment for some reason (indicating the failure) and then also creating an empty file (!). We do further checks for readability/writability (which are passing based on your logs). So all we know is what PHP’s reporting: copy() failed for that file.

It’s very interesting the file is getting created and you’re getting a file size of zero.

Unfortunately though that indicates this is something within PHP itself.

  • What OS/distribution is this?
  • What PHP version?
  • What’s the underlying filesystem type for /data/ownCloud? and/or /data/ownCloud/data?
  • Any NFS, CIFS/SMB, WSL/WSL2, VirtualBox VM shared folder, or anything like that involved?
7

Linux 6.8.7-1-default x86_64
openSUSE Tumbleweed 20240428

PHP 8.3.6

ZFS (both directories are on the same filesystem)

No, none

8

There have been some copy() related bugs in PHP, as recently as 8.3.x.

I’d be curious what the following returns when ran from the command line in your environment:

php -r 'var_dump(copy("/data/ownCloud/composer.lock", "/data/ownCloud/data/updater-oc06twnzrsh4/backups/nextcloud-28.0.3.2-1714549902/composer.lock"));'

(it should manually copy the file).

If that fails in the same was as when used via the Updater, then it’s presumably either an environment issue or PHP issue.

If that fails, using strace on it might turn up something useful:

strace php -r 'var_dump(copy("/data/ownCloud/composer.lock", "/data/ownCloud/data/updater-oc06twnzrsh4/backups/nextcloud-28.0.3.2-1714549902/composer.lock"));'

If it does not fail in your environment that would be interesting to know to.

9

TL;DR: Alright, looks good overall!

In detail:

Manually copying the file via the PHP call fails, so you have indeed encircled the problem! The command creates a zero-length file and writes bool(false) to the console.

I tried the PHP copy command with the other target filesystem types I have at hand on that machine (EXT4, XFS, VFAT), and they all work fine. The copy fails only if the target filesystem is ZFS. (BTW, I also tried the copy function with another ZFS filesystem I have available on that machine.)

And here’s the relevant portion from the strace output:

newfstatat(AT_FDCWD, "/data/ownCloud/composer.lock", {st_mode=S_IFREG|0644, st_size=3140, ...}, 0) = 0
newfstatat(AT_FDCWD, "/data/ownCloud/data/updater-oc06twnzrsh4/backups/nextcloud-28.0.3.2-1714549902/composer.lock", {st_mode=S_IFREG|0644, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/data/ownCloud/composer.lock", {st_mode=S_IFREG|0644, st_size=3140, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/data/ownCloud", {st_mode=S_IFDIR|0755, st_size=32, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/data", {st_mode=S_IFDIR|0755, st_size=13, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "/data/ownCloud/composer.lock", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=3140, ...}) = 0
lseek(3, 0, SEEK_CUR)                   = 0
newfstatat(AT_FDCWD, "/data/ownCloud/data/updater-oc06twnzrsh4/backups/nextcloud-28.0.3.2-1714549902/composer.lock", {st_mode=S_IFREG|0644, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/data/ownCloud/data/updater-oc06twnzrsh4/backups/nextcloud-28.0.3.2-1714549902", {st_mode=S_IFDIR|0750, st_size=3, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/data/ownCloud/data/updater-oc06twnzrsh4/backups", {st_mode=S_IFDIR|0750, st_size=5, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/data/ownCloud/data/updater-oc06twnzrsh4", {st_mode=S_IFDIR|0755, st_size=3, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/data/ownCloud/data", {st_mode=S_IFDIR|0750, st_size=264, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "/data/ownCloud/data/updater-oc06twnzrsh4/backups/nextcloud-28.0.3.2-1714549902/composer.lock", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
lseek(4, 0, SEEK_CUR)                   = 0
lseek(3, 0, SEEK_SET)                   = 0
lseek(4, 0, SEEK_SET)                   = 0
copy_file_range(3, NULL, 4, NULL, 9223372036854775807, 0) = -1 EOPNOTSUPP (Operation not supported)
close(3)                                = 0
close(4)                                = 0
write(1, "bool(false)\n", 12bool(false)
)           = 12

Investigating further, it turned out that the generic_copy_file_range() function has been removed from the Linux kernel. The good news is that ZFS 2.2.4 already has a fix for that, see [2.2] Linux 6.8 compat: use splice_copy_file_range() for fallback by robn · Pull Request #16017 · openzfs/zfs · GitHub.

Even better: ZFS 2.2.4 is already available for Tumbleweed. I just ran an update, and probably everything will work fine after the next reboot – which unfortunately I cannot do right now.

However, subject to that reboot, I think we can close this issue now.

1 Like
10

Just for the sake of completeness: After rebooting the machine, everything works as expected (ZFS 2.2.4, Linux kernel 6.8.8).

1 Like