Update from 20.0.9.1 to 21.0.1 fails and leaves me clueless

Nextcloud version: 20.0.9
Operating system and version: jail in TrueNAS-12.0-U3.1 (FreeBSD 12.2-RELEASE-p6 amd64)
Apache version: apache24-2.4.46_2
PHP version (eg, 7.4): php74-7.4.16

The settings/admin/overview web page said a new version was available: Nextcloud 21.0.1, so I diligently took snapshots of the ZFS datasets that hold the Nextcloud jail, data folder and database and finally went back to the web page and clicked the ‘Open updater’ button.


2. Nextcloud will be updated to version 21.0.1

So I ran php occ upgrade as user www and got this:

Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Setting log level to debug
Updating database schema
Updated database
Updating <lookup_server_connector> ...
An unhandled exception has been thrown:
TypeError: in_array() expects parameter 2 to be array, string given in /usr/local/www/apache24/data/nextcloud/lib/private/legacy/OC_App.php:978
Stack trace:
#0 /usr/local/www/apache24/data/nextcloud/lib/private/legacy/OC_App.php(978): in_array('lookup_server_c...', 'calendar', true)
#1 /usr/local/www/apache24/data/nextcloud/lib/private/Updater.php(340): OC_App::updateApp('lookup_server_c...')
#2 /usr/local/www/apache24/data/nextcloud/lib/private/Updater.php(260): OC\Updater->doAppUpgrade()
#3 /usr/local/www/apache24/data/nextcloud/lib/private/Updater.php(132): OC\Updater->doUpgrade('21.0.1.1', '20.0.9.1')
#4 /usr/local/www/apache24/data/nextcloud/core/Command/Upgrade.php(255): OC\Updater->upgrade()
#5 /usr/local/www/apache24/data/nextcloud/apps/bookmarks/vendor/symfony/console/Command/Command.php(255): OC\Core\Command\Upgrade->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#6 /usr/local/www/apache24/data/nextcloud/apps/bookmarks/vendor/symfony/console/Application.php(1009): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#7 /usr/local/www/apache24/data/nextcloud/apps/bookmarks/vendor/symfony/console/Application.php(273): Symfony\Component\Console\Application->doRunCommand(Object(OC\Core\Command\Upgrade), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#8 /usr/local/www/apache24/data/nextcloud/apps/bookmarks/vendor/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#9 /usr/local/www/apache24/data/nextcloud/lib/private/Console/Application.php(215): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#10 /usr/local/www/apache24/data/nextcloud/console.php(100): OC\Console\Application->run()
#11 /usr/local/www/apache24/data/nextcloud/occ(11): require_once('/usr/local/www/...')

My Nextcloud log is a headache to look at and at a first glance it doesn’t provide useful clues, but I can provide it if you want it.

The output of my config.php file:

<?php
$CONFIG = array (
  'passwordsalt' => 'redacted',
  'secret' => 'redacted',
  'trusted_domains' =>
  array (
    0 => 'nextcloud.redacted.com',
    1 => 'nextcloud.lan',
    2 => 'localhost',
  ),
  'trusted_proxies' => '10.0.0.20',
  'htaccess.RewriteBase' => '/',
  'datadirectory' => '/mnt/files',
  'version' => '20.0.9.1',
  'dbtype' => 'mysql',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost:/tmp/mysql.sock',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'nextcloud',
  'dbpassword' => 'redacted',
  'mysql.utf8mb4' => true,
  'installed' => true,
  'instanceid' => 'redacted',
  'log_type' => 'file',
  'logfile' => '/var/log/nextcloud/nextcloud.log',
  'loglevel' => 0,
  'log_rotate_size' => 104857600,
  'logtimezone' => 'Europe/Paris',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
    'host' => '/var/run/redis/redis.sock',
    'port' => 0,
  ),
  'maintenance' => false,
  'theme' => '',
  'updater.release.channel' => 'stable',
  'app_install_overwrite' => 'calendar',
  'mail_from_address' => 'redacted',
  'mail_smtpmode' => 'smtp',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_domain' => 'redacted',
  'mail_smtpsecure' => 'tls',
  'mail_smtpauth' => 1,
  'mail_smtphost' => 'smtp.redacted',
  'mail_smtpname' => 'redacted',
  'mail_smtppassword' => 'redacted',
  'mail_sendmailmode' => 'pipe',
  'mail_smtpport' => '587',
  'forwarded_for_headers' =>
  array (
    0 => 'HTTP_X_FORWARDED',
    1 => 'HTTP_FORWARDED_FOR',
    2 => 'HTTP_X_FORWARDED_FOR',
    3 => 'HTTP_X_REAL_IP',
  ),
  'overwritehost' => 'nextcloud.redacted.com',
  'overwrite.cli.url' => 'https://nextcloud.redacted.com/',
  'overwriteprotocol' => 'https',
  'overwritewebroot' => '/',
  'updater.secret' => 'redacted',
);

The output of my Apache error_log (20:45 is when I stopped the Nextcloud jail to take a snapshot of it):

2021-05-12 09:08:11.643890 [error] [pid 3503] mod_access_compat.c(350): [client 10.0.0.20:56444] AH01797: client denied by server configuration: /usr/local/www/apache24/data/nextcloud/config
2021-05-13 08:50:26.644054 [error] [pid 3503] mod_access_compat.c(350): [client 10.0.0.20:53862] AH01797: client denied by server configuration: /usr/local/www/apache24/data/nextcloud/config
2021-05-14 08:52:51.548865 [error] [pid 3502] mod_access_compat.c(350): [client 10.0.0.20:24620] AH01797: client denied by server configuration: /usr/local/www/apache24/data/nextcloud/config
2021-05-14 20:45:20.875830 [warn] [pid 3498] mpm_unix.c(145): [client AH00045: child process 18176 still did not exit, sending a SIGTERM
2021-05-14 20:45:22.924028 [warn] [pid 3498] mpm_unix.c(145): [client AH00045: child process 18176 still did not exit, sending a SIGTERM
2021-05-14 20:45:24.928254 [warn] [pid 3498] mpm_unix.c(145): [client AH00045: child process 18176 still did not exit, sending a SIGTERM
2021-05-14 20:45:26.936315 [error] [pid 3498] mpm_unix.c(157): [client AH00046: child process 18176 still did not exit, sending a SIGKILL
2021-05-14 20:45:27.944336 [notice] [pid 3498] event.c(3109): [client AH00491: caught SIGTERM, shutting down
2021-05-14 20:45:44.637909 [notice] [pid 40159] event.c(3082): [client AH00489: Apache/2.4.46 (FreeBSD) OpenSSL/1.1.1h-freebsd configured -- resuming normal operations
2021-05-14 20:45:44.638122 [notice] [pid 40159] log.c(1563): [client AH00094: Command line: '/usr/local/sbin/httpd -D NOHTTPACCEPT'

If I now run php occ status, I get:

Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
  - installed: true
  - version: 21.0.1.1
  - versionstring: 21.0.1
  - edition:

Now I’m out of ideas, what do you think I could try next?
For the time being, I’ll be rolling the datasets back to the snapshots’ states!
Thanks.

You can disable all Nextcloud apps by the occ command by hand and try to update again. I had a similar issue a couple of years ago and it worked for me.

It is best to install as few apps as possible!

Thanks for the help!
Do you know if there’s a way I can disable all of them at once?
By looking at php occ help app:disable, it looks like I can only disable one at a time…

You can use the awk command and the xargs command to do it at once

Ok, I’ve disabled all apps (though some of them such as workflowengine or viewer can’t be disabled), but now when I go to the settings/admin/overview web page, the Update button is gone! Is it because of some app I’ve disabled?


Can I run the update via the command line instead?
php occ upgrade returns Nextcloud is already latest version… why is that?
I’ve run php occ maintenance:repair just in case, but there was no particular issue to be resolved.

Then you have the latest version installed. Maybe they have withdrawn 21.0.1?

As it turns out, after running php occ app:enable updatenotification, the Update button reappeared!
So I ran the update but it ended, as before, with the unhelpful message “An error occurred.”
Back to square one.
Any more suggestions, please??

tail -f /your/next/cloud/data/folder/updater.log

Thank you. I didn’t know that such a log even existed!
I pasted it the relevant part here on PasteBin.
I gave it a look but couldn’t find anything useful in the last lines… what do you think?

Everything looks fine:

endStep("12")

It does! It looks fine in the log, but in reality I still get that “An error occurred” message and Nextcloud stays in maintenance mode.
I disabled maintenance mode via command line, reloaded the web page and got the web updater again:


Unfortunately, as soon as I clicked the Start update button, I got the “An error occurred” message again.

Here’s another failed attempt, this time at the command line:

21:26:40 www@nextcloud /usr/local/www/apache24/data/nextcloud % php occ upgrade
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Setting log level to debug
Updating database schema
Updated database
Updating <lookup_server_connector> ...
An unhandled exception has been thrown:
TypeError: in_array() expects parameter 2 to be array, string given in /usr/local/www/apache24/data/nextcloud/lib/private/legacy/OC_App.php:978
Stack trace:
#0 /usr/local/www/apache24/data/nextcloud/lib/private/legacy/OC_App.php(978): in_array('lookup_server_c...', 'calendar', true)
#1 /usr/local/www/apache24/data/nextcloud/lib/private/Updater.php(340): OC_App::updateApp('lookup_server_c...')
#2 /usr/local/www/apache24/data/nextcloud/lib/private/Updater.php(260): OC\Updater->doAppUpgrade()
#3 /usr/local/www/apache24/data/nextcloud/lib/private/Updater.php(132): OC\Updater->doUpgrade('21.0.1.1', '20.0.9.1')
#4 /usr/local/www/apache24/data/nextcloud/core/Command/Upgrade.php(255): OC\Updater->upgrade()
#5 /usr/local/www/apache24/data/nextcloud/3rdparty/symfony/console/Command/Command.php(255): OC\Core\Command\Upgrade->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#6 /usr/local/www/apache24/data/nextcloud/3rdparty/symfony/console/Application.php(1009): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#7 /usr/local/www/apache24/data/nextcloud/3rdparty/symfony/console/Application.php(273): Symfony\Component\Console\Application->doRunCommand(Object(OC\Core\Command\Upgrade), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#8 /usr/local/www/apache24/data/nextcloud/3rdparty/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#9 /usr/local/www/apache24/data/nextcloud/lib/private/Console/Application.php(215): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#10 /usr/local/www/apache24/data/nextcloud/console.php(100): OC\Console\Application->run()
#11 /usr/local/www/apache24/data/nextcloud/occ(11): require_once('/usr/local/www/...')
#12 {main}

No new lines were added to updater.log as a result of the above command.
This TypeError: in_array() expects parameter 2 to be array, string given in /usr/local/www/apache24/data/nextcloud/lib/private/legacy/OC_App.php:978 is the only vaguely actionable hint I can find, but even then I wouldn’t know what to do with it… what do you think?

What is the current status? Is the issue solved?

/usr/local/www/apache24/data/nextcloud/lib/private/legacy/OC_App.php:978

In my case 20.0.9 it contains the following code:

 972                 if (file_exists($appPath . '/appinfo/database.xml')) {
 973                         OC_DB::updateDbFromStructure($appPath . '/appinfo/database.xml');
 974                 } else {
 975                         $ms = new MigrationService($appId, \OC::$server->getDatabaseConnection());
 976                         $ms->migrate();
 977                 }
 978 
 979                 self::executeRepairSteps($appId, $appData['repair-steps']['post-migration']);
 980           

It could either fail because of the lines above, or the one following. What does your code have?

Thank you for caring @radoeka .
I wasn’t able to solve the issue.
Then, one day, update to version 20.0.10 became available, so I went ahead and updated, and the update was successful!
After that, version 21.0.2 became available, but updating to it fails. I guess my installation somehow has a compatibility issue with all versions 21.x…
Another thing, which may or may not be related, this messages showed up on my settings > admin > overview page:

Some files have not passed the integrity check. Further information on how to resolve this issue can be found in the [documentation]. ([List of invalid files…]/[Rescan…])

The invalid file is a php_errors.log file in my nextcloud folder. Its content:

[22-May-2021 15:55:10 UTC] PHP Fatal error:  Uncaught PDOException: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away in /usr/local/www/apache24/data/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php:127
Stack trace:
#0 /usr/local/www/apache24/data/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php(127): PDOStatement->execute(NULL)
#1 /usr/local/www/apache24/data/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(1062): Doctrine\DBAL\Driver\PDOStatement->execute()
#2 /usr/local/www/apache24/data/nextcloud/lib/private/DB/Connection.php(222): Doctrine\DBAL\Connection->executeUpdate('UPDATE `oc_file...', Array, Array)
#3 /usr/local/www/apache24/data/nextcloud/lib/private/Lock/DBLockingProvider.php(227): OC\DB\Connection->executeUpdate('UPDATE `oc_file...', Array)
#4 /usr/local/www/apache24/data/nextcloud/lib/private/Lock/AbstractLockingProvider.php(126): OC\Lock\DBLockingProvider->releaseLock('files/f14e2c68d...', 2)
#5 /usr/local/www/apache24/data/nextcloud/lib/private/Lock/DBLo in /usr/local/www/apache24/data/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/AbstractMySQLDriver.php on line 106
[27-May-2021 07:41:12 UTC] PHP Fatal error:  Uncaught PDOException: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away in /usr/local/www/apache24/data/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php:127
Stack trace:
#0 /usr/local/www/apache24/data/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php(127): PDOStatement->execute(NULL)
#1 /usr/local/www/apache24/data/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(1062): Doctrine\DBAL\Driver\PDOStatement->execute()
#2 /usr/local/www/apache24/data/nextcloud/lib/private/DB/Connection.php(222): Doctrine\DBAL\Connection->executeUpdate('UPDATE `oc_file...', Array, Array)
#3 /usr/local/www/apache24/data/nextcloud/lib/private/Lock/DBLockingProvider.php(227): OC\DB\Connection->executeUpdate('UPDATE `oc_file...', Array)
#4 /usr/local/www/apache24/data/nextcloud/lib/private/Lock/AbstractLockingProvider.php(126): OC\Lock\DBLockingProvider->releaseLock('files/f14e2c68d...', 2)
#5 /usr/local/www/apache24/data/nextcloud/lib/private/Lock/DBLo in /usr/local/www/apache24/data/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/AbstractMySQLDriver.php on line 106
[29-May-2021 14:27:10 UTC] PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 56730792 bytes) in /usr/local/www/apache24/data/nextcloud/3rdparty/sabre/http/lib/Sapi.php on line 82

php occ maintenance:repair hasn’t deleted the file, so I deleted it manually and now my settings > admin > overview page says All checks passed.
Apart from this php_errors.log file and the inability to update to version 21.x, everything works just fine… I’m starting to think my only way upwards is installing nextcloud version 21 in a new jail…

The problem seems to be with:

Updating <lookup_server_connector> …

At least that is what is shown in the message that you posted.

I’ve no idea what that is doing. I think that it is part of Nextcloud core. I’ve installed as well, but can’t remember that I installed it. (link to the code: server/apps/lookup_server_connector at master · nextcloud/server · GitHub)

Can you disable this app?
Can you remove it? And re-install it after the upgrade?

Just guessing, what could help…

Perhaps contact some of the people mentioned in this file: server/RetryJob.php at master · nextcloud/server · GitHub perhaps they’ve some tips and hints?

Thank you for going out of your way to help out!
Unfortunately, I’m not able to either disable or remove the lookup_server_connector app.


I guess I’m going to have to migrate to a new nextcloud server installation in a new jail.
Luckily, everything is still working so I’m going to do it when I have time.

After much research and failed attempts, I finally figured out why php occ upgrade was failing.
All I had to do was remove the following line from my config.php file:
'app_install_overwrite' => 'calendar',
I probably should’ve turned it into an array:
'app_install_overwrite' => array ( 0 => 'calendar' )
I don’t even know why it’s there, who inserted it and for what reason, but at this point, who cares? Being able to upgrade my Nextcloud server is all that matters!
And by the way, the Nextcloud log was indeed giving me a clue, but I could only see it as the cause of my issue in retrospect…

[settings] Warning: The value given for app_install_overwrite is not an array. Ignoring...
GET /index.php/settings/apps/list
from 10.0.0.20 by admin at 2021-11-01T07:48:53+01:00

Immense satisfaction…

2 Likes