Upgrade issues to 21.0.1

Dear nextcloud community. I did an upgrade to my nc instance today from 20.0.8 to 21.0.1.

I found the source of this behaviour. I thought the upgrade would be to 20.0.9 but unfortunately it was to 21.0.1

Unfortunately the occ upgrade hangs until it is killed by the kernel:

I give you the STDOUT of a recurrent occ upgrade:

sudo -u www-data php /var/www/nextcloud/updater/updater.phar
Nextcloud Updater - version: v20.0.0beta4-11-g68fa0d4

Current version is 21.0.1.

Update to Nextcloud 21.0.1 available. (channel: "stable")
Following file will be downloaded automatically: https://download.nextcloud.com/server/releases/nextcloud-21.0.1.zip
Open changelog ↗

Steps that will be executed:
[✔] Check for expected files
[✔] Check for write permissions
[✔] Create backup
[✔] Downloading
[✔] Verify integrity
[✔] Extracting
[✔] Enable maintenance mode
[✔] Replace entry points
[✔] Delete old files
[✔] Move new files in place
[✔] Done

Continue update? [y/N] y

Info: Pressing Ctrl-C will finish the currently running step and then stops the updater.

[✔] Check for expected files
[✔] Check for write permissions
[✔] Create backup
[✔] Downloading
[✔] Verify integrity
[✔] Extracting
[✔] Enable maintenance mode
[✔] Replace entry points
[✔] Delete old files
[✔] Move new files in place
[✔] Done

Update of code successful.

Should the "occ upgrade" command be executed? [Y/n] 
Killed

Keep maintenance mode active? [y/N] y
Maintenance mode kept active

[/details]

Nextcloud version : 21.0.1
Operating system and version: Debian Buster (up to date)
Apache or nginx version: Apache/2.4.38 (Debian)
PHP version: PHP 7.3.27-1~deb10u1 (cli) (built: Feb 13 2021 16:31:40) ( NTS )

Apache Error:

[php7:error] [pid 18126] [client 2.206.161.85:36466] PHP Fatal error:  Uncaught Error: Class 'Exception' not found in /var/www/nextcloud/remote.php:46\nStack trace:\n#0 {main}\n  thrown in /var/www/nextcloud/remote.php on line 46

My instance is not reachable right now and I don’t know what to do.

2 Likes

As an extra bonus. If you are using shared Calendars I would NOT upgrade anyway unless you want to fix your broken installation afterwards. They still have a major bug in Caldav support that they keep ignoring and breaks shared calendars.

Maybe same issue as here https://help.nextcloud.com/t/server-not-responding-after-nextcloud-upgrade-20-0-0-9-to-21-0-1/ ?

Do you have any details? Or links to issues?
I’d like to find out more about this before upgrading my server…

Thanks!

I think that you should wait for an update, or at least have a solid backup.

Sorry, this might sound rude but it’s not important what you or Charles_Williams think.
He claimed that there is a bug but I don’t find any evidence or bug reports for it.

So please give us some more details (best option: open an issue) or please don’t say such things.

I have been updating between stable and beta versions for years, and never had this kind of issue. Yesterday, I attempted an upgrade from 20.0.8 to 21.0.1, and my server is now dead and unresponsive. And it has been for the past 9 hours, causing a severe outage and downtime, including other services hosted on the same server.

All I can say is: I started an upgrade from the admin panel, waited a few minutes (backing up, downloading, etc). Then, I decided to keep going with the web interface, and waited for about half an hour on the Nextcloud spinning wheel. After that time, the update failed with “integrity check” errors, and the server died. Refreshing the page did nothing, and that’s the last time I had news from my server.

All I can show is this: my CPU usage before and after attempting the update.

I would also show RAM and swap, but the hosting company’s control panel does not show them, and I can’t access Grafana - but I’d be more than happy to share any needed info.

I can also show this, however:

1 Like

Update (i can’t post more than 2 media): here’s how Grafana looked.

Hi. Similar problem here. Leaving a comment to bump the thread :ok_man:t3:

@mind-overflow Unfortunately you overlooked that I responded to the post of @Charles_Williams. He claimed that there are problems with “shared Calendars” without giving any details. So you are accusing me of things that I never wrote. That’s what I would call rude. :frowning:

You however are talking about the problem of the original poster which I had also and that’s why I looked at this thread. What might help you also: do a search for “APCu”.

I removed the line from my config:
‘memcache.local’ => ‘\OC\Memcache\APCu’,

Alternatively you can add this line to /etc/php/7.3/mods-available/apcu.ini :
apc.enable_cli=1

My server is up and running again (after a long outage also) sigh

I would appreciate an apology…

1 Like

Ok. I unfortunately have no great way to prove this, but here’s what I did.

I restored 20.0.8 from the /data/updater-xxxxxxx/backups/ folder. I used rsync

rsync -Aaxv /var/www/nextcloud/data/updater-xxxxxxx/backups/nextcloud-20.0.8.1-xxxxxxxx /var/www/nextcloud/

(be careful with the correct folder names).

Then, without touching the database, I restarted everything. I visited the Nextcloud URL on the browser, and I logged in. After logging in, I got an internal server error.

I looked into the Nextcloud log (/var/www/nextcloud/data/nextcloud.log), and I saw an error about missing database tables.

My intuition was that, probably, the database was updated to the 21.0.1 format. So, I did this:

  • I went into the nextcloud/updater/ directory.
  • I ran the updater manually.
sudo -u www-data php updater.phar

It went fairly smoothly, downloaded a bunch of stuff and moved it in place. Then it asked

Should the "occ upgrade" command be executed? [Y/n]

I said “yes”. The update process started, and it showed a bunch of “repair steps”. Nice.
At the end, there was a final step, where the server died on me last time:

2021-04-10T09:25:32+00:00 Starting code integrity check...

At this point, I launched a parallel SSH terminal and launched htop, to monitor resources usage.

The Nextcloud cron.php script was launcher once, and taking up about 6GBs of RAM. After 3/4 minutes, it was spawned ANOTHER time, and started taking up RAM and CPU. I killed both of them, and RAM usage dropped to 2GB. The update script did not fail however, saying that “integrity check” was passed. (I doubt it, as I literally killed the script that was checking for it - but it doesn’t mean it failed either). After that, Nextcloud was responsive again, on latest version 21.0.1.

2021-04-10T09:30:30+00:00 Finished code integrity check
2021-04-10T09:30:31+00:00 Update successful
2021-04-10T09:30:31+00:00 Maintenance mode is kept active
2021-04-10T09:30:31+00:00 Resetting log level

Skipping integrity check is not a good thing nor a good practice, but it is what it is. It never happened to me in the past that the integrity check would take up so many resources.

I am terribly sorry. I removed the triggered parts from my message and I’d like to apologize for it. I absolutely confused the two people and mentally connected your message to the wrong one.
Hope I haven’t ruined your day.

Btw, I don’t understand. Why should I remove APCu from my Nextcloud config? It wasn’t there to begin with, but the Nextcloud docs suggested enabling it, if I’m not wrong.

Apology accepted. :slight_smile:

This was my first failed update since my migration from owncloud to nextcloud. All the previous ones went really smooth. I hope the NC guys will come back to the usual quality next time.

If the APCu line wasn’t in your config then your exact bug might be different than mine. I added that line 1-2 years ago after the admin page suggested that. In the meantime there was a php upgrade on my Debian machine which lead to the missing apc.enable in the php config.

I hope your NC instance is stable now again. Mine seems to be. Calendars work as before.
Haven’t seen the promised big speed improvement yet but didn’t do any real tests so it might be there. :slight_smile:

Thank you.

Actually, I haven’t been clear - the APCu line is and already was in my Nextcloud config file, as suggested by the manual, but the apc.enable_cli=1 was not in my apcu.ini file. I was just wondering why that entry would be needed, as the manual didn’t say anything about it - or, why would I need to disable APCu cache, since the manual suggests the opposite.

Actually I cannot remember what or who told me to add that php config line.
But someone did. :slight_smile: And it worked.

I solved the issues with the changed apc settings, but I didn’t changed it globally but only in cront jobs as described in [NC 21] OCC Update error: allowed memory size exhausted · Issue #25742 · nextcloud/server · GitHub

2 Likes