Server not responding after Nextcloud upgrade (20.0.0.9 to 21.0.1)

ℹ️ Support
,
1

Nextcloud version : 21.0.1
Operating system and version : Debian 10 Buster (up to date)
Apache or nginx version : NginX 1.14.2
PHP version : PHP 7.3

The issue you are facing:

Hi, I have a problem after upgrading Nextcloud to the last stable version (21). Few minutes after performing the upgrade, the server became extermely slow (CPU, RAM and SWAP @ 100%) and I had no way to connect to it with SSH. After hard rebooting it via Web Manager (the server is hosted by Kimsufi), it was unable to reply by ping. I can’t provide log currently, I’m waiting to get my ssh access back.

Yesterday, Nextcloud was upgraded from 20.0.0.8 to 20.0.0.9 without any problem. But it seems the upgrade 20 to 21 was fatal (I didn’t switch update channel to beta, 21.0.1 was proposed in the stable channel.)

Anybody encounters the same issue already? What can I do or check when I’ll be able to connect to my server?

Thanks in advance for your help and suggessions.

Is this the first time you’ve seen this error? : Yes

2

Some news about my problem. SSH worked again during 5 minutes so I was able to monitor my processes on the server and I saw two cron tasks launched by my nextcloud user that eat all the memory. So I know where is the problem. The first task is the nextcloud/cron.php and the second is the occ preview:pre-generate job, launched with php7.3. The two task share all the memory. I didn’t have the time to stop the cron tasks… And I’m not able again to access to my server now…

Is there known issues with NC21 and PHP7.3 ?
Have a nice evening.

3

Same issue here, upgrade caused server to be fully unresponsive and after long waiting the upgrade process failed at the “code integrity check” stage.

I reverted back to 20.0.9 using the downloaded code and a backup of my config. This started another “web-upgrade” with several checks. Although this is not a clean restore (e.g. SQL database not reverted), it seems to work fine. Worst case I have a full backup including database, but to restore this it would be much more work.

4

Same issue I saw the update 20.0.8 > 20.0.9. Then it went to 21.0.1 instead and the server froze like others.

5

I did the upgrade from 20.0.8 >20.0.9 the day before, still the upgrade from 20.0.9 > 21.0.1 badly failed :frowning:
Using NC since many years, (since OC times) and the last years never had any major issues. This is by far the worst upgrade failure I had.

1 Like
6

Hi friends :slight_smile: “Happy” to see that I’m not alone in this difficult situation…

I can confirm that the problem of RAM comes when my cron jobs starts, the two jobs calls php7.3 to run. I currently comment those line in the crontab of my Nextcloud user and start php7.3-fpm to run my Nextcloud pool. Everything looks fine now. But of course it’s not satisfactory…

For you all, what is your php version?

7

I’m on php 7.4. @Mainmain, may I ask how/where exactly you commented out the cron jobs?

8

Logged with my nextcloud linux user, I run the ‘crontab -e’ command and add a # at the beginning of each line.

crontab

9

All right, then I got you right :slight_smile: The second is for a plugin, though, right? at least frees enough memory for me to try & restore the backup :grimacing:

10

Yes, I use this app: Preview Generator - Apps - App Store - Nextcloud

Good luck with the restoration :slight_smile:

I’m wondering, if I limit RAM and CPU resources to my nextcloud user (with cgroup), maybe it will be enough to do some tests with php. I would like to test with php8.0 without loosing my server at each try.

11

From the issue on Github: There is a (crude) workaround for the out-of-memory issue. Enable apcu for CLI PHP processes. That “solves” the issue for most users when they’re using apcu caching.

Add this to your PHP config
apc.enable_cli = 1

Why “a crude workaround”? Under normal circumstances it is not recommended to enable apcu for PHP CLI processes. It is a setting for testing and debugging. So I don’t understand why Nextcloud is recommending this setting and making it now a hard requirement for some setups.

See: PHP: Runtime Configuration - Manual

12

Just so you (and devs) know that this is a real issue with real people experiencing it, I have exactly the same problem here with Nextcloud 21 updates. It all goes well until cron jobs run or if I set up a manual occ files:scan job whereupon the server freezes at 100% cpu and ram use and becomes inaccessible.
I take a backup image file before attempting any of this so I was able to revert to 20.08 quite quickly and without fuss.
Would really like to get 21 installed though given that the main benefits are supposed to be a speed up of the web GUI :slight_smile:
I’m using Ubuntu server 20.10 with Apache2 and php7.4 and currently nextcloud 20.08

I joined another thread about this here:
https://help.nextcloud.com/t/freezes-entire-server-computer-caused-by-var-www-nextcloud-cron-php/112359
James

13

Current situation is that if you have configured APCU properly (on CLI as well, you don’t have this problem. Is it the right way? Github is the right place to discuss this with the developers: https://github.com/nextcloud/server/issues/25742

On the forum, we can just share the workarounds and hope for a nicer fix that not all people fall into this trap.

1 Like
14

Agree. This is a disastrous upgrade, and i hope a fix will be issued asap

1 Like
15

WOW Nextcloud 21 upgrade is very bad nothing but issues i’ve reported so many issues all the way to encryption :frowning:

1 Like
16

Tried to install the latest upgrade to Nexctcloud 21 today, hopeful that this issue would have been resolved, that is version 21.0.2 RC1
Same old same old, server freezes after about two minutes of use and becomes inaccessible to all.
At this point it is impossible to do any work on the server, including attempting to edit the config.php so, I reinstated the server image backup I took and edited the config.php first.
Nextcloud 20 will not run with that lne added t my config.php unless I am putting it in the wrong place?
My config.php looks very like a script rather than a config file, is this correct? If so, where should that line be added please?
Bottom line is, still no upgrade to 21 for me

====config.php=============(
?php
$CONFIG = array (
‘instanceid’ => ‘oc4repx2mfk4’,
‘passwordsalt’ => 'redacted,
‘secret’ => ‘redacted’,
‘trusted_domains’ =>
array (
0 => ‘192.168.1.4’,
1 => ‘redacted’,
2 => ‘broncloud.sasstaff.co.uk’,
),
‘datadirectory’ => ‘/mnt/Bronstore/nextcloud_data’,
‘dbtype’ => ‘mysql’,
‘version’ => ‘20.0.10.0’,
‘overwrite.cli.url’ => ‘https://192.168.1.4/nextcloud’,
‘dbname’ => ‘nextcloud’,
‘dbhost’ => ‘localhost’,
‘dbport’ => ‘’,
‘dbtableprefix’ => ‘oc_’,
‘dbuser’ => ‘nextclouduser’,
‘dbpassword’ => ‘removed’,
‘installed’ => true,
‘mail_smtpmode’ => ‘sendmail’,
‘mail_smtpauthtype’ => ‘LOGIN’,
‘mail_from_address’ => ‘jkp’,
‘mail_domain’ => ‘bronyaur.co.uk’,
‘maintenance’ => false,
‘theme’ => ‘’,
‘loglevel’ => 0,
‘memcache.local’ => ‘\OC\Memcache\APCu’,
‘mysql.utf8mb4’ => true,
‘updater.release.channel’ => ‘beta’,
‘app_install_overwrite’ =>
array (
0 => ‘fulltextsearch’,
1 => ‘calendar’,
2 => ‘spreed’,
3 => ‘ransomware_protection’,
4 => ‘maps’,
),
‘preview_max_x’ => ‘1080’,
‘preview_max_y’ => ‘1920’,
‘data-fingerprint’ => ‘557ec2843a35aa888a855e1715d9482f’,
);

17

Sadly I have the same issue: nextcloud just stops responding after a while, and I receive bad gateway errors/timeouts/etc. Works fine again after a restart of php7.4-fpm it seems. Sometimes a mysql restart helps as well.

Any idea? Any updates from developers?

18

This is my (and others) issue. The upgrade goes smoothly enough but, as soon as the cron job runs and invokes occ the server resources max out and the whole thing freezes.

Very sad. I have tried twice, once upgrading using the web updater and the other I uninstalled NC20 and started with a fresh installation. Same thing happened.

I seem to be stuck for now. But, I will hope that jumping to 22 will be a goer :slight_smile:
James

19

In my config, I have opcache.enable_cli=1 - I guess it does the same thing?!?! I didn’t have any issues upgrading, at least.

For people who missed it, this is the solution for the problem - and a newer version of Nextcloud will simply warn when you don’t have this enabled. Alfred pointed out this is against the recommended settings for PHP - I’ll ask the devs to figure out if that’s so and if our solution (enforcing a non-recommended option) is a good idea.

Looking at the second answer here, it seems that having an opcache for the CLI is actually fine, recommended even if you want the performance benefit:

20

Good morning from Scotland.
I have tried this fix, as I mentioned in one of posts above where I reproduced my config.php.
Sadly this completely destroyed my NC setup which wouldn’t run until I removed the entry apc.enable_cli = 1
Incidentally, this entry didn’t previously exist so I was not sure where to put it and what syntax to use since the config.php in my setup looks rather more lie a scripting language than a simple config file.

Some assistance with this might get me to the point I have been aiming for since the release of NC21 :slight_smile:

My config.php again for ease of reference:

====config.php=============(
?php
$CONFIG = array (
‘instanceid’ => ‘oc4repx2mfk4’,
‘passwordsalt’ => 'redacted,
‘secret’ => ‘redacted’,
‘trusted_domains’ =>
array (
0 => ‘192.168.1.4’,
1 => ‘redacted’,
2 => ‘broncloud.sasstaff.co.uk’,
),
‘datadirectory’ => ‘/mnt/Bronstore/nextcloud_data’,
‘dbtype’ => ‘mysql’,
‘version’ => ‘20.0.10.0’,
‘overwrite.cli.url’ => ‘https://192.168.1.4/nextcloud’,
‘dbname’ => ‘nextcloud’,
‘dbhost’ => ‘localhost’,
‘dbport’ => ‘’,
‘dbtableprefix’ => ‘oc_’,
‘dbuser’ => ‘nextclouduser’,
‘dbpassword’ => ‘removed’,
‘installed’ => true,
‘mail_smtpmode’ => ‘sendmail’,
‘mail_smtpauthtype’ => ‘LOGIN’,
‘mail_from_address’ => ‘jkp’,
‘mail_domain’ => ‘bronyaur.co.uk’,
‘maintenance’ => false,
‘theme’ => ‘’,
‘loglevel’ => 0,
‘memcache.local’ => ‘\OC\Memcache\APCu’,
‘mysql.utf8mb4’ => true,
‘updater.release.channel’ => ‘beta’,
‘app_install_overwrite’ =>
array (
0 => ‘fulltextsearch’,
1 => ‘calendar’,
2 => ‘spreed’,
3 => ‘ransomware_protection’,
4 => ‘maps’,
),
‘preview_max_x’ => ‘1080’,
‘preview_max_y’ => ‘1920’,
‘data-fingerprint’ => ‘557ec2843a35aa888a855e1715d9482f’,
);