Nextcloud consuming all memory and swap and crashing server

Nextcloud version: Nextcloud 21.0.0.18
Operating system and version: Ubuntu 20.04
Apache or nginx version: Apache 2.4.41
PHP version: 7.4

The issue you are facing:
Nextcloud cron script consumed all memory 4GB + swap 28GB on the server. After restarting, occ script does the same. I limited the memory allocated to the script, and it always crashes with the following message, whether I give it 600MB of memory or 6GB of it:

sudo -u www-data php -dmemory_limit=600000000 occ

PHP Fatal error: Allowed memory size of 600000000 bytes exhausted (tried to allocate 20480 bytes) in /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php on line 133

sudo -u www-data php -dmemory_limit=6000000000 occ

PHP Fatal error: Allowed memory size of 6000000000 bytes exhausted (tried to allocate 20480 bytes) in /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php on line 133

Currently I cannot bring the server up, and browsing the server shows the maintenance mode message. This is quite a strange error, since I’m the only user on this server and I really didn’t touch it, and there’s no change on the files either. This has to be the work of auto-update, but I’m not sure what was updated and why this is happening.

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:

  1. run sudo -u www-data php occ or cron.php
  2. it immediately fills available memory and starts swapping.

The output of your Nextcloud log in Admin > Logging:

Cannot browse

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'passwordsalt' => '###REDACTED###',
  'secret' => '###REDACTED###',
  'trusted_domains' => 
  array (
    0 => 'localhost',
    1 => '192.168.1.XXX',
    2 => '###REDACTED###',
    3 => '###REDACTED###',
  ),
  'datadirectory' => '/mnt/ncdata',
  'dbtype' => 'pgsql',
  'version' => '20.0.7.1',
  'overwrite.cli.url' => '###REDACTED###',
  'dbname' => 'nextcloud_db',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'ncadmin',
  'dbpassword' => '###REDACTED###',
  'installed' => true,
  'instanceid' => '###REDACTED###',
  'log_type' => 'file',
  'logfile' => '/var/log/nextcloud/nextcloud.log',
  'loglevel' => 0,
  'mail_smtpmode' => 'smtp',
  'remember_login_cookie_lifetime' => '1800',
  'log_rotate_size' => '10485760',
  'trashbin_retention_obligation' => 'auto, 180',
  'versions_retention_obligation' => 'auto, 365',
  'simpleSignUpLink.shown' => 'false',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'filelocking.enabled' => true,
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' => 
  array (
    'host' => '/var/run/redis/redis-server.sock',
    'port' => 0,
    'timeout' => 0.5,
    'dbindex' => 0,
    'password' => '###REDACTED###',
  ),
  'logtimezone' => '###REDACTED###',
  'htaccess.RewriteBase' => '/',
  'enable_previews' => true,
  'preview_libreoffice_path' => '/usr/bin/libreoffice',
  'maintenance' => true,
  'mail_sendmailmode' => 'smtp',
  'mail_from_address' => '###REDACTED###',
  'mail_domain' => '###REDACTED###',
  'mail_smtpsecure' => 'ssl',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpauth' => 1,
  'mail_smtphost' => '###REDACTED###',
  'mail_smtpport' => '###REDACTED###',
  'mail_smtpname' => '###REDACTED###',
  'mail_smtppassword' => '###REDACTED###',
  'onlyoffice' => 
  array (
    'verify_peer_off' => true,
  ),
  'theme' => '',
);

The output of your Apache/nginx/system log in /var/log/____:

[Sat Feb 20 22:21:32.297155 2021] [:notice] [pid 1119] ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/) configured.
[Sat Feb 20 22:21:32.343896 2021] [:notice] [pid 1119] ModSecurity: APR compiled version="1.6.5"; loaded version="1.6.5"
[Sat Feb 20 22:21:32.343928 2021] [:notice] [pid 1119] ModSecurity: PCRE compiled version="8.39 "; loaded version="8.39 2016-06-14"
[Sat Feb 20 22:21:32.343941 2021] [:notice] [pid 1119] ModSecurity: LUA compiled version="Lua 5.1"
[Sat Feb 20 22:21:32.343953 2021] [:notice] [pid 1119] ModSecurity: YAJL compiled version="2.1.0"
[Sat Feb 20 22:21:32.343964 2021] [:notice] [pid 1119] ModSecurity: LIBXML compiled version="2.9.4"
[Sat Feb 20 22:21:32.344222 2021] [:notice] [pid 1119] ModSecurity: StatusEngine call: "2.9.3,Apache,1.6.5/1.6.5,8.39/8.39 2016-06-14,Lua 5.1,2.9.4,b927f9868080bc83cdc5ab14613c97fbc4fbcaf7"
[Sat Feb 20 22:21:32.469290 2021] [:notice] [pid 1119] ModSecurity: StatusEngine call successfully sent. For more information visit: http://status.modsecurity.org/
[Sat Feb 20 22:21:32.517307 2021] [http2:warn] [pid 1438] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.
[Sat Feb 20 22:21:34.740282 2021] [mpm_prefork:notice] [pid 1438] AH00163: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1f configured -- resuming normal operations
[Sat Feb 20 22:21:34.740318 2021] [core:notice] [pid 1438] AH00094: Command line: '/usr/sbin/apache2'

Never mind. Restored from a file-level backup from 3 days ago and everything’s ok now. I think it was the document server that broke the installation by auto-update.

May I ask how you made and recovered from backup?

I use rsnapshot to keep 6 daily, 4 weekly and 4 monthly snapshots of my filesystem. The snapshots are saved to a different physical drive mounted on /media/backup, and rsnapshot is in the crontab to run on the scheduled period. The archwiki page for rsnapshot has a very nice tutorial: rsnapshot - ArchWiki

In order to restore from a snapshot, all I need to do is to boot from a live cd (I use system rescue cd: https://www.system-rescue.org/) and copy over the changed files. Since I explicitly back everything up (you must set that in rsnapshot config), there’s often no problem copying over even the whole filesystem, but more often than not, I just need to only copy over /var, /usr, and /etc and the rest is alright. In this case, since I knew the system was fine 3 days ago and also that I didn’t upload anything to Nextcloud in the past 3 days, that was a fine snapshot to go for, but normally even the day before is fine.

Hey I’m encountering the exact same issue. But occ crashes even if I remove all third party apps from the app directory. What is causing this issue? Unfortunately I have no recent backup since upgrading to Nextcloud 21 :frowning:

The only things that changed for me was two upgrades: document server and mail module. I thought it might have been document server, but it could be mail as well. I saw just now that they released another update for mail, so it might be that the problem is already fixed there. It would be nice if someone from Nextcloud team could help.

EDIT: this might be related: Allowed memory size of 536870912 bytes exhausted · Issue #4528 · nextcloud/mail · GitHub

Also relevant: Cronjob exhausts memory because of apps/mail/lib/IMAP/Threading/Container.php · Issue #3869 · nextcloud/mail · GitHub

It seems like this is definitely an issue with Mail module.

1 Like

Thanks for the info. I upgraded mail to the latest version 1.8.3 but occ still fails with fatal error. Maybe there is already a corruption in the database?

Created a bug report upstream: Running occ: php fatal error, allowed memory size exhausted · Issue #25742 · nextcloud/server · GitHub

1 Like

I think that’s the best way to go. Clearly there’s a bug somewhere.

There is a good and effective suggestion pointed out on the github bug @onny opened ([NC 21] OCC Update error: allowed memory size exhausted · Issue #25742 · nextcloud/server · GitHub) which is to make sure apc is enabled for the php cli you use.

php --version -> gives you the version you use (7.3 for debian 10, 7.4 for most ubuntus, 8.0 for the bleeding edge)
in /etc/php/php_version/cli/conf.d/10-opcache.ini or 20-apcu.ini or whatever .ini in that directory add:

apc.enable_cli=1

that is of course if you use APCu as a caching mechanism (check on config.php), but if you have this problem, I’m pretty sure you do…

hope this helps

3 Likes

Going forward this seems to be necessary, so might set it already as well. More info here:

Oh thats cool thanks