Newly upgraded NC 21.0.1 runs OOM during CRON and process gets killed

Nextcloud version: 21.0.1
Operating system and version: Ubuntu 18.04.5 LTS
Apache or nginx version : Apache/2.4.46 (Ubuntu)
PHP version : 7.4.16

The issue you are facing:
I am seeing CPU usage spikes (really high load) and all my 32 gb of ram being eaten up, as well as 25-30% of my swap (32 gb swapfile) - and then after 40 seconds it just crashes and gets killed. Repeats every 10 minutes as the cronjob starts every 10 minutes. This started happening when i tried to resort one of the “missing indices” messages in the overview using: occ db:add-missing-indices

Is this the first time you’ve seen this error? (Y/N): Yes

The output of your Nextcloud log in Admin > Logging:

No log entries since upgrade

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => '',
  'passwordsalt' => '',
  'secret' => '',
  'trusted_domains' => 
  array (
    0 => 'cloud.xxxx',
  ),
  'trusted_proxies' => 
  array (
    0 => '10.42.0.0/16',
    1 => '103.21.244.0/22',
    2 => '103.22.200.0/22',
    3 => '103.31.4.0/22',
    4 => '104.16.0.0/12',
    5 => '108.162.192.0/18',
    6 => '131.0.72.0/22',
    7 => '141.101.64.0/18',
    8 => '162.158.0.0/15',
    9 => '172.64.0.0/13',
    10 => '173.245.48.0/20',
    11 => '188.114.96.0/20',
    12 => '190.93.240.0/20',
    13 => '197.234.240.0/22',
    14 => '198.41.128.0/17',
    15 => '2400:cb00::/32',
    16 => '2606:4700::/32',
    17 => '2803:f800::/32',
    18 => '2405:b500::/32',
    19 => '2405:8100::/32',
    20 => '2c0f:f248::/32',
    21 => '2a06:98c0::/29',
    22 => '',
    23 => '',
  ),
  'forwarded_for_headers' => 
  array (
    0 => 'CF-Connecting-IP',
    1 => 'HTTP_CF_CONNECTING_IP',
    2 => 'x-forwarded-for',
  ),
  'datadirectory' => '/var/www/nextcloud/data',
  'dbtype' => 'mysql',
  'version' => '21.0.1.1',
  'overwrite.cli.url' => '',
  'overwriteprotocol' => 'https',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'oc_Administrator',
  'dbpassword' => '',
  'installed' => true,
  'theme' => '',
  'loglevel' => 0,
  'maintenance' => false,
  'mail_domain' => '',
  'mail_from_address' => 'cloud',
  'mail_smtpmode' => 'smtp',
  'mail_smtphost' => '',
  'mail_smtpauthtype' => '',
  'mail_smtpauth' => 1,
  'mail_smtpport' => '',
  'mail_sendmailmode' => '',
  'mail_smtpsecure' => 'tls',
  'mail_smtpname' => '',
  'mail_smtppassword' => '',
  'filesystem_check_changes' => 1,
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' => 
  array (
    'host' => '/var/run/redis/redis-server.sock',
    'port' => 0,
  ),
  'updater.release.channel' => 'stable',
  'app_install_overwrite' => 
  array (
    0 => 'files_trackdownloads',
    1 => 'spreed',
    2 => 'afterlogic',
    3 => 'camerarawpreviews',
    4 => 'ocsms',
    5 => 'files_readmemd',
    6 => 'socialsharing_email',
    7 => 'documents',
    8 => 'external',
    9 => 'admin_notifications',
    10 => 'twofactor_email',
  ),
  'updater.secret' => '$2y$10$.xxxxxxxxx',
);

The output of your Apache/nginx/system log in /var/log/____:

theres really no useful info (just access denied to some clients by configuration, theres no traces of anything useful in the logs.) 
1 Like

Have you checked the syslog yet to see the memory footprint of all processes at the time the oom-killer is doing it’s job?

Please copy/paste your php error here.

Here’s the only thing i manage to get;

Apr 25 18:20:01 big-cookie CRON[819]: pam_unix(cron:session): session opened for user www-data by (uid=0)
Apr 25 18:20:01 big-cookie CRON[820]: (www-data) CMD (php -f /var/www/nextcloud/cron.php)

Apr 25 18:20:44 big-cookie kernel: php[821]: segfault at 7ffc51d5eff8 ip 0000564e7a0ef1b2 sp 00007ffc51d5f000 error 6 in php7.4[564e79e67000+3e3000]

Apr 25 18:20:46 big-cookie CRON[819]: (CRON) info (No MTA installed, discarding output)
Apr 25 18:20:46 big-cookie CRON[819]: pam_unix(cron:session): session closed for user www-data
Apr 25 18:20:51 big-cookie sudo[550]: pam_unix(sudo:session): session closed for user root

I got this output from journalctl -xe

Here is screenshot of usage 2 seconds before it crashed (CPU maxed on 1 core, RAM Maxed out and SWAP being eaten up rapidly;

1 Like

Hi again,

I solved my issue by following the instructions to enable CLI on APCU config.
Answer in this comment: [SOLVED] Occ command; PHP Fatal error, Allowed memory size of XXX bytes exhausted - #17 by andreasus

OCC now runs normally.

1 Like