[SOLVED] PHP trying to allocate absurd quantities of memory with local memcache enabled

nlindq · June 21, 2019, 9:28pm

Support intro

Sorry to hear you’re facing problems

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can

Nextcloud version (eg, 12.0.2): 16.0.1
Operating system and version (eg, Ubuntu 17.04): CentOS 6.10 (x86_64)
Apache or nginx version (eg, Apache 2.4.25): Apache 2.2.15
PHP version (eg, 7.1): 7.1 (packages from Webtatic repository)

The issue you are facing:

After upgrading from PHP 5.6 to PHP 7.1 and upgrading NextCloud from 12.0.12.2 to 16.0.1, we are experiencing intermittent (but multiple times / day occurrences of NextCloud freezing up with associated memory exhaustion errors in nextcloud.log and Apache error logs related to SCSSCacher.php.

Very similar to, for eg. this issue but we already have PHP 7.1 Zend opcache installed, configured and working.

Is this the first time you’ve seen this error? (Y/N):

Steps to replicate it:

Enable memcache.local using APCu
Use NextCloud for a variable amount of time (sometimes minutes, sometimes hours)

The output of your Nextcloud log in Admin > Logging:

{"reqId":"WHKAvxQdrnrKN2P9juRN","level":3,"time":"2019-06-21T10:27:22-06:00","remoteAddr":"192.168.60.129","user":"nels","app":"PHP","method":"GET","url":"\/index.php\/apps\/files\/","message":"Allowed memory size of 536870912 bytes exhausted (tried to allocate 8608480258408150608 bytes) at \/var\/www\/html\/maei.ca\/nextcloud\/lib\/private\/Template\/SCSSCacher.php#352","userAgent":"Mozilla\/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko\/20100101 Firefox\/60.0","version":"16.0.1.1"}

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'passwordsalt' => 'REDACTED',
  'datadirectory' => 'REDACTED/owncloud-data',
  'dbtype' => 'pgsql',
  'version' => '16.0.1.1',
  'dbname' => 'REDACTED',
  'dbhost' => '/tmp',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'oc_admin',
  'dbpassword' => '',
  'installed' => true,
  'instanceid' => 'REDACTED',
  'ldapIgnoreNamingRules' => false,
  'maintenance' => false,
  'forcessl' => true,
  'share_folder' => '/Shared',
  'maxZipInputSize' => 838860800,
  'allowZipDownload' => true,
  'trusted_domains' => 
  array (
    0 => 'owncloud.example.org',
    1 => 'nextcloud.example.org',
 ),
  'overwrite.cli.url' => 'https://owncloud.example.org/',
  'cron_log' => true,
  'logtimezone' => 'America/Edmonton',
  'theme' => '',
  'mail_smtpmode' => 'smtp',
  'mail_from_address' => 'webmaster',
  'mail_domain' => 'example.org',
  'secret' => 'REDACTED',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'has_internet_connection' => true,
  'appstoreenabled' => true,
  'apps_paths' =>
  array (
    0 =>
    array (
      'path' => 'REDACTED/owncloud/apps',
      'url' => '/apps',
      'writable' => true,
    ),
  ),
  'updatechecker' => true,
  'default_language' => 'en',
  'defaultapp' => 'files',
  'enable_previews' => true,
  'preview_libreoffice_path' => '/usr/bin/libreoffice',
  'enabledPreviewProviders' =>
  array (
    0 => 'OC\\Preview\\PNG',
    1 => 'OC\\Preview\\JPEG',
    2 => 'OC\\Preview\\BMP',
    3 => 'OC\\Preview\\XBitmap',
    4 => 'OC\\Preview\\MP3',
    5 => 'OC\\Preview\\TXT',
    6 => 'OC\\Preview\\MarkDown',
    7 => 'OC\\Preview\\PDF',
    8 => 'OC\\Preview\\MSOffice2003',
    9 => 'OC\\Preview\\MSOffice2007',
    10 => 'OC\\Preview\\MSOfficeDoc',
    11 => 'OC\\Preview\\OpenDocument',
  ),
  'cache_path' => '',
  'filesystem_check_changes' => 1,
  'asset-pipeline.enabled' => true,
  'mount_file' => 'data/mount.json',
  'redis' =>
  array (
    0 => 'tmp/redis.sock',
    'timeout' => 0,
    'dbindex' => 0,
  ),
  'filelocking.enabled' => true,
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'loglevel' => 2,
  'trashbin_retention_obligation' => 'auto',
  'ldapProviderFactory' => '\\OCA\\User_LDAP\\LDAPProviderFactory',
  'mail_smtphost' => 'REDACTED',
  'mail_sendmailmode' => 'smtp',
  'mail_smtpport' => '587',
);

The output of your Apache/nginx/system log in /var/log/____:

[Fri Jun 21 10:27:22 2019] [error] [client 192.168.60.129] PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 8608480258408150608 bytes) in /var/www/html/maei.ca/nextcloud/lib/private/Template/SCSSCacher.php on line 352

nlindq · July 2, 2019, 3:47pm

Any thoughts?

gas85 · July 3, 2019, 10:37am

You have more complicated setup as in tread that you pointed. First of all you are using 2 caches: APCu for local and Redis for locking. I do not see any Redis server config in you config.php. Could this be a problem?
From here: Memory caching — Nextcloud latest Administration Manual latest documentation
You can verify that the Redis daemon is running with ps ax:

ps ax | grep redis
22203 ? Ssl    0:00 /usr/bin/redis-server 127.0.0.1:6379

Restart your Web server, add the appropriate entries to your config.php, and refresh your Nextcloud admin page. This example config.php configuration uses Redis for the distributed server cache:

'memcache.distributed' => '\OC\Memcache\Redis',
'redis' => [
     'host' => 'redis-host.example.com',
     'port' => 6379,
],

f you want to connect to Redis configured to listen on an Unix socket (which is recommended if Redis is running on the same system as Nextcloud) use this example config.php configuration:

'memcache.local' => '\OC\Memcache\APCu',
'memcache.distributed' => '\OC\Memcache\Redis',
'redis' => [
     'host'     => '/var/run/redis/redis.sock',
     'port'     => 0,
     'dbindex'  => 0,
     'password' => 'secret',
     'timeout'  => 1.5,
],

gas85 · July 3, 2019, 10:39am

I believe this is best example for you: https://docs.nextcloud.com/server/16/admin_manual/configuration_server/caching_configuration.html?highlight=opcache#organizations-with-single-server-and-clustered-setups

nlindq · July 3, 2019, 9:53pm

It is there, but using legacy (migrated from ownCloud) syntax, and with errors:

There is a running redis server on localhost, and looking at lib/private/RedisFactory.php, it appears the default options were sufficient to connect to it (if I intentionally break it by setting an incorrect port/host/etc. then NextCloud won’t load at all).

nb that I have no issue running with the redis server for file locking; the problem only occurs if the APCu memcache is enabled; if I comment out the

line, then it will function indefinitely without error (albeit significantly slower for certain operations).

nlindq · July 5, 2019, 3:32pm

Just a quick FYI update to mention there was an update to the PHP 7.1 packages available which includes fixes for heap corruption, etc. when processing graphics. As we do have image galleries, etc. present it’s possible this may have resolved my issue. I’ve provisionally re-enabled the local APCu memcache and have been running for more than 24 hours with no crashes.

nlindq · July 12, 2019, 4:01pm

We’ve experienced no further issues since last report. I’m going to mark this as solved.