High-Performance-Backend (Talk) slows down Nextclound dramatically with more than 5 participants in a call

Greetings all together :slight_smile:

Nextcloud version (eg, 20.0.5): 23.0.3
Operating system and version (eg, Ubuntu 20.04): Ubuntu 20.04 LTS
Apache or nginx version (eg, Apache 2.4.25): nginx 1.20.2
PHP version (eg, 7.4): 7.4

Nextcloud:
Nextcloud is deployed localy on relatively outdated hardware (Intel Core2Quad Q8200), 4GB RAM, 2x 4TBHDD Raid1. Connection is 250Mbit down and 40Mbit up. Everxthing runs using a modified docker-compose file from official docker github examples.

High-Performance-Backend / Signaling / coTurn:
Additional a TURN (coTurn) with a “high-performance-backend” is deployed on a rented vServe with 4 CPU cores, 8GB RAM and 300MBit up/down. I mostly followed the instructions on Git: Nextcloud-Spreed-Signaling with some additional letsencrypt/certbot magic.

The issue you are facing:

Nextcloud is snappy and responsive

If a Talk-Conversation with 4 or more participants exist, the nextcloud becomes noticably slow. Adding another person becomes painfully slow and with 6 or more participants all pages on Nextclound with have loading times > 30s. After a while the calls breaks together, all participants loose connection and the Nextcloud recoveres back to normal.

Nextcloud CPU usage is 30-50% during calls, RAM is might be a issue since it is pretty maxed out.
Signaling CPU usage is <10% and RAM usage also <10%

Traffic on Nextcloud-Server is <1Mbit/s both ways.
Download (RX) is < 10Mbit/s and upload (TX) on signaling maxed out at 20Mbit/s. So there is also prenty of room.

I’ll replace the Nextclouds hardware within the next days with a Xeon E5-1231v3 and 32GB RAM to see if thats the bottleneck.

Does anyone have an Idea. Am I missing something conceptual here or is something with this setup wrong at all? I’m running out of idears othern than replacing the hardware.

Is this the first time you’ve seen this error? (Y/N): Yes

Steps to replicate it:

  1. Install docker on a local machine
  2. Install coTurn and High-Performance-Backend on another remote server
  3. Have oarticipants with increasing number.

The output of your Nextcloud log in Admin > Logging:

Logs are clean. Only once in a while a failed login.

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'apps_paths' =>
  array (
    0 =>
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 =>
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
),
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
    'host' => 'redis',
    'password' => '',
    'port' => 6379,
  ),
  'instanceid' => '<ID>',
  'passwordsalt' => '<SALT>',
  'secret' => '<SECRET>',
  'trusted_domains' =>
  array (
    0 => '<URL>',
  ),
  'datadirectory' => '/var/www/html/data',
  'dbtype' => 'mysql',
  'version' => '23.0.3.2',
  'overwrite.cli.url' => 'https://<URL>',
  'overwitehost' => '<URL>',
  'overwriteprotocol' => 'https',
  'dbname' => 'nextcloud',
  'dbhost' => 'db',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud',
  'dbpassword' => '<DB_PASSWORDD>',
  'installed' => true,
  'mail_from_address' => '<DOMAIN>',
  'mail_smtpmode' => 'smtp',
  'mail_sendmailmode' => 'smtp',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_domain' => '<MAIN_DOMAIN>',
  'mail_smtphost' => '<MAIN_SMTP>',
  'mail_smtpauth' => 1,
  'mail_smtpport' => '465',
  'mail_smtpname' => '<MAIN_LOGIN>',
  'mail_smtppassword' => '<SMTP_PASSWORD>',
  'mail_smtpsecure' => 'ssl',
  'default_phone_region' => 'DE',
  'maintenance' => false,
  'loglevel' => 2,
  'twofactor_enforced' => 'false',
  'twofactor_enforced_groups' =>
  array (
    0 => 'F...',
    1 => 'L...',
    2 => 'O...',
    3 => 'P...',
    4 => 'P...',
     ...
  ),
  'twofactor_enforced_excluded_groups' =>
  array (
    0 => 'admin',
    1 => 'L...',
    2 => 'O...',
    3 => 'P...',
  ),
);

The output of your Apache/nginx/system log in /var/log/____:

No errors at all.

I found the issue:

The problem was that that php-fpm has a default value of pm.max_children = 6. With increasing number of connections the NC-Server ran out of php-fpm-processes to handle the connections. I applied the recommended setting and the server handles 30+ simultaneous talk connections without a blink and the NC remains super snappy.

Lesson learned :wink:
Nextcloud Server Tuning Documentation

1 Like