Cluster of nextcloud 19 sessions with redis + Proxy load balancer round robin

Cluster:

  • 19.0.1-apache
  • Redis 5.0.6 for file locking and session storage
  • PHP 7
  • Apache 2
  • Minio S3 for storage
  • Percona xtradb cluster 8.0.19

The whole cluster sits on top of kubernetes cluster.
Each component has 3 instances, e.g. 3 redis(s) , 3 Nextcloud instances and so forth…

The traffic is behind a load balancer proxy (NginX)

when I enable load balancer round robin on the proxy level (NginX),
the server throws to the user 401, and tells that the user is not logged in

What Have I missing ? =S

Kind of the cluster architecture setup

Server configuration

Docker image:

Docker file
FROM nextcloud:19.0.1-apache

RUN set -ex; \
    \
    apt-get update; \
    apt-get install -y --no-install-recommends \
        ffmpeg \
        libmagickcore-6.q16-6-extra \
        procps \
        smbclient \
        supervisor \
#       libreoffice \
    ; \
    rm -rf /var/lib/apt/lists/*

RUN set -ex; \
    \
    savedAptMark="$(apt-mark showmanual)"; \
    \
    apt-get update; \
    apt-get install -y --no-install-recommends \
        libbz2-dev \
        libc-client-dev \
        libkrb5-dev \
        libsmbclient-dev \
    ; \
    \
    docker-php-ext-configure imap --with-kerberos --with-imap-ssl; \
    docker-php-ext-install \
        bz2 \
        imap \
    ; \
    pecl install smbclient; \
    docker-php-ext-enable smbclient; \
    \
# reset apt-mark's "manual" list so that "purge --auto-remove" will remove all build dependencies
    apt-mark auto '.*' > /dev/null; \
    apt-mark manual $savedAptMark; \
    ldd "$(php -r 'echo ini_get("extension_dir");')"/*.so \
        | awk '/=>/ { print $3 }' \
        | sort -u \
        | xargs -r dpkg-query -S \
        | cut -d: -f1 \
        | sort -u \
        | xargs -rt apt-mark manual; \
    \
    apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false; \
    rm -rf /var/lib/apt/lists/*

RUN mkdir -p \
    /var/log/supervisord \
    /var/run/supervisord \
;

RUN chmod -R 777 /usr/local/etc/php/conf.d

COPY supervisord.conf /

ENV NEXTCLOUD_UPDATE=1

RUN useradd -ms /bin/bash next
USER next

CMD ["/entrypoint.sh", "apache2-foreground"]

Nextcloud configuration:

Config report
<?php
$CONFIG = array (
  'htaccess.RewriteBase' => '/',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'apps_paths' => 
  array (
    0 => 
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 => 
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'overwriteprotocol' => 'https',
  'overwritehost' => '*******',
  'logfile' => '/dev/stdout',
  'trusted_proxies' => 
  array (
    0 => '*******',
  ),
  'skeletondirectory' => '',
  'activity_expire_days' => 14,
  'auth.bruteforce.protection.enabled' => false,
  'knowledgebaseenabled' => false,
  'trashbin_retention_obligation' => 'auto, 30',
  'secret' => '*******',
  'passwordsalt' => '*******',
  'updater.release.channel' => 'stable',
  'integrity.check.disabled' => false,
  'mysql.utf8mb4' => true,
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'filelocking.enabled' => 'true',
  'redis' => 
  array (
    'host' => 'redis-redis-ha-haproxy',
    'port' => '6379',
  ),
  'objectstore' => 
  array (
    'class' => '\\OC\\Files\\ObjectStore\\S3',
    'arguments' => 
    array (
      'bucket' => 'nextcloud',
      'hostname' => 'minio',
      'port' => 9000,
      'autocreate' => true,
      'key' => '*******',
      'secret' => ''*******',',
      'region' => 'optional',
      'use_ssl' => false,
      'use_path_style' => true,
    ),
  ),
  'mail_smtpmode' => 'smtp',
  'mail_smtphost' => 'smtp.sendgrid.net',
  'mail_smtpport' => '465',
  'mail_smtpsecure' => 'ssl',
  'mail_smtpauth' => true,
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpname' => 'apikey',
  'mail_smtppassword' => '*******',',
  'mail_from_address' => 'admin',
  'mail_domain' => '*******',',
  'instanceid' => '*******',
  'trusted_domains' => 
  array (
    0 => 'files.pica.co.il',
  ),
  'datadirectory' => '/usr/local/data',
  'dbtype' => 'mysql',
  'version' => '19.0.1.1',
  'overwrite.cli.url' => '*******',
  'dbname' => 'nextcloud',
  'dbhost' => 'haproxy',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'oc_kfirfer',
  'dbpassword' => '''*******'',',
);

Sorry no solution. But perhaps you find in this document some hints.

https://indico.cern.ch/event/663264/contributions/2818170/attachments/1592445/2520694/An_insiders_look_into_scaling_Nextcloud_-_Matthias_Wobben.pdf

1 Like

duno.

but the config.php doesn’t needs to be shared among the nextcloud instances? (e.g. the password salt and secret should be in sync.)

@kfirfer compared page 15 of the presentation to your diagram I think that /var/www/nextcloud is also on the nfs share.

did you put this folder in a shared kubernetes volume?

again: i never did it, but i would set it up this way.

the 3 instances filesystem are separate, and I gave the same config files to the docker entrypoint script to install the instances

I will try to install only on the first instance, and copy the whole /var/www folder to the two other instances, and see whats happen…

you have to keep the folder in sync.

try: install an app in one nextcloud instance. use another client to connect to another instance and see if the app is installed as well. or change the background on the login page.

I successed by sync the config.php
Seems the installation change the passwordsalt and secret (hash them any time theres a installation)
So when they are different, the sessions are different as-well , Thanks !

I will try to setup a NFS server only for the folders I need to sync

well. you also could share a persistent volume among the nextcloud container.

Yes , thanks

I have the option to use rook ceph shared filesystem: https://rook.io/docs/rook/v1.4/ceph-filesystem.html

But I prefer making nextcloud filesystem separate and stateless as possible, its overkill to install shared storage for one time sync, my prospective…

P.S.
I had also to sync the “.htaccess” file