Redis cluster support and configuration

šŸš§ Installation
1

Hi everybody,

I canā€™t figure out how to configure nextcloud to use a external redis cluster with master and slaves as a cache.
I have read that this is supported, but despite of very long and deep searches on the web, i couldnā€™t find any explanation on how to configure it.

I have a two nodes redis cluster working, with a master and a slave. If I configure the master node in the nextcloud config.php file, it works like a charm. But if the master redis node fails, nothing seems to handle the redirection on the old slave and promote it as new master.

Maybe I should use somewhere redis-sentinel to promote the slave to master, but how announce it to nextcloud ?

If someone has informations about that, or links where I can find it, thanks a lot !

Best regards.

2

Maybe I found the beginning of a response by exploring files on the Nextcloud Server Github page.

I found there a commit called ā€˜redis cluster support addedā€™, where there are modifications in the config.sample.php file talking about Redis clustering.

There is this section in that file :

/**
 * Connection details for a Redis Cluster
 *
 * Only for use with Redis Clustering, for Sentinel-based setups use the single
 * server configuration above, and perform HA on the hostname.
 *
 * Redis Cluster support requires the php module phpredis in version 3.0.0 or higher.
 *
 * Available failover modes:
 *  - \RedisCluster::FAILOVER_NONE - only send commands to master nodes (default)
 *  - \RedisCluster::FAILOVER_ERROR - failover to slaves for read commands if master is unavailable
 *  - \RedisCluster::FAILOVER_DISTRIBUTE - randomly distribute read commands across master and slaves
 */
'redis.cluster' => [
	'seeds' => [ // provide some/all of the cluster servers to bootstrap discovery, port required
		'localhost:7000',
		'localhost:7001'
	],
	'timeout' => 0.0,
	'read_timeout' => 0.0,
	'failover_mode' => \RedisCluster::FAILOVER_DISTRIBUTE
],

I will rebuild my Redis clustering setup with the new clustering capabilities of Redis 3.0 (which seems not use redis-sentinel to promote masters), and then install a nextcloud server using these settings to check if all is working, then Iā€™ll let you know the results.

1 Like
3

I have implemented Netflix Dynomite ( https://github.com/Netflix/dynomite ) as an active-active Redis cluster. Works nice and you donā€™t have to do any failover management/configuration in Nextcloud.

4

Did you have any luck?

5

Good day Ruud,

after understanding that a redis cluster needs at least 3 working master nodes to work - too many for my resources - I started looking for an alternative.
I found your post, and set up a dynomite cluster.
it consists of two racks of three nodes each:
server1: rack1-node1, rack2-node3
server2: rack1-node2, rack2-node1
server3: rack1-node3, rack2-node2
The three nodes of each rack cover the keyspace. The second rack replicates the first.

In Nextcloud, I thought that setting the nodes of both racks as a cluster would be enough to get redundancy and load balancing, but Nextcloud complains and does not work.
I tried to remove the three nodes of second rack from the configuration, but no luck.
It only worked when I used a single redis server configuration pointing at one of dynomite nodes/ports

If that node fails, data are still available, but Nextcloud would not be able to retrieve them.

Is this assumprion correct?

If so, how do I configure Nextcloud (or setup dynomite) to get automatic failover in case of node failure?

Regards.

6

Nowhere its mentioned how to setup redis cluster before we can fill in configuration into config.php !

7

I run Nextcloud in a Kubernetes cluster, and I have setup a Redis cluster using the Bitnami Helm chart.

Nextcloud is configured just as mentioned by @ssouron - with the exception that my seeds list only holds the DNS name of the Kubernetes service pointing to the Redis cluster nodes. Kubernetes will then handle the load balancing between the available cluster nodes.

Works like a charm.

8

Can you share your config.php lines for the redis cluster with the bitnami helm? Iā€™m trying to set exactly this up but my nextcloud canā€™t see my redis seeds.

9

Sure - hereā€™s the relevant snippet:

  'redis.cluster' => 
  array (
    'seeds' => 
    array (
      0 => 'cloud-redis-cluster-nextcloud:6379',
    ),
    'dbindex' => 0,
    'timeout' => 1.5,
    'password' => '<password>',
    'failover_mode' => 2,
  ),

The DNS name cloud-redis-cluster-nextcloud is the name of the Kubernetes service created by the Helm chart. You should use the ordinary service - not the headless that is created by the chart as well.

Depending on the name you chose for your Helm release, your DNS/service name will most likely be different.

10

Damn thatā€™s exactly what Iā€™ve tried and my nextcloud just keeps complaining it canā€™t see the seeds so Iā€™ve had to revert to standalone redis in the meantime. Iā€™ve made a support thread with my issues too.

11

Is your Redis running in the same namespace as Nextcloud?

Otherwise you need to add the namespace to the DNS name: You can refer to a service running in the same namespace using <servicename> alone as the short form, but if itā€™s running in another namespace you need the slightly longer <servicename>.<namespace> syntax.

12

It is. I made my own thread with logs here