Note that Nextcloud doesnât support a master-master on database level, this can lead to nasty problems. Weâre usually recommending using something like MaxScale to split read/writes to different databases.
Oh full stop? Rough.
The idea was straightforward - if someone uploads a doc in Amsterdam it updates all databases, so someone in UK sees it on their node instantenously. Similarly UK to Amsterdam. This way a master db can sit alongside each node, they all stay in sync whoever does a write from whatever node and winner-winner chicken dinner.
Doing a master-slave setup would be OK in terms of reads (slave with each node), but for writes the latency will grow the further away it is, obviouslyâŠ
@LukasReschke whatâs the resolution in this case? Donât say Global Scale (yet).[quote=âguidtz, post:17, topic:12863, full:trueâ]
Very interesting.
Do you try it with mariadb 10.1 wich integrate master-master gallera ?
And what do you use for data sync ?
Regards
Guidtz
[/quote]
Galera master-master is in place and working fine with 6 web nodes at the moment, each node that does a write is directed via haproxy to any master and the others replicate immediately. Itâs beautiful.
Data sync is syncthing, which Iâve now setup to basically replicate the whole nextcloud
install folder, not just data. I did this for two reasons:
- The nodes are, and should remain identical
- I wanted to upgrade to 12 without doing it on each of the 6 nodes manually given the db will only be upgraded once. The upgrade of all 6 nodes was incredible; I left it in maintenance mode while it synced, and had replicated from the upgraded node to all the others within about 3 minutes, before then taking it out of maintenance mode and it all working perfectly.
Good very good. I use also master-master with haproxy in front in another projet and Itâs works fine and add a new mariadb master is so simply.
What is âData syncâ ? I try glusterfs, lsyncd, drbd for found a a multidirection data sync. So I want only opensource solutions.
Problem, glusterfs and drbd make storage slower. And lsync is not done to bi-directionnal sync.
Regards
Try SyncThing. FOSS and works well.
Ok I note to try it
Thanks to everyone whoâs taken an interest in this, with master-master not supported I donât see a way of expanding this beyond a local network PoC and so will spin it all down.
What Iâve learned:
-
SyncThing is super reliable, handles thousands upon thousands of files and has the flexibility to be used in a usecase such as this; full webapp replication between several servers without any fuss however:
-
Feasibility in a distributed deployment model where latency and potentially flaky connections wasnât tested thoroughly, and while the version management was spot on at picking up on out of sync files, it would need far more testing before being even remotely considered for a larger deployment
-
Load on the server can be excessive if left setup in a default state, however intentionally reducing the cap for the processes will obviously impact sync speed and reliability
-
-
Data/app replication between sites is not enough if you have multiple nodes sharing a single database
-
The first few attempts at a NC upgrade for example led to issues with every other node once the first had a) updated itself and b) made changes to the database.
- This may be rectified by undertaking upgrades on each node individually but stopping at the DB step, opting to only run that on the one, I didnât test this though.
-
I think a common assumption is you only need to replicate /data, however what happens if an admin adds an app on one node? It doesnât show up on the others. Same for themes.
-
-
Galera is incredible
-
The way it instantly replicates, fails over and recovers, particularly combined with HAProxy which can instantly see when a DB is down and divert, is so silky smooth I couldnât believe it.
-
Although a powercut entirely wiped Galera out and I had to build it again, this wouldnât happen in a distributed scenario⊠despite the cluster failure I was still able to extract the database and start again with little fuss anyway.
-
The master-master configuration is not compatible with NC, so at best youâll have a master-slave(n) configuration, where all nodes have to write to the one master no matter where in the world it might be located. Another solution for multimaster is needed in order for nodes to be able to work seamlessly as if theyâre the primary at all times.
-
-
Remote session storage is a thing, and NC needs it if using multiple nodes behind a floating FQDN
-
Otherwise refreshing the page could take you to another node and your session would be nonexistent. Redis was a piece of cake to setup on a dedicated node (though it could also live on an existing node) and handled the sessions fine.
-
It doesnât seem to scale well though, with documentation suggesting VPN or tunneling to gain access to each Redis node in a Redis âclusterâ (not a real cluster) as authentication is plaintext or nothing, and thatâs bad if youâre considering publishing it to the internet for nodes to connect to.
-
-
When working with multiple nodes, timing is everything.
-
I undertook upgrades that synced across all nodes without any further input past kicking it off on the first node; however donât ever expect to be able to take the node out of maintenance mode in a full-sync environment until all nodes have successfully synced, otherwise some nodes will sit in a broken state until complete
-
Upgrading/installing apps/other sync related stuff takes quite a bit longer, but status pages of SyncThing or another distributed storage/sync solution will keep you updated on sync progress
-
-
The failover game must be strong
- By default HAProxy wonât check nodes quickly enough for my liking, meaning if a node suddenly fails and I refresh, it may result in an error. I found the following backend setting resulted in faster checks and was more ruthless about when a node comes back up (requires 2 successful responses to come back into the cluster, but only on fail to drop out):
server web1 10.10.20.1:80 check fall 1 rise 2
In summary, it was a good experiment and I learned a few things. Although I faced no issue with Galera in master-master I understand it could lead to problems down the road and itâs therefore not something Iâd invest any more time into. I will however use some of the things I learned here to improve other things I manage and perhaps will be able to revisit this in the future if Global Scale flops ( jokes)
If anyone does wish to have a poke around, check performance and whathaveyou, by all means feel free to do so for the rest of today, this evening itâs all being destroyed
Thanks particularly to @LukasReschke for your advice!
Edit: All now shut down.
Wrote it up in a bit more detail here:
Wow, very very useful stuff here, many thanks @JasonBayton
Iâm working with a NC12 installation with 40 users not very active.
At the moment I have a single instance of NC+DB+Redis but the Virtual Machines involved (3) are already clustered. Indeed I have the following already in-place:
- MariaDB 10.1 galera master-master (2 nodes + 1 arbiter)
- GlusterFS, 2 nodes
- 2 HAproxy with floating IP, all configured as active-backup
Iâm not interested on balancing, but only on the failover features: I need to shutdown a node without users noticing it.
I have a couple of questions:
- (because of the âREAD-COMMITTEDâ ) Am I in a dangerous situation even with 1 NC instance pointing to my master-master clustered DB through HAProxy in an active-backup fashion (all queries goes to the active DB, the secondary master is not used until the first one goes down)?
- Having glusterfs in place, adding a second NC instance (application server) with the same gluster volume for the entire nextcloud webroot is feasible?
- Redis: my nodes are on private network, do you see any issue using a clustered redis here?
Many thanks
Yes. Youâll need to migrate away from this ASAP.[quote=âaventrax, post:25, topic:12863â]
2) Having glusterfs in place, adding a second NC instance (application server) with the same gluster volume for the entire nextcloud webroot is feasible?
[/quote]
Worked for me perfectly well over a few-week period. It made the most sense given the nodes share the same database, why shouldnât they all act like one server? However, longer-term testing (and bullet-proof backups) is absolutely required. What I did was a PoC, not a recommendation.
Nope, in fact that seems to be how it was designed. Optionally make use of the authentication requirement to deter other users on the LAN from snooping, even if it is stored plaintext it can be a deterrent.
Ok. Iâm again on a single MariaDB instance.
Iâm now wondering how to reach my goal: being able to shutting down the master database with nobody noticing it.
I searched and I havenât found anything like galera master-slave setup, it seems that galera is master-master ONLY, am I right?
Searching another master-slave setup I found a lot of documentation but Iâm still asking myself whatâs the simplest master-slave setup with an automatic failover.
It seems to me that MariaDB have a tool called âreplication managerâ that can be installed on a third host and can provide a master switchover or failover. Not bad, but not enought as I understood.
The remaining piece of my puzzle is something that can detect which node has the master role and point all the writes to it, and I thought it can be maxscale with a floating IP running with keepalived.
It seems to me âa bitâ complex, I had configured a galera master-master setup in minutes, and for a less âadvancedâ master-slave setup it seems weird to have all this software required (replication manager, maxscale, keepalivedâŠ), isnât it?
Am I missing something?
No indeed, the NC docs repeatedly talk about Galera but in a master-slave capacity for which there doesnât appear to be a valid configuration option.
Master-slave with failover is equally not well documented from what I can see online, though the tools you mention will help.
@LukasReschke @nickvergessen @jospoortvliet can any of you shed light on this? Why is Galera mentioned in the docs if master-master seems to be the only valid config and NC doesnât support it? Will NC support it in the future? [quote=âaventrax, post:27, topic:12863â]
Am I missing something?
[/quote]
No, Galera is frighteningly simple to set up.
I believe a working setup would require âa SQL-Proxy for read/write splittingâ - I know the TU berlin has that as this is their description of their setup:
tubIT employs a setup with F5 Big IP Load Balancers ahead of a number of web application servers. Data is stored on a GPFS storage system. The database is on a Galera MySQL cluster using a SQL-Proxy for read/write splitting, in LXD containers managed with OpenStack. File locking and session management is provided using Redis, also on LXD managed with OpenStack. Authentication is handled through LDAP and Kerberos.
The TU currently uses 4 application servers running a LNMP (Linux/NGINX/MySQL/PHP) stack with PHP 5.6 on CentOS 7.3. Further application servers are running the DFN-Cloud instances. A locally replicated LDAP instance runs on each of the application servers. Each server has about 16 virtual CPU cores with 18GB RAM.
The Galera cluster employs 3 container with 32-core cpus and 128GB ram each. Its performance is very critical to the performance of the overall cloud. To serve more incoming queries the number of instances can be increased, however every new instance increases the resources required to synchronize within the cluster, limiting scalability.
The GPFS cluster is load balanced between four GSS nodes to improve response times and provide failover.
(text from a case study Iâm working on, hopefully coming out in a week or two)
I put it between ââ as I have no idea what Iâm even saying but perhaps Jason does
The SSL proxy is fine and I understand why youâd use one for a clustered system (the deployment recommendations also suggest using one). However there again Galera is mentioned. The only Galera Iâve seen is master-master which apparently both is and isnât supported due to:
Then:
Master-Slave doesnât appear to be available for Galera, but then also:
Emphasis mine. If Galera is master-master only, what are the docs referring to?
Honestly, Iâm lost, I donât know the details here. @MorrisJobke I know youâre on holiday but when youâre back perhaps you can enlighten Jason
Note that this whole Galera stuff is of course for larger installations so our docs might not be super up to date - we have some internal docs we share with customers.
Can I get those please? Iâd love to get a PoC v.2 up and running on solid foundations. I can sign an NDA (on the docs, obviously methods would be revealed on a documented build) if required.
I honestly donât know exactly where they are or even if itâs mostly in peopleâs head - Iâm entirely not involved in that stuff, I just know we help customers with this and assume it is written down. Really would need Morris or maybe @MariusBluem to chip in
For the read-write split you need a MaxScale proxy And the galera cluster needs to be in master-slave mode with the masters receiving the writes and the slave(s) the reads.
Does this help you?
No sorry Galera is master-master and explicitly states that in the (MariaDB) Galera docs, so I donât understand where the master-slave piece comes into it.
Ah right - it is master-master, but we need to make it a somehow master-slave (slave as in âhandles only readsâ) by the MaxScale proxy.