How to Sync two NextCloud Server from two different site with INTERNET/VPN connection not always available?

🚧 Installation
1

SCENARIO: two NextCloud servers in two different site connected in VPN through internet connection that is not always available, users read and write only on the server of their site.

What can be the alternative to the procedure occ files:rescan to be able to synchronize also the data present only in the DB such as shares and permissions ?

I searched in the forum and on the internet but there is talk of successful configurations only for clusters (MariaDB Galera and SyncThing) and load-balancer that require installation in the same datacenter, instead in my case the problem to be solved is that the two servers are not always connected for varius internet lines problems (upload congestion, breakdowns, line derating, etc.).

For the replication of the MariaDB database I followed this guide:


and until errors like:
Last_SQL_Errno: 1062
Last_SQL_Error: Could not execute Write_rows_v1 event on table mndrive.oc_filecache; Duplicate entry ‘1004’ for key ‘PRIMARY’, Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event’s master log mysqld-bin.000001, end_log_pos 261338
everything works perfectly… (pity it doesn’t last long :frowning: )

The real problem is to manage the synchronization of the two systems immediately after being offline with each other, since each server in its headquarters has continued to work regularly and therefore the two DBs have become misaligned.

NC
NC695×844 95.9 KB

Best regards
GMR

2

Master master replica is troublesome if your not reading up on it enough and you can make sure that the apps table really can handle it without duplicates and so on.

Syncing file systems is the easy part here. The problem is the database.

Does the sites really need to work as one? Can federation be a good enough solution here?

One thing you also have to think about is the metadata. Should one file be sharable from two sites?

Are these hooked up to be able to share information to the internet? That’s also something to think about.

This is complex and should be thought on and do a study in what’s really really needed.

3

Hi SmallOne,
thanks for your interest, I’ll answer you between the lines:

Master master replica is troublesome if your not reading up on it enough and you can make sure that the apps table really can handle it without duplicates and so on.

Yes I realize, but at the moment I have not found any other specific indication from NextCloud to solve this scenario

Syncing file systems is the easy part here. The problem is the database.

Yes, the filesystems of the two sites synchronize perfectly and share exactly the same data space. Now the two DBs are not in replica and I run occ files:rescan to re-synchronize the new files but I need to synchronize the shares and other app info present only in the DB.

Does the sites really need to work as one? Can federation be a good enough solution here?

Yes, the sites can be considered as one, the organization is the same and the users are the same, in fact some users are physically on site A on some days and other days they are physically on site B.
I don’t know if the federation can help in this scenario my aim is to always have all the data and information in each site so that if the remote site is not available the local site can still work with the data present from the last one synchronization, if I’m not mistaken, NextCloud Federation is a mechanism for sharing data between the two servers but the data remains on each server and if one of them fails at that moment its data are not available, correct?

One thing you also have to think about is the metadata. Should one file be sharable from two sites?

Yes the data are indeed must be exactly the same

Are these hooked up to be able to share information to the internet? That’s also something to think about.

In reality, to simplify the scenario, I have not indicated other remote users who remotely access each site in VPN and therefore can be considered as local, but here the real problem is that the internet lines are not good and often block for this, I need a local self-consistent system.
In conclusion I cannot always rely on the internet and therefore I cannot even consider a cloud system.

This is complex and should be thought on and do a study in what’s really really needed.

Yes, I understood it because I thought it appropriate to open this discussion and try to understand first of all what would be the best minimum system to solve the problem and then how to implement it.

Thanks again for your interest and I hope that the topic also involves other users.
Best Regards
Gianmichele

4

How do you want to handle conflicts? You already see with the Nextcloud desktop client, the conflict management is not great (there you have the situation of a bad internet connection). If it is just an archive where files are not changed (only new ones added), there could be ways, perhaps do the syncing in the background by other programs.

All the multi-server setups in the old documentation supposes that all the servers have a good connected between each other.

For a fully supported version, you’d have to think about a write cache in case the connection is bad, and then when executing the changes you need to handle conflicts.

5

I won’t recommend to master<>master database setup on a bad/often down link too, because you will very likely run out of the binlogs and then you need to do a manual restore to the second server, and resync them. That is not a fun job, especially if you not proficient in that you likely lose data, especially if you update on both DB Servers at the same time.

However after reviewing your manual, there is a reason about the duplicate error.
I am doing a master master setup too and I have in the replication.cnf on the first server:

auto_increment_increment= 2
auto_increment_offset = 1

and on the second:

auto_increment_increment= 2
auto_increment_offset = 2

so the primary entries should not collide anymore, except mariadb itself is handling that with another parameter, I use that in mysql. However I would take a look into that direction.

6

Hi tflidd,
thanks for your interest, I’ll answer you between the lines:

To better manage the conflicts and therefore the alignment between the servers, I think the best way is to manage them at a high level NextCloud application and not at a low level directly between the DBs (for this reason I had indicated ???).
Do you know if there is any nextcloud application under development?
For example federation with copying also users data?

yes I read, in fact in this scenario the cluster / datacenter model should be optimized adapted to the problem of line interruptions but also of congestion and limited bands.
However, even in the same datacenter a server can get out of alignment due to blocks or breakages, so in any case it is necessary to manage the synchronization from a misalignment start point.

yes, in fact, as I told you, management at an application level would be appropriate.
does something already exist?
Is it possible that this problem has not already been addressed and solved by someone?

7

Hi dev0,
thanks for your interest, I’ll answer you between the lines:

What do you recommend to do to solve this scenario?

Yes, the solution of writing all the indexes of the tables with odd values ​​on one db and with even values ​​on the other t is a great idea to resolve conflicts up to a maximum of two master servers, you shiould think about how to do it even in the case of more than two.

I have already implemented and tested this solution with only two masters, the result is that the problem of conflicts in the various tables no longer occurs but as soon as one of the two DBs is restarted or remains disconnected from the other, the replication process stops with a example error:

Last_SQL_Errno: 1032
Last_SQL_Error: Could not execute Update_rows_v1 event on table nexcloud.oc_jobs; Can't find record in 'oc_jobs', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysqld-bin.000002, end_log_pos 55879

You have other suggestions?

8

Hi,

Yes I know that kind of error, in that case you “run out” of the bin logs, so the “good” server is not having the transaction information for this event anymore.
So you need to do a dump with the current master data, inset that to the “bad” server and resync the master data on both.

To reduce this, you need either increase the storage space of the bin logs, or get a better link.

If you want to use more then 2 DB Servers in your setup then, it will be very interesting, because that feature is afaik generally not meant to compensate a bad link, rather to avoid a service outage if one DB server is broken.

In my setup I use a haproxy and set it that way that the clients are preferring a specific server to the other.

9

I haven’t seen a public announcement like for other features (client-side encryption, virtual drive). A first step would be to improve the conflict handling of the clients first, there are still conflicts of files that haven’t been changed at all…

10

Dit you try to install the client on the server and let the client sync with the other server

11

Hi ManuelD, thanks for your interest.
Which linux client are you referring to?
My need is to synchronize all folders of all users between the two Nextcloud Linux servers including sharing information.
How would you configure the linux client to synchronize two Linux Nextcloud servers?

12

hello russi,
im facing the same problem did you solve ?

13

Hi mohd.jarmouzi,
unfortunately I haven’t solved it yet.
If you have had the opportunity to read the content of the topic you will have noticed that we have already tried the only things we could do without modifying the nextcloud software and unfortunately it is not enough.
I believe that development is needed to support the function, I don’t know if at the level of an additional plugin or even at the nextcloud core software level.

Always working externally to the nextcloud software, I made site B clone of site A in readonly for backup, exporting and re-importing the entire DB of site A into site B based on a scheduling time.