Large scale environment

🏷️ General
1

HI,

I am going to configure a nextcloud in large scale environment, Where under a LB, 2 Next cloud web instances are running(Active/Active), each web server is connecting with share storage(NAS). I have configured one server,

All users data will be encrypted
SSO authentication is enabled.
User data will save on shared storage

now i am going to setup other web server, but i have following concerns
Can i sync nextcloud web root directory from 1 web server to another?
Is there any challanges in future, with this approach?
Or is there another method?

Here is nextcloud post, is there any official documentation

Thanks
Mudasar yasin

2

What do you mean by “large scale”? How many user accounts and how much storage are you planning to utilize?

Keep in mind that HA is different from scale.

Cheers,
budy

3

I want to make it HA and high scale as well.

Initially we will go with 5000 users and 5TB of space.

4

Well… I can’t vouch for HA, but performance-wise, this can be easily be served by a commodity server. We’re running 1200 users with (currently) 7TB of used storage and our server, doesn’t barely scratch the 10% load margin.

Okay, maybe not a low-level commodity one, but still… not a big deal, if NC is an important part of your infrastucture. You really should make your assessment, if a HA setup for a relatively simple setup, is worth the additinal overhead, introducing additional complexity to the setup.

NC really is a straightforward setup and is very well suited for single server setup and I’d rather invest some more time into a stable hardware setup than a fancy HA construct.

Just my 2c, though.

1 Like
5

buddy, i really appreciate your input.
NC will be tier 1 application for us, and we are looking to extent it to 15000 users.
I have study few case studies of HA and large scale environment,But unfortunately there are no technical details of these case studies.

I am preparing my notes and will share when it will be completed.

6

I remember, that NC said, that they’re running an instance for the Berlin University, which scales up to 20k users on one single server and I have bo doubt about that. Scaling further, they suggest to use federation with more servers, none of them HA though.

7

What do you mean “other web server”? Another Nextcloud application server? Or something else? Will you then have 2 or 3 Nextcloud Application servers?

Why would you want to do that? If you are referring to the data folder, this should reside on your NAS, where the storage is exposed via NFS or any service alike.

1 x HAProxy
2 x Nextcloud application servers
1 x Database server (depending on your number of users as @budy has stated
1 x Redis server (at least)
1 x NFS server to expose a common drive between two application servers
Preferably 1 x LDAP server to help with user provisioning and easing pain with user issues later on. You can combine LDAP with your SSO.

HTH.

1 Like
8

What do you mean “other web server”? Another Nextcloud application server? Or something else? Will you then have 2 or 3 Nextcloud Application servers?

Not another Nextcloud, single NC application, where multiple websevers under HA mode

Why would you want to do that? If you are referring to the data folder, this should reside on your NAS, where the storage is exposed via NFS or any service alike.

When i build new webserver, do i need to copy complete web root directory of NC or config.php is enough.

Your last part of question

On top we are using Netscaler LB
2 x Nextcloud application servers
2 x Database server(Master/Master)
1 x NFS server
1 x onlyoffice
SSO/SAML with AD
Currently Redis is running on web servers, so i am planing to separate and use my onlyoffice server.[Update: I have moved my Redis to separate machine]

9

Just from my own experience, NFS will be one bottleneck. Before I moved our NC instance onto out current host, I had NFS on SSD storage for storage and that introduced a lot of wait I/O, since NFS is especially not very good for interactive workloads, as NC is supposed to be.

File sync, and at the scale you are aiming at, really needs los latency disk I/O and you won’t get that with NFS. I am all bullish about NFS and I do use it extensively for my OracleVM cluster nodes, but it’s not a good choice for NC.

However, just my 2c, like always.

10

Ditto. If one has the capability and the means, try something like CEPH with Rados. Low latency, high throughput, cost-efficient and built in redundancy in the form of replication.

11

Hi Mudasar,
I’m also interested in having a look at your notes when ready.
My client want to know how much ressources it takes for a large scale environment: ~20K active users, 5GB / user.

12

You’re talking about a 100 PB storage (5 TB x 20000)? In this case, your client should have enough money to afford a support subscription from nextcloud.com.

2 Likes
13

+1.

The subscription will help a lot in making the right decisions and in keeping things running smoothly.

14

There is much content about large scale.

Check out:
http://www.cs3community.org/
Especially the 2014, 2016 and 2017 (that i present a large scale design bases on docker/k8s).

Other sources are the nextcloud conferences, enterprise track.

This might be also of interest for you:
https://www.360ict.nl/blog/my-nextcloud-scalability-session-online/

Its a bit old, but the post contains a sheet which shows the large scale installations of owncloud. At the tine, nextcloud was just founded, but you can substitute owncloud for nextcloud concerning the design.

The basic setup is for a monolith large setup:

  • n+1 webservers containing nextcloud php. Based on the load, anything fron 2-12.
  • galera mysql database setup, mostly 4 server setup with one write and multiple reads with a loadbalnancer (maxscale was popular, check out https://severalnines.com/ for some excellent galera blogposts.
  • some loadbalancers for the webservers. We use kemp, but haproxy still seems popular. But its a basic webserver loadbalancing.
  • some storage. Glusterfs was the official recommendation from owncloud at the time, but now everyone haa moved awat fron that. Nfs is a decent startpoint, but i think a bottleneck in the long run (again, depends on the usage of your users). Now i’ve seen Ceph more and more being used. We too use the block storage from ceph for the databases and the object storage for files. If you have any ha storage available, you probably can use that natively.

What i’ve seen is shat this scales until about 10-20.000 concurrent users, depending on the load/usage (just using the webui is offcourse much different then users syncing on mutiple desktops and autosaving pics fron their mobiles). The limit seems to be at the database layer, beyond 20k users the databases are just overloaded.

After the 20k users, Nextcloud has global scale as a solution. Basicly a setup like before but scaled horizontally, so lots of monolith setups. Global scale offers a directory service as a entry point to route you to the correct site.

We ourselves scale based on small setups, docker container instances of nextcloud including the database, orchestrated with kubernetes. Our load tests show 1000 users per container based on smashbox. As we abstract storage from compute, we can scale as much as Ceph can handle. And with CERN having a 10k storage node setup, im confident we can scale enough. The downside of Ceph is quite an initial investment to get the basic storage infra (8 servers minimum with objectstore i believe).

Anyway, lots of info. Let me know if you have any more questions. I typed it on a mobile, so forgive any weird sentences or typO’s.

15

grafik

16

~120tb for dell r740xd2 for about 35k eu as ceph osd nodes with 3-way copy (erasure coding could be more efficient)

17

Not to mention Intel/Supermicro 1U 1PB ruler drive units. Plenty of good non-vibrations!

18

Guys - bricelux said 5 GB per user, not TB, so only 100TB at all …

19

He edited it, initially it was 5TB/user.

20

Interesting, do you have a link?

I know dropbox uses 4u, 46 drive units as 1pb storage units.