Nextcloud faster than ever! Introducing Files High Performance Back-end

Originally published at: Nextcloud faster than ever! Introducing Files High Performance Back-end – Nextcloud

At a virtual presentation streamed worldwide, the Nextcloud team introduced the availability of Nextcloud Hub 21.

Nextcloud Hub 21 brings a wide range of improvements in file handling performance with Files High Performance Back-end and important new collaboration features in Text, Talk, Groupware and Files. In this blog, we go into a bit more detail of the performance improvements.

For Nextcloud 21, we wanted to do something that benefits all users: make it faster.

To increase the capacity of especially large installations and keep Nextcloud snappy, our team profiled and optimized various core areas of Nextcloud. We made all kinds of improvements and you can watch our release video above to get some more details.

With the COVID crisis continuing to force employees to work from home, Nextcloud server administrators need to keep adding capacity. Nextcloud 21 will reduce the load on servers, enabling more employees to work without requiring additional hardware. At the same time, the new collaboration features improve the day to day productivity and support team members in getting more work done in less time.

Frank Karlitschek, CEO and founder of Nextcloud GmbH

We did a number of improvements that reduce the amount of code loaded on each page, for example handling the translations and theming more efficiently. We also worked hard to optimize our use of the database, as this is an important limit for large deployments. Many common database queries were reduced or even outright eliminated. The move to PHP 8 will also give a nice performance boost for systems that have it installed!

Besides this generic performance work, we also looked at some specific scenarios where servers would run into performance limitations. Nextcloud Text, widely used in many organizations for note taking and basic writing, creates load on the server by checking in for changes constantly. We reduced this load by 25% and introduced a ‘idle mode’ where a document not used for 30 minutes stops polling the server. We also optimized object storage handling, made loading the trash view twice as fast, optimized LDAP, installation and upgrading, the app store and much more.

High Performance Back-end 🚀

The performance improvements above should impact nearly every server, giving a better user experience and enabling system administrators to grow the user base without having to add more hardware. But there is another, big change that we have been working on, and it requires a little background.

Nextcloud offers near-perfect scalability

By using the traditional LAMP stack with Linux, Apache, Mysql and PHP, Nextcloud builds on some of the most popular technologies on the web. This has many benefits. Our platform is easy to deploy and maintain and scales extremely well. The scaling, in particular, is a huge benefit of PHP over practically every other technology out there: to double the performance of a PHP server, you can simply have twice as many cores and things will work without any changes needed. Not only is the scaling you get pretty much 100%, unlike other languages, this scaling continues beyond a single server! You can put a second server next to it and let it handle requests. This is possible because PHP handles each request entirely independently from every other request. The result is that it is easy to create a cluster that offers Nextcloud to hundreds of thousands of users on a single instance.

The scalability of Nextcloud itself is thus near-infinite, but file systems and databases are not. The improvements we did in this release will help to scale further, but inevitably they will become the limiting factor. That is why we invented Global Scale! This innovation allows multi-continental deployments with hundreds of millions of users. We already deployed the first Global Scale setup in early 2018 and our largest customers use it with tens of millions of users today. Scalability is certainly not something Nextcloud lacks!

Taking advantage of new technologies

But every technical choice has two sides to the coin, and some emerging technologies on the internet provide other, unique benefits. To make sure our users always get the best of both worlds, Nextcloud 21 introduces a new, completely optional component: our High Performance Back-end (HPB) for Files.

The new HPB is a binary written in Rust and enables Nextcloud to keep a permanent connection open to our clients. So what impact does this have?

Faster notifications and syncing

First, you no longer have to wait! Currently, the client checks every 30 seconds to see if there have been changes on the server. With the open connection, the server notifies the client of changes and it can begin downloading new files right away. So when a colleague asks “did you get the file I just shared”, you no longer have to say: “wait, it just started syncing!”

This does not only impact file syncing, however. It also means you get instantly notified of new chat messages in Talk or an incoming call. New files shared with you. A colleague commenting on a file. A Deck card running into its deadline. And much more! In many of these cases, the faster response time makes a real difference.

90% less load from polling

Server administrators will also be delighted. One check every 30 seconds equates to 2880 times per day. We reduced this number by 90%, only checking in once per five minutes.

To give you an idea of the impact of this change, we did an analysis of traffic at the German DFN Cloud, with more than 3 dozen educational and research institutes managed by the TU Berlin. In a 15 minute period, more than 250.000 requests are made by desktop clients just asking if there were any new files to download! This accounted for over 2/3rd of their network connections. You can imagine the impact of reducing that traffic by 90%. And that is just the desktop client – we also implemented this in the web interface, which also checks every 30 seconds! A user with 3 tabs open is polling the server 3 times every 30 seconds, 10 tabs means a ping every 3 seconds, all day long. You can imagine what impact it can have to reduce this by 90%.

At a scale of millions of users, milliseconds start to add up. After analyzing application and caching server, storage and database behavior, our team was able to significantly reduce the impact of common operations. The High Performance Back-end, on the other hand, provides a completely new way of reducing server load while bringing a new level of responsiveness to users.

— Roeland Douma, Server Engineering and Support Lead

The HPB for Files requires a reverse-proxy setup and Redis, among other pieces of infrastructure. You can find instructions here.

Read the full announcement about Nextcloud Hub 21 features and improvements!

6 Likes

Is there any advice on configuring preload, given the significant performance improvement shown in the graphs above?

My understanding is that this needs a file containing opcache_compile_file() for each file to be preloaded, but I don’t see anything similar in the nextcloud-21.zip, or previous versions. I also can’t see any docs on the admin manual.

Thanks!

2 Likes

Here are the full instructions

1 Like

Hi @just, I believe you were replying to me and not the original post given the message in my email.

Thanks for that documentation but notify_push and php preload are different things. I’m looking for information on the latter.

I had a lot of trouble configuring this thing.
All (notify_push daemon, apache, nextcloud, redis) is running on a single server. I configured apache as a reverse proxy for the push daemon as suggested in the instructions. But notify_push:setup failed when checking for trusted proxies. And it was not clear to me what was meant here.
I tried adding the local IP to the trusted_proxies in nextcloud config, but to no avail.
Turned out I had to enable the apache remoteip module and add

RemoteIPHeader X-Forwarded-For
RemoteIPTrustedProxy <notify_push IP>

to the Apache config.
Now even after removing the IP again from the nextcloud config trusted_proxies, it still works.
So I assume this is what that error meant: add the push client to the trusted proxies of the apache server, not the nextcloud config. I think this should be made more obvious in the documentation. Certainly the aforementioned instructions should mention this.

4 Likes

To enjoy the new HPB backend, have we only update to v21.0.0 and install and setup Notify Push ?

I found this tutorial in German from the great dev “Decatec”: Nextcloud 21: High Performance Backend für Dateien » DecaTec

Further question on this:

  • Of course only clients that explicitly added support for the push feature will benefit. Do all clients, Windows/Linux/macOS, Android, iOS support it already?
  • Do clients get the info on initial server connection, whether they get push notifications or how does it work?
  • Above it’s mentioned that polling intervals are raised from 30 seconds to 5 minutes. I guess this is only the case when the clients know that notify_push is available and they hence get push notification, right? Do clients automatically fallback to lower polling intervals when the push connection fails?
  • It’s only about server notifying clients about changes, not the other way round, right? I though I heard something about clients informing the server about changes faster as well, during the announcements video stream, but I think I misunderstood that.
1 Like

@RobinR1 Thank you for your suggestion, as it helped me get this problem sorted. However, upon re-opening the Nextcloud config file, I noticed that a second ‘trusted_proxies’ entry was added at the bottom of my configuration file and that appeared to overwrite my manual entry.

I just added the correct ip_address to that array at the bottom, and removed the additional configuration you have proposed, and everything is working.

1 Like