Originally published at: https://nextcloud.com/blog/85-faster-ldap-10x-readwrite-speed-of-windows-network-drive-welcome-to-nextcloud-13-beta/
We know, it is crazy, this is just the beta. You’ve already seen us publish a draft of our End-to-end Encryption feature coming to Nextcloud 13 and here is another peek into what is coming: SPEED. Boatloads.
Remember Nextcloud 11? This release resulted in the TU Berlin cutting their database load in half upon migration and while this performance advantage is yet unmatched, the upcoming 13 is slated to introduce its own slew of improvements. The story starts with LDAP returning user information over 85% faster in a torturous test we devised; and continues with improvements to our external storage capabilities giving system administrators employing php-smbclient for their Windows Network Drive a 10x speedup in sequential read/write to look forward to. There is more and we encourage you to read on but, perhaps most importantly, download the beta and give it some in depth testing! Open Source software is only as good as its numerous contributors and no test matrix can replicate what you lot throw at our software every single day. Testing Nextcloud 13 Beta 1 against your unique requirements is what will ensure its ultimate stability and usability, so give it a good run!
First awesomeness: LDAP
For Nextcloud 13, we'll introduce a small but very nice feature: auto-complete of user names in the comments field. Here's a video for you to enjoy the results, made by Arthur, its principal author.For this feature to work well, it is important to get a list of matching users to the browser as quickly as possible. On enterprise installations, this typically means going through the LDAP code in Nextcloud, ultimately checking with the LDAP server what users match the name you’re typing. You can probably imagine that with 150.000 users, this isn’t as snappy as on your home server with 5 users!
Consequently, this area needed work. A torture test was devised: setup a server with 150.000 users and ask it for users 1001 to 1500 that match the characters “ha”. This took a whole 15.2 seconds before the work started, making auto-complete in the share or comments dialogs less than fun. However, after the work was done, reviewed and merged, this same action took no more than 1.8 seconds! This will only work if the users have been fetched before at least once but on a running, busy, active server, the impact of this change should be noticeable very quickly.
Stay tuned for a post about the work on this by Arthur on his blog!
Besides some small improvements (like the addition of a save button for LDAP user credentials to avoid issues with browser auto-complete), another major change was integrated: a plugin mechanism which allows users to extend the Nextcloud LDAP features. Normally, Nextcloud does not require an LDAP server to give it write access, something security-conscious administrators appreciate. But sometimes it would just be darn useful to manage LDAP users from within Nextcloud and with this plugin structure, this would now be possible. An application to do some of this work was developed by the contributor of this code and while it is specific for a certain LDAP structure its code can certainly serve as an example for others.
Storage, Encryption and more
Our external storage code has seen optimizations as well. The load on the database was reduced for all external storage folders and as mentioned before, systems using php-smbclient to provide access to a Windows Network Drive or Samba share will see an improved sequential read/write speed of up to 10x! S3 external storage reduced its CPU load by 50% and its execution time by 73% (that means "it is almost 4x as fast with small files"), in addition to adding the ability to handle large files. It was previously limited to files about 5GB in size.Our Server-side Encryption has been sped up as well, at least on new installations. Until now, per-user keys were used by default and administrators could enable the master key. This is now enabled by default, resulting in a significant speedup, while also working better when users are provisioned over LDAP and change their passwords. The change helps especially with large folders and makes group sharing instant. Login resets also work faster and easier. We recommend administrators to switch from per-user to master key encryption when enabling encryption to benefit from this performance improvement as well. Unfortunately existing installations can not switch over without disabling encryption, decrypting files, and re-enabling encryption.
There is more: search was improved, especially for large searches on large servers, reducing database load and runtime. The Contacts app loads large numbers of contacts much faster and deferred script loading in the browser was shown to decrease time it takes for the first page loading our files app by more than 50%!
