How to: make an HTTP3 Nextcloud on Openlitespeed and saturate your internet line

Here’s a quick summary:

Stack:

  • Debian
  • Openlitespeed
  • MariaDB
  • PHP 8.1

Basically, it’s the LAMP stack, but on Openlitespeed instead of Apache.

  • Update packages
  • Install
    • Nextcloud dependencies
    • Mariadb Server
    • Certbot
    • Redis server
  • Add Openlitespeed repo
  • Install
    • Openlitespeed
    • Openlitespeed PHP
    • Openlitespeed PHP modules
  • Edit PHP .ini for large file transfers:
    • max_execution_time = 3600
    • memory_limit = 1024
    • sys_temp_dir = “/usr/local/lsws/nc_tmp” (temp dir must be on same drive as Nextcloud files)
    • output_buffering = 0
    • apc.enable_cli = 1
  • Set admin password for Openlitespeed
    • /usr/local/lsws/admin/misc/admpass.sh
  • Make folder structure for your Nextcloud Virtual Host
  • Log in to Openlitespeed management interface:
    • Server Configuration:
      • External App - change PHP to version 8.1
    • Virtual Host:
      • Add Nextcloud Virtual Host:
        • Basic:
          • Name: Nextcloud
          • Root: Nextcloud
          • Config file: conf/vhosts/$VH_NAME/vhconf.conf
        • General:
          • Document Root: $VH_ROOT/html
        • Rewrite:
          • Enable Rewrite: yes
          • Auto load from .htaccess: yes
        • Context:
          • 1: Static, /data, Accesible = No
          • 1: Static, /config, Accesible = No
    • Listeners
      • Default:
        • General:
          • IP Address = ANY
          • Port = 80
          • Secure = No
          • Virtual host Mappings = Nextcloud, Domains = *
      • HTTPS:
        • General:
          • IP Address = ANY
          • Port = 443
          • Secure = Yes
          • Virtual host Mappings = Nextcloud, Domains = *
        • SSL:
          • Private Key File: /etc/letsencrypt/live//privkey.pem
          • Certificate File: /etc/letsencrypt/live//fullchain.pem
          • Chained Certificate = Yes
  • Run certbot and get your certs (HTTP3 does not work with self-signed I believe)
  • Run mysql_secure_installation
  • Add Nextcloud user to MariaDB
  • Install Nextcloud:
    • Download to /usr/local/lsws/Nextcloud/html
    • Change owner to nobody:nogroup (Openlitespeed’s user)
    • Run installer
  • Use OCC to configure for large files:
    • occ config:app:set files max_chunk_size --value 0
  • Configure crontab for user ‘nobody’
    • */5 * * * * /usr/local/lsws/lsphp81/bin/lsphp -f /usr/local/lsws/Nextcloud/html/cron.php
    • Set Nextcloud to use cron for background jobs

By now, it should be working. I can post some more regarding Redis for locking and APCu for memcache another time.

Hope to hear from you if you try this out! If there is some interest in this, I can post a more detailed walkthrough and a half-finished bash script.

5 Likes

Hello,
I will ask for a detailed guide and even a partial script I want to test.

Sure thing. I have a busy week ahead but will put something together when I get a chance.

Thanks for the guide - as much as I like Nextcloud, no matter what hardware I have thrown at it, have not been able to get great speed out of it - I think version 25 seems even slower ( must be the round corners :grinning: ) Did you have a chance to do any benchmarks ?

It will be interesting to see if http/3 over UDP helps things like file uploads or downloads - I have tested php and php-fpm versions 7 and 8 and http/2 - nothing seems to improve the speed to the point where it makes a big difference. Without detail tracing it is hard to know if the bottleneck is in php or the sql queries on large installs.

For anybody that wants to see how really slow Nextcloud is for file upload or download, try to scp or sftp a large file ( multi GB ) between the client and the server, and then try using Nextcloud to upload or download the same file. To get a feel for the speed of your connection here is a good command:

time cat large_file | ssh user@nextcloud_host ‘dd bs=1M status=progress > /dev/null’ [1]

If large_file size is smaller than your Ram, if your re-run the above command you may get even better performance as the above will be testing your disk read speed and the network performance ( or use /dev/zero as the source of data in place of large_file with dd if=/dev/zero bs=1M count=X to avoid any disk read).

We also use iperf3 but find coping a large file to be a more realistic test and most servers we deal with have sshd installed. One additional note - the size of large_file or value of X when using /dev/zero should be adjusted so that tests take at least 15 sec or more. Also when using ssh or scp you may want to test with -o compression=no or =yes - if you are CPU bound no should be faster.

For the actual file transfer the command would be:

time cat large_file | ssh user@nextcloud_host ‘dd bs=1M status=progress > large_file’ [1]

or use the scp command.

For a quick test of a local system, sysbench is a nice tool and is available on most Linux distros.

1. This tests the upload network speed or file transfer - reverse the commands to test download if your connection speed is not symmetric.

Detailed Openlitespeed Setup guide

Containing plenty of copy-pastable commands and handy screenshots.

Update packages


apt update

apt upgrade -y

Install necessary packages


apt install -y sudo

Install Nextcloud Dependencies


apt install -y gnupg2 mariadb-server apt-transport-https ffmpeg libreoffice imagemagick ghostscript software-properties-common

Install certbot for SSL


apt install -y certbot

Install Redis for locking


apt install -y redis-server

Install Openlitespeed


wget -O - http://rpms.litespeedtech.com/debian/enable_lst_debian_repo.sh | bash

apt install -y openlitespeed

Install PHP


apt install -y lsphp81

apt install -y lsphp81-apcu lsphp81-common lsphp81-curl lsphp81-imagick lsphp81-intl lsphp81-ldap lsphp81-mysql lsphp81-opcache lsphp81-redis

Set up PHP for large file transfers


php_loc8="/usr/local/lsws/lsphp81/etc/php/8.1/litespeed/php.ini"

mods_loc8="/usr/local/lsws/lsphp81/etc/php/8.1/mods-available"

sed -i 's/^\(max_execution_time = \?\).*$/\13600/' "$php_loc8"

sed -i 's/^memory_limit =.*$/memory_limit = 1024M/' "$php_loc8"

sed -i "s/^output_buffering = .*/output_buffering = 0/" "$php_loc8"

echo "apc.enable_cli = 1" >> "$php_loc8"

echo "apc.enable_cli = 1" >> "$mods_loc8/40-apcu.ini"

Make Nextcloud Virtual Host folder structure


ncvhost_name="Nextcloud"

ncvhost_root="/usr/local/lsws/$ncvhost_name"

mkdir "$ncvhost_root"

mkdir "$ncvhost_root"/{conf,html,logs}

chown -R lsadm:lsadm "$ncvhost_root/conf"

chown -R nobody:nogroup "$ncvhost_root/html"

Make Nextcloud Virtual Host config file


ncvhost_name="Nextcloud"

ncvhost_conf="/usr/local/lsws/conf/vhosts/$ncvhost_name"

mkdir "$ncvhost_conf"

chown lsadm:lsadm "$ncvhost_conf"

touch "$ncvhost_conf"/vhconf.conf

chown lsadm:lsadm "$ncvhost_conf"/vhconf.conf

chmod 750 "$ncvhost_conf"/vhconf.conf

Set admin password for Openlitespeed


/usr/local/lsws/bin/lswsctrl start

/usr/local/lsws/admin/misc/admpass.sh

Now you can log in to the Openlitespeed dashboard and configure in the GUI:

  • External App (PHP binary to be used)

  • Virtual Host

  • HTTP & HTTPS Listeners

External App (PHP binary)

Edit SAPI App:

Change command to lsphp81:

Virtual Host

Make a ‘Nextcloud’ Virtual Host:

Set up Virtual Host:

For security, make data and config inaccessible (this mimics htaccess configs in NC on Apache)

Listeners

One listener for HTTP, one for HTTPS:

HTTP is port 80, mapped to Nextcloud

HTTPS is port 443, mapped to Nextcloud

The HTTPS Listener contains the file paths to the certs we will get from Certbot


(Make sure to have plain HTTP working for certbot. We will redirect all HTTP traffic to HTTPS later)

Get cert from LetsEncrypt

certbot certonly

Adjust Virtual Host settings here to use HTTPS only

Now we have got our cert, we can redirect all HTTP traffic to HTTPS as well as enabling htaccess (which Nextcloud needs)

Download Nextcloud

ncvhost_root="/usr/local/lsws/Nextcloud"

ncdl="https://download.nextcloud.com/server/releases/latest.tar.bz2"

ncsha="https://download.nextcloud.com/server/releases/latest.tar.bz2.sha256"

latest="latest.tar.bz2"

wget "$ncdl" -P "$ncvhost_root/html/"

wget "$ncsha" -P "$ncvhost_root/html/"

cd "$ncvhost_root/html" || exit 1

sha256sum -c "${latest}.sha256" < "$latest"

tar -xjf "$latest" --directory "$ncvhost_root/html/"

rm $latest*

shopt -s dotglob

mv nextcloud/* ./

rmdir nextcloud

# Litespeed runs as nobody

chown -R nobody:nogroup "$ncvhost_root/html"

Enable OCC command

php_bin="/usr/local/lsws/lsphp81/bin/php"

ncvhost_root="/usr/local/lsws/Nextcloud"

echo "

alias occ='sudo -u nobody ${php_bin} ${ncvhost_root}/html/occ'

" >> ~/.bashrc

source ~/.bashrc

Setup mariadb


mysql_secure_installation

Create Nextcloud database and user

Optional: use apg here to generate a strong password


apt install -y apg

nc_db_pass=$(apg -a 1 -M ncl -n 1 -m 16)

Create Nextcloud database and user


nc_db_user="nextcloud"

nc_db_db="nextcloud"

# If you didn't use apg

nc_db_pass=ql1zGBf2DlEKi2nv

echo "

CREATE USER '$nc_db_user'@'localhost' IDENTIFIED BY 'ql1zGBf2DlEKi2nv';

CREATE DATABASE IF NOT EXISTS $nc_db_db CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;

GRANT ALL ON $nc_db_db.* TO '$nc_db_user'@'localhost' with grant option;

FLUSH PRIVILEGES;

" | mysql -u root

Install Nextcloud

Navigate to GUI and run installer

Speed things up in Nextcloud


# Disable rich workspaces

occ config:app:set text workspace_available --value=0

# Turn off upload chunking

occ config:app:set files max_chunk_size --value 0

NOT SHOWN: Customize Nextcloud config.php

Setup Nextcloud to use cron for background tasks


ncvhost_root="/usr/local/lsws/Nextcloud"

echo "

*/5 * * * * /usr/local/lsws/lsphp81/bin/lsphp -f ${ncvhost_root}/html/cron.php

0 7 * * * /usr/local/lsws/lsphp81/bin/lsphp ${ncvhost_root}/html/occ trashbin:cleanup --all-users

*/45 * * * * /usr/local/lsws/lsphp81/bin/lsphp ${ncvhost_root}/html/occ files:scan --all

0 5 * * * /usr/local/lsws/lsphp81/bin/lsphp ${ncvhost_root}/html/updater/updater.phar --no-interaction

" | crontab -u nobody -

occ background:cron

By now, your Nextcloud should be good enough for testing.

NOT SHOWN: configure Redis for file locking and APCU for caching.

2 Likes

Hi @bluesky, thanks for checking out the post. I hope more people can get this working.

While I didn’t do formal benchmarks, I certainly did tests and comparisons. I am using pretty good hardware, and have a small number of users with large files. I’m not using any of the extra apps or collaboration, and my users and I love the simplicity of just having the files app - the only thing left is the speed of the file transfers.

My current Apache HTTP2 setup is working well, and I am sticking with it for now because most of my collaborators have a slower internet speed than me and so I am already making full use of their connections. For the couple of people with really fast internet, using HTTP3 I found about a 20% increase in download speed (70MB/s → 100MB/S) and a 3-4x increase in upload speed (25 MB/S → 75 MB/S). So for large files on high spec Nextcloud servers with >1Gb/S Internet, I think running HTTP3 could be a serious boon.

There were some recent vulnerabilities in Litespeed. If trying out this setup, please make sure your version of Litespeed is up to date.

https://blog.litespeedtech.com/2022/11/14/cve-2022-0072-cve-2022-0073-and-cve-2022-0074/

1 Like

Wow - just wow - fantastic guide. With this info anybody should be able to set it up. Your speed up numbers look very interesting. Your original post got me interested in other possible options to replace Apache - I have not setup any experiments as yet but plan to test out the following:

ols with their php-fpm
nginx with php-fpm
haproxy connected directly to php-fpm
caddy with php-fpm

One things that bugs me a bit with ols is that you just can’t not download pure web server - even when I select lsphp8.1 they still pull in lsphp7.4 and other junk - I am using very small lxc type containers and for security keep the software installed to the minimum. Trying to instal ols brings in about 200MB of stuff with it, compared to the Debian11 container being only 400MB for everything else :frowning:

I have checked the current status of http/3 and looks like out of the box ols is the only one that has it - some concerns expressed on-line were that LightSpeed is in fact a Chinese company. Also they run a benchmark site, that they using for their speed comparisons - I have not looked at that in detail - might not be an issue.
In addition because I have firewall rules in both directions noticed connections to 104.131.32.114 on regular bases - either some form of tracking or check for updates - we do all software updates using the OS tools and do not allow applications to update them self ( this has been the nice thing about Linux - Window has been able to update the OS files but not all the extra packages, whereas Linux installs and updates all packages using yum,apt, etc ).

Now , back to the subject of Nextcloud and getting the max out of it - I have high hopes for haproxy v2.5 that came out recently - it now supports http/3 quic - part of the issue for any of the servers that don’t support .htaccess is to rewrite the access rules - speaking of that, the best practice in security is to block everything and then allow as needed - Nextcloud seems to use selective blocking after allowing everything.

Googling is not showing many efforts in using non Apache server - some notes for Nginx and now this guide for ols (OpenLiteSpeed) - I really hope Nextcloud itself starts doing this and moves away from Apache - unless they believe it has not been the bottleneck. Your tests seem to prove that switching to http/3 might have large benefits for dealing with large files. Again, thanks for your work.

Look forward to hearing how your tests go!

Thank you, @Doozy, for your time and detailed post! I followed your instructions and successfully set up OpenLiteSpeed + Nextcloud + Mariadb + Redis docker containers. I compared its performance with my standard php-fpm nginx approach, also deployed via docker.

I noticed that the upload speed improved significantly for large files, which is great. However, when dealing with many small files (around 2000 files totaling 20MB), I did not observe any noticeable improvement in upload speed. There was a slight reduction in server (cpu) load, but that was all. I believe this situation is quite common for Nextcloud users who frequently upload numerous smaller files, and I consider it to be Nextcloud’s Achilles’ heel in terms of performance. All the tests were conducted in my 1Gbps LAN environment.

Furthermore, I encountered difficulties in getting Rich Workspaces, Notify Push, and Nextcloud Office (Collabora) to work with LiteSpeed. Unfortunately, I forgot to test Imaginary for image previews.

After considering everything, I have decided to stick with my reliable and familiar php-fpm + nginx setup.

2 Likes