Nextcloud Talk High Performance Backend (HPB) - Multi-Domain Setup Guide

I have been trying to setup my multi-domain Nexcloud installation (which has been running for more than a decade) together with the High Performance Backend service for Talk conversations. After many attemps, I managed to make it work by creating my own docker-compose stack with 3 services (nats, janus and spreed-signaling), and avoid using Package aio-talk · GitHub

This guide provides a complete setup for deploying a Nextcloud Talk High Performance Backend (HPB) server that supports multiple Nextcloud domains using a single HPB instance. I hope it is useful for anyone dealing with this issue.

Architecture

The setup consists of three main components running in Docker containers:

  • NATS: Message broker for signaling
  • Janus Gateway: WebRTC media server
  • Nextcloud Spreed Signaling: Signaling server that connects Nextcloud to Janus

All components use network_mode: host for optimal WebRTC performance and to avoid port mapping issues with large UDP port ranges.

Step 1: Directory Structure

Create the configuration directory structure:

sudo mkdir -p /opt/hpb/config/janus
cd /opt/hpb

Step 2: Generate Secrets

Generate secure random secrets for your configuration:

# TURN secret (64 characters / 32 bytes)
openssl rand -hex 32

# Signaling secret (64 characters / 32 bytes)
openssl rand -hex 32

# Hash key (64 characters / 32 bytes)
openssl rand -hex 32

# Block key - IMPORTANT: Must be exactly 16 bytes (32 hex chars)
openssl rand -hex 16

# Internal secret (64 characters / 32 bytes)
openssl rand -hex 32

CRITICAL: The blockkey must be exactly 16, 24, or 32 bytes (32, 48, or 64 hex characters). Using 16 bytes (32 characters) is recommended. If you use a longer key, the signaling server will fail to start with an error about key length.

Step 3: NATS Configuration

Create /opt/hpb/config/nats.conf:

listen: 127.0.0.1:4222

Step 4: Janus Gateway Configuration

Main Janus Configuration

Create /opt/hpb/config/janus/janus.jcfg:

general: {
    configs_folder = "/usr/local/etc/janus"
    plugins_folder = "/usr/local/lib/janus/plugins"
    transports_folder = "/usr/local/lib/janus/transports"
    events_folder = "/usr/local/lib/janus/events"
    loggers_folder = "/usr/local/lib/janus/loggers"
    debug_level = 4
    log_to_stdout = true
}

nat: {
    ice_lite = false
    ice_tcp = false
    full_trickle = true
    rtp_port_range = "20000-40000"
}

media: {
    ipv6 = false
}

plugins: {
}

transports: {
}

WebSocket Transport Configuration

Create /opt/hpb/config/janus/janus.transport.websockets.jcfg:

general: {
}

admin: {
    admin_ws = true
    admin_ws_port = 7188
}

ws: {
    ws = true
    ws_port = 8188
    ws_interface = "127.0.0.1"
}

wss: {
    wss = false
}

VideoRoom Plugin Configuration

Create /opt/hpb/config/janus/janus.plugin.videoroom.jcfg:

general: {
    admin_key = "your_admin_key_here"
}

room-1234: {
    description = "Demo Room"
    secret = "room_admin_password"
    publishers = 6
    bitrate = 128000
    fir_freq = 10
    audiocodec = "opus"
    videocodec = "vp8"
    record = false
}

Step 5: Signaling Server Configuration

Create /opt/hpb/config/server.conf:

[http]
listen = 0.0.0.0:8081

[app]
# Shared secret - must match what you configure in Nextcloud Talk settings
secret = YOUR_SIGNALING_SECRET_HERE

[sessions]
# Hash key: 64 hex characters (32 bytes)
hashkey = YOUR_HASH_KEY_HERE
# Block key: MUST be exactly 32 hex characters (16 bytes), 48 (24 bytes), or 64 (32 bytes)
blockkey = YOUR_BLOCK_KEY_HERE

[nats]
url = nats://127.0.0.1:4222

[mcu]
type = janus
url = ws://127.0.0.1:8188

[backend]
# List all your Nextcloud backends here
backends = backend1, backend2, backend3

# Backend 1
[backend1]
url = https://cloud.example.com
# This secret must match the signaling secret above
secret = YOUR_SIGNALING_SECRET_HERE

# Backend 2
[backend2]
url = https://files.example.org
secret = YOUR_SIGNALING_SECRET_HERE

# Backend 3
[backend3]
url = https://nextcloud.another-domain.net
secret = YOUR_SIGNALING_SECRET_HERE

# TURN server configuration
[turn]
# Use "api" not "apikey"
api = static
secret = YOUR_TURN_SECRET_HERE
servers = turn:hpb.example.com:3478?transport=udp,turn:hpb.example.com:3478?transport=tcp

Important Notes:

  • All backends must use the same signaling secret
  • The api parameter should be static, not apikey
  • Replace YOUR_*_SECRET_HERE with the secrets you generated in Step 2
  • Replace domain names with your actual domains

Step 6: Docker Compose Configuration

Create /opt/hpb/docker-compose.yml (or wherever you prefer to store it):

ame: 'hpb'
services:
  nats:
    container_name: nats_server
    image: nats:latest
    command: ["-c", "/config/nats.conf"]
    volumes:
      - /opt/hpb/config/nats.conf:/config/nats.conf:ro
    network_mode: host
    restart: unless-stopped

  janus:
    container_name: janus_gateway
    image: canyan/janus-gateway:latest
    network_mode: host
    environment:
      - JANUS_API_HTTP=yes
      - JANUS_API_HTTPS=no
      - JANUS_API_WS=yes
      - JANUS_API_ADMIN_WS=yes
      - JANUS_RTP_PORT_RANGE=20000-40000
    restart: unless-stopped

  signaling:
    container_name: spreed_signaling
    image: strukturag/nextcloud-spreed-signaling:latest
    depends_on:
      - nats
      - janus
    network_mode: host
    volumes:
      - /opt/hpb/config/server.conf:/config/server.conf:ro
#    command: ["-config", "/config/server.conf"]
    environment:
      - CONFIG_FILE=/etc/signaling/server.conf
    restart: unless-stopped

Step 7: Firewall Configuration

Allow the necessary ports through your firewall:

# TURN/STUN
sudo ufw allow 3478/tcp
sudo ufw allow 3478/udp

# Signaling (if accessing directly)
sudo ufw allow 8081/tcp

# RTP for WebRTC
sudo ufw allow 20000:40000/udp

# Verify rules
sudo ufw status numbered

Step 8: Nginx Reverse Proxy (Recommended)

For production use, set up a reverse proxy with SSL termination:

Create /etc/nginx/sites-available/hpb:

server {
    listen 80;
    server_name hpb.example.com;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name hpb.example.com;

    ssl_certificate /etc/letsencrypt/live/hpb.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/hpb.example.com/privkey.pem;
    
    # SSL configuration
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers on;

    location /standalone-signaling/ {
        proxy_pass http://127.0.0.1:8081/;
        proxy_http_version 1.1;
        
        # WebSocket support
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        
        # Headers
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        
        # Long-lived connections for WebSocket
        proxy_connect_timeout 7d;
        proxy_send_timeout 7d;
        proxy_read_timeout 7d;
    }
}

Enable the site:

sudo ln -s /etc/nginx/sites-available/hpb /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

Step 9: Start the Services

cd /opt/hpb  # or wherever your docker-compose.yml is located
docker compose up -d

Step 10: Verify Installation

Check that all services are running:

# View running containers
docker ps

# Check NATS logs
docker logs nats_server

# Check Janus logs
docker logs janus_gateway

# Check Signaling logs (should show all backends loaded)
docker logs spreed_signaling

In the signaling logs, you should see:

Backend backend1 added for https://cloud.example.com/
Backend backend2 added for https://files.example.org/
Backend backend3 added for https://nextcloud.another-domain.net/
...
Listening on 0.0.0.0:8081

Test the endpoint:

curl http://localhost:8081/api/v1/welcome
# Should return: {"version":"X.X.X"}

# Or with SSL via Nginx:
curl https://hpb.example.com/standalone-signaling/api/v1/welcome

Step 11: Configure Nextcloud Talk

On each of your Nextcloud instances:

  1. Go to Administration → Talk → High-performance backend

  2. Configure the following (identical on all instances):

    • Signaling server URL: https://hpb.example.com/standalone-signaling
    • Shared secret: Use the same YOUR_SIGNALING_SECRET_HERE from your server.conf
    • Verify SSL certificate: ✓ Enabled
  3. Configure TURN servers:

    • TURN server: hpb.example.com:3478
    • Protocols: UDP and TCP
    • Secret: Use the YOUR_TURN_SECRET_HERE from your server.conf
  4. You should see: ✓ OK: Version in use: X.X.X~docker

Adding More Domains

To add additional Nextcloud instances:

  1. Edit /opt/hpb/config/server.conf
  2. Add the new backend to the backends list and create its section:
[backend]
backends = backend1, backend2, backend3, backend4

[backend4]
url = https://new-domain.example.com
secret = YOUR_SIGNALING_SECRET_HERE
  1. Restart the signaling service:
docker compose restart signaling
  1. Configure Talk on the new Nextcloud instance
1 Like

The instructions are good but missing details.

Example: meetecho/janus-gateway:latest is missing/not available. I found an alternative, but then it complains about the TURN server not being configured (or something, iirc. This was hours ago).

I ended up just using the aio-talk docker.

below a draft of a very similar article I planned for long time. maybe over time we can combine both into a best practice running HPB as container

Talk HPB overview

Long planned article and implementation of Nextcloud Talk HPB. I’m doing it hard way with all components configured one by one. this is little harder than simply adding AiO Talk container (look at High Performance Backend for Talk on Nextcloud with Docker for details) which does the job a s well but I don’t like balck boxes and prefer to know what is inside and how it works. this manual way allows more control about individual parts and gives your understanding how they work together.

I had really hard time understanding how it works as almost no docs exist and things are not really hidden but not documented well too. In fact the integration turns out pretty simple with 3 (or 4) components required. Start with the turn server which is a must for each Talk installation. I’m using coturn but there are alternatives and I think there is no real difference. TURN server is workhorse doing hard job connecting clients in the internet bypassing firewall NAT, even CG-NAT to make client talk to each other.

Once the TURN server is in place participants can talk to each other but one problem remains - each client must send it’s media to every other meeting participant.. which becomes problematic if cpu power and bandwidth is limited. this is where the Talk HPB comes into play - it works as central instance receiving one media stream from each participant multiplexing it with other streams and sending one stream to each other participant. this components is known as MCU in other communication systems. HPB consists of 3 components: signalling server, janus and nats (which might be optional for small installations).

archticture

flowchart LR
  fritz.box-- port forward<br>udp/3478<br>udp/50000-500100 -->TURN;
  fritz.box-- port forward<br>http tcp/80<br>https tcp/443 -->RP;
  subgraph fritzbox
   fritz.box(router<br>192.168.179.1);
  end
  subgraph srv
		subgraph nc.mydomain.tld
		  NC[app];
		  NCNW([more comonents like db, redis...]);
		end
		subgraph dev-nc.mydomain.tld
		  DEV-NC[app];
		  DEV-NCNW([more comonents like db, redis...]);
		end
        subgraph talk-hpb
		  TURN[coturn]
		  SIGNALLING(talk-hpb)
		  NATS
		  JANUS
		  SIGNALLING & JANUS -- turn:corurn:3478 --> TURN
		  SIGNALLING -- ws://nats://nats:4222 --> NATS
		  SIGNALLING -- ws://janus:8188 --> JANUS
		end

		RP-- https:// ${TALKHPB_FQDN} -->SIGNALLING
		RP-- https -->NC & DEV-NC
		
	    TURN <-.-> NC & DEV-NC
		RP[reverse proxy<br>:80 + :443] 
	end

configuration

below compose file creates the “talk-hpb” part of the above architecture image. strictly speaking coturn is not part of an HPB but both are required for Talk and could be shared for multiple independent Nextcloud installations.

global config .env

this files contains global settings and settings shared between services

# .env
#TURN server
COTURN_SECRET=<<<shared TRUN secret>>>
COTURN_FQDN=turn.mydomain.tld
TALKHPB_FQDN=talk-hpb.mydomain.tld
NATS_VERSION=2.11.7       # omit for :latest tag
SIGNALLING_VERSION=2.0.1  # omit for :latest tag
# janus
JANUS_VERSION=1.3.2
LIBSRTP_VERSION=2.7.0
USRSCTP_VERSION=b28f0b55b00bde67f6be80d6623e2775b88026b8
compose.yml

static docker compose file - config is done using .env file

# compose.yml
services:
  nats:
    image: nats:${NATS_VERSION:-latest}
    #command: ["-c", "/config/gnatsd.conf"]
    command:
      - -p=4222
      #- -m 8222  # management port
      #- -D       # enable debug for logs
    restart: unless-stopped

  janus:
    image: local/janus-custom
    build:
      context: ./talk-hpb # Point to the directory containing your Dockerfile
      dockerfile: ./janus-dockerfile
    restart: unless-stopped
    networks:
      - proxy
    command:
      - janus
      - --full-trickle
      - --stun-server=coturn:3478

  signalling:
    image: strukturag/nextcloud-spreed-signaling:${SIGNALLING_VERSION:-latest}
    restart: unless-stopped
    depends_on:
      - nats
      - janus
      - coturn
    networks:
      - proxy
      - default
    env_file:
      - ./talk-hpb/signalling.env
    environment:
      - EXTERNAL_HOSTNAME=${TALKHPB_FQDN}
      - TURN_API_KEY=${COTURN_SECRET}
      - TURN_SECRET=${COTURN_SECRET}
    # Define traefik router for the signaling server
    labels:
      - traefik.enable=true
      - traefik.http.routers.signalling.rule=Host(`${TALKHPB_FQDN}`)
      - traefik.http.routers.signalling.tls=true
      - traefik.http.routers.signalling.tls.certresolver=letsencryptresolver
      - traefik.http.services.signalling.loadbalancer.server.port=8080
      - traefik.docker.network=proxy

  coturn:
    image: coturn/coturn
    container_name: coturn
    restart: unless-stopped
    ports:
      - 3478:3478
      - 3478:3478/udp
      - 50000-50099:50000-50099/udp
      - 9641:9641
    environment:
      - DETECT_EXTERNAL_IP=yes
      - DETECT_RELAY_IP=yes
    command:
      - -n
      - --log-file=/var/turn.log
      - --realm=${COTURN_FQDN}
      - --use-auth-secret
      - --static-auth-secret=${COTURN_SECRET}
      - --verbose
    volumes:
      - ./coturn/:/var/
      - ./turnserver.conf:/etc/coturn/turnserver.conf
    networks:
      - proxy
talk-hpb/signalling.env
# talk-hpb/signalling.env
# https://github.com/strukturag/nextcloud-spreed-signaling/tree/master/docker
HTTP_LISTEN=0.0.0.0:8080
TRUSTED_PROXIES=172.16.0.0/12,192.168.0.0/1,10.0.0.0/8,fc00::/7,fe80::/10,2001:db8::/32
MAX_STREAM_BITRATE=2097152 # 2Mbit/s
MAX_SCREEN_BITRATE=4194304 # 4Mbit/s
# Janus/WebRTC
USE_JANUS=1
JANUS_URL=ws://janus:8188
# No need for nats when only one Spreed-Server
NATS_URL=nats://nats:4222
TURN_SERVERS=turn:coturn:3478
#LOG_LEVEL="debug"
LOG_LEVEL="info"

#BACKENDS_ALLOWALL: Allow all backends. Extremly insecure - use only for development!
#BACKENDS_ALLOWALL_SECRET: Secret when BACKENDS_ALLOWALL is enabled.
BACKENDS="NC DEVNC TESTNC"
# nc
BACKEND_NC_URL=https://nc.mydomain.tld
BACKEND_NC_SHARED_SECRET=<<<replace me with a secret>>>
BACKEND_NC_SESSION_LIMIT=15
BACKEND_NC_MAX_STREAM_BITRATE=2097152
BACKEND_NC_MAX_SCREEN_BITRATE=4194304
# dev-nc
BACKEND_DEVNC_URL=https://dev-nc.mydomain.tld
BACKEND_DEVNC_SHARED_SECRET=<<<replace me with a secret>>>
BACKEND_DEVNC_SESSION_LIMIT=5
BACKEND_DEVNC_MAX_STREAM_BITRATE=1048576
BACKEND_DEVNC_MAX_SCREEN_BITRATE=2097152
# test-nc
BACKEND_TESTNC_URL=https://test-nc.mydomain.tld
BACKEND_TESTNC_SHARED_SECRET=<<<replace me with a secret>>>
BACKEND_TESTNC_SESSION_LIMIT=5
BACKEND_TESTNC_MAX_STREAM_BITRATE=1048576
BACKEND_TESTNC_MAX_SCREEN_BITRATE=2097152