Raspberry Pi4, Docker, Nginx, worker process 100% when downloading/uploading big files

ℹ️ Support 📦 Appliances (Docker, Snappy, VM, NCP, AIO)
, , ,
1

Hi, i have created a server based on a RaspberryPi4.
When i download big file like mp4 from my nextcloud account, i can see that “worker process” process keep a single core at 100%,

Htop while downloading

image
image881Ă—171 8.57 KB

this apparently is a bottleneck because my bandwidth is only saturated at 50% of its maximum (about 130Mbps (around 16MB/s) out of 300Mbps available)

If i download from 2 account/client simultaneously, i can see 2 worker process and if each is on a different core i can reach 130Mbs on each client so i can clearly say that bottleneck is the CPU single core power.

I have tried to set gzip off on nginx.conf (in http, server and location session) but nothing change.

I would ask:

  • Why downloading a file from nextcloud throught nginx keep worker process stuck at 100% of CPU? Even with gzip OFF?
  • there is a method to lighten the load?
  • Can i reach my max upload powr of 300Mbps?

Thanks in advance

My configuration:
Hardware

  • Raspberry Pi 4 (4GB RAM)
  • NvME LexarNP610Pro on USB10Gbps Enclosure
  • Intranet: Gbit Ethernet/Router
  • Internet: FTTH 940Mbsp DL 300Mbps UL

Relevant BenchMark:

  • Speedtest
   Speedtest by Ookla

      Server: ***
         ISP: ***
Idle Latency:    14.56 ms   (jitter: 0.21ms, low: 14.47ms, high: 14.91ms)
    Download:   938.16 Mbps (data used: 1.0 GB)
                 81.85 ms   (jitter: 13.49ms, low: 16.06ms, high: 299.82ms)
      Upload:   312.89 Mbps (data used: 156.2 MB)
                 60.78 ms   (jitter: 6.93ms, low: 10.49ms, high: 115.34ms)
 Packet Loss:     0.0%
  Result URL: ***
  • Disk Speed (simple dd test only to prove this is not the bootleneck)
sudo dd if=/dev/zero of=benchmark.img bs=1G count=5 status=progress
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 25 s, 212 MB/s
5+0 records in
5+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 25.3033 s, 212 MB/s

Software

  • debian.11~bullseye
  • OMV6 6.9.6-2 (Shaitan)
  • Docker docker-ce 5:24.0.7-1
  • Compose 2.21.0-1
  • Nginx v2.10.4
  • Nextcloud 27.1.3

This is my nextcloud compose yml

version: '2'

networks:
  nextcloud_network:
    external: false

services:
  nextcloud-db:
    image: yobasystems/alpine-mariadb:latest
    restart: unless-stopped
    ports:
      - 3316:3306
    command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW --innodb-file-per-table=1 --skip-innodb-read-only-compressed
    volumes:
      - .../db:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=***
      - MYSQL_PASSWORD=***
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud

    networks:
      - nextcloud_network

  nextcloud-app:
    image: nextcloud
    restart: unless-stopped
    ports:
      - 8888:80
    volumes:
      - .../app:/var/www/html 
      - .../data:/var/www/html/data 
    environment:
      - MYSQL_PASSWORD=***
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
      - MYSQL_HOST=nextcloud-db
      - NEXTCLOUD_TRUSTED_DOMAINS=my.domain
      - OVERWRITEPROTOCOL=https
      - REDIS_HOST=nextcloud-redis
    networks:
      - nextcloud_network

  nextcloud-redis:
    image: redis:alpine
    container_name: nextcloud-redis
    volumes:
      - .../redis:/data
    networks:
      - nextcloud_network
    restart: unless-stopped

This is my nginx compose yml

version: '3'

networks:
  frontend:
  backend:

services:

  npm-app:
    image: jc21/nginx-proxy-manager:latest
    restart: unless-stopped
    ports:
      - "80:80"
      - "81:81"
      - "443:443"
    environment:
      - DB_MYSQL_HOST=npm-db
      - DB_MYSQL_PORT=3306
      - DB_MYSQL_USER=npm
      - DB_MYSQL_PASSWORD=***
      - DB_MYSQL_NAME=npm
    volumes:
      - .../data:/data
      - .../nginxconf:/etc/nginx
      - .../ssl:/etc/letsencrypt
    networks:
      - frontend
      - backend

  npm-db:
    image: jc21/mariadb-aria:latest
    restart: unless-stopped
    ports:
      - 3326:3306 
    environment:
      - MYSQL_ROOT_PASSWORD=***
      - MYSQL_DATABASE=npm
      - MYSQL_USER=npm
      - MYSQL_PASSWORD=***
    volumes:
      - .../db:/var/lib/mysql
    networks:
      - backend

my nginx.conf is leaved default as the yml docker compose file

# run nginx in foreground
daemon off;
pid /run/nginx/nginx.pid;
user npm;

# Set number of worker processes automatically based on number of CPU cores.
worker_processes auto;

# Enables the use of JIT for regular expressions to speed-up their processing.
pcre_jit on;

error_log /data/logs/fallback_error.log warn;

# Includes files with directives to load dynamic modules.
include /etc/nginx/modules/*.conf;

events {
	include /data/nginx/custom/events[.]conf;
}

http {
	include                       /etc/nginx/mime.types;
	default_type                  application/octet-stream;
	sendfile                      on;
	server_tokens                 off;
	tcp_nopush                    on;
	tcp_nodelay                   on;
	client_body_temp_path         /tmp/nginx/body 1 2;
	keepalive_timeout             90s;
	proxy_connect_timeout         90s;
	proxy_send_timeout            90s;
	proxy_read_timeout            90s;
	ssl_prefer_server_ciphers     on;
	gzip                          on;
	proxy_ignore_client_abort     off;
	client_max_body_size          2000m;
	server_names_hash_bucket_size 1024;
	proxy_http_version            1.1;
	proxy_set_header              X-Forwarded-Scheme $scheme;
	proxy_set_header              X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header              Accept-Encoding "";
	proxy_cache                   off;
	proxy_cache_path              /var/lib/nginx/cache/public  levels=1:2 keys_zone=public-cache:30m max_size=192m;
	proxy_cache_path              /var/lib/nginx/cache/private levels=1:2 keys_zone=private-cache:5m max_size=1024m;

	log_format proxy '[$time_local] $upstream_cache_status $upstream_status $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Length $body_bytes_sent] [Gzip $gzip_ratio] [Sent-to $server] "$http_user_agent" "$http_referer"';
	log_format standard '[$time_local] $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Length $body_bytes_sent] [Gzip $gzip_ratio] "$http_user_agent" "$http_referer"';

	access_log /data/logs/fallback_access.log proxy;

	# Dynamically generated resolvers file
	include /etc/nginx/conf.d/include/resolvers.conf;

	# Default upstream scheme
	map $host $forward_scheme {
		default http;
	}

	# Real IP Determination

	# Local subnets:
	set_real_ip_from 10.0.0.0/8;
	set_real_ip_from 172.16.0.0/12; # Includes Docker subnet
	set_real_ip_from 192.168.0.0/16;
	# NPM generated CDN ip ranges:
	include conf.d/include/ip_ranges.conf;
	# always put the following 2 lines after ip subnets:
	real_ip_header X-Real-IP;
	real_ip_recursive on;

	# Custom
	include /data/nginx/custom/http_top[.]conf;

	# Files generated by NPM
	include /etc/nginx/conf.d/*.conf;
	include /data/nginx/default_host/*.conf;
	include /data/nginx/proxy_host/*.conf;
	include /data/nginx/redirection_host/*.conf;
	include /data/nginx/dead_host/*.conf;
	include /data/nginx/temp/*.conf;

	# Custom
	include /data/nginx/custom/http[.]conf;
}

stream {
	# Files generated by NPM
	include /data/nginx/stream/*.conf;

	# Custom
	include /data/nginx/custom/stream[.]conf;
}

# Custom
include /data/nginx/custom/root[.]conf;

conf.d/default.conf

# "You are not configured" page, which is the default if another default doesn't exist
server {
	listen 80;
	listen [::]:80;

	set $forward_scheme "http";
	set $server "127.0.0.1";
	set $port "80";

	server_name localhost-nginx-proxy-manager;
	access_log /data/logs/fallback_access.log standard;
	error_log /data/logs/fallback_error.log warn;
	include conf.d/include/assets.conf;
	include conf.d/include/block-exploits.conf;
	include conf.d/include/letsencrypt-acme-challenge.conf;

	location / {
		index index.html;
		root /var/www/html;
	}
}

# First 443 Host, which is the default if another default doesn't exist
server {
	listen 443 ssl;
	listen [::]:443 ssl;

	set $forward_scheme "https";
	set $server "127.0.0.1";
	set $port "443";

	server_name localhost;
	access_log /data/logs/fallback_access.log standard;
	error_log /dev/null crit;
	include conf.d/include/ssl-ciphers.conf;
	ssl_reject_handshake on;

	return 444;
}
2

I wouldn’t say it is too bad. I’d try to use a plain scp/sftp connection as a benchmark, Nextcloud usually performs worse since the database is involved.

Not sure if encryption uses a lot of performance, for disk encryption it seems to be the case (Raspberry pi4 and low crypto performance | The FreeBSD Forums). Perhaps there are ciphers that perform better than others, but you still want to use secure ciphers and cover a certain range of supported clients.

1 Like