Upload to datadir slower than external storage SMB share on *same* disk

[jahey]

Nextcloud version: 28.0.4
Operating system and version: Debian Bookworm
Apache or nginx version: nginx 1.25.5
PHP version: 8.2-fpm
DB Version: Postgresql 15

The issue you are facing:
Slow upload speed on a new Nextcloud install (0.5 MB/s max).

For example:

• 14MB JPG Upload Time: 35 seconds
• 16MB DNG Upload Time: 39 seconds
• 10MB DNG Upload Time: 25 seconds

Is this the first time you’ve seen this error? (Y/N): N

Had a Nextcloud instance running on a raspi with the same slow upload speed and put it off on the “slow” hardware.
New instance is a VM running 6 vCPUs, 8 GB Ram, 32 GB Boot NVMe SSD and 2x 10TB HDD (Raid 1 ncdata disks - dd speed: 245 MB/s).
So shouldn’t be a resource problem (if necessary, more ram can be allocated).

Steps to replicate it:

1. Install Nextcloud with Setup Guide:
   https://decatec.de/home-server/nextcloud-auf-ubuntu-server-22-04-lts-mit-nginx-postgresql-mariadb-php-lets-encrypt-redis-und-fail2ban/
2. Configure Imaginary
3. Upload File

Only thing I left out was Fail2Ban.

The output of your Nextcloud log in Admin > Logging:

nothing relevant

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

$CONFIG = array (
  'maintenance_window_start' => 3,
  'instanceid' => 'balela',
  'passwordsalt' => 'blabla',
  'secret' => 'blabla,
  'trusted_domains' =>
  array (
    0 => 'my-domain',
  ),
  'datadirectory' => '/mnt/ncdata/data',
  'dbtype' => 'pgsql',
  'version' => '28.0.4.1',
  'overwrite.cli.url' => 'https://my-domain,
  'dbname' => 'nextcloud_db',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => '',
  'dbpassword' => '',
  'installed' => true,
  'default_phone_region' => 'DE',
  'memcache.local' => '\OC\Memcache\APCu',
  'overwriteprotocol' => 'https',
  'logtimezone' => 'Europe/Berlin',
  'filelocking.enabled' => true,
  'memcache.locking' => '\OC\Memcache\Redis',
  'redis' => array (
     'host' => '/var/run/redis/redis-server.sock',
     'port' => 0,
     'timeout' => 0.0,
    ),
  'enable_previews' => true,
  'enabledPreviewProviders' =>
  array (
    0 => 'OC\\Preview\\PNG',
    1 => 'OC\\Preview\\JPEG',
    2 => 'OC\\Preview\\GIF',
    3 => 'OC\\Preview\\HEIC',
    4 => 'OC\\Preview\\BMP',
    5 => 'OC\\Preview\\XBitmap',
    6 => 'OC\\Preview\\MP3',
    7 => 'OC\\Preview\\TXT',
    8 => 'OC\\Preview\\MarkDown',
    9 => 'OC\\Preview\\Movie',
    10 => 'OC\\Preview\\MKV',
    11 => 'OC\\Preview\\MP4',
    12 => 'OC\\Preview\\AVI',
    13 => 'OC\\Preview\\PDF',
    14 => 'OC\\Preview\\Imaginary',
  ),
  'preview_imaginary_url' => 'http://127.0.0.1:9000',
  'preview_max_x' => 4096,
  'preview_max_y' => 4096,
  'preview_max_filesize_image' => 50,
  'preview_max_memory' => 256,
);

The output of your Apache/nginx/system log in /var/log/____:

nothing indicating an error

I’d really appreciate the help to configure this instance correctly for a somewhat reasonable upload speed.

The instance is btw. local so it can’t be a network bottleneck.
The test device is connected via 1 Gbit Lan to the server.
(iPerf says 940 Mbit in both directions)

Thank you in advance

[jtr]

@jahey

The instance is btw. local so it can’t be a network bottleneck.
The test device is connected via 1 Gbit Lan to the server.

Is this via the Web UI or one of the installable client apps?

If the former, what does the Network tab in your browser console look like during these transactions? They definitely should be much faster!

I know nothing of that third-party guide you referenced, but have you cross referenced it with the Nginx config in the official Admin Manual for >=v28? https://docs.nextcloud.com

[jahey]

Thank you very much for the quick response.

I applied every part of the default nginx config, sadly with no luck.
Nextcloud Nginx Conf:

upstream php-handler {
    server unix:/run/php/php8.2-fpm.sock;
}

# Set the `immutable` cache control options only for assets with a cache busting `v` argument
map $arg_v $asset_immutable {
    "" "";
    default "immutable";
}

server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name domain localip;
    
    # Prevent nginx HTTP Server Detection
    server_tokens off;

    root /var/www;

    location ^~ /.well-known/acme-challenge {
        default_type text/plain;
        root /var/www/letsencrypt;
    }

	location / {
		return 301 https://$host$request_uri;
	}
}


server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name domain localip;

    # Path to the root of your installation
    root /var/www/nextcloud;

    # SSL configuration
	# RSA certificates
        # currently self-signed cert, so please don't mind the commented out ssl lines
	ssl_certificate /etc/letsencrypt/rsa/fullchain.pem;
	ssl_certificate_key /etc/letsencrypt/rsa/key.pem;
	# ECC certificates
	#ssl_certificate /etc/letsencrypt/ecc/fullchain.pem;
	#ssl_certificate_key /etc/letsencrypt/ecc/key.pem;

	# This should be ca.pem (certificate with the additional intermediate certificate)
	# See here: https://certbot.eff.org/docs/using.html
	# ECC
	#ssl_trusted_certificate /etc/letsencrypt/nextcloud.meinedomain.de/ecc/ca.pem;
  
    # Prevent nginx HTTP Server Detection
    server_tokens off;
  	
#
# SSL Configuration
#

# Not using TLSv1 will break:
# Android <= 4.4.40 IE <= 10 IE mobile <=10
# Removing TLSv1.1 breaks nothing else!
ssl_protocols TLSv1.2 TLSv1.3;

# SSL ciphers: RSA + ECDSA
# Two certificate types (ECDSA, RSA) are needed.
#ssl_ciphers 'TLS-CHACHA20-POLY1305-SHA256:TLS-AES-256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-#POLY1305:ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384';

# Diffie-Hellman parameter for DHE ciphersuites, recommended 4096 bits
#ssl_dhparam /etc/nginx/dhparams/dhparams.pem;

# Use multiple curves.
#ssl_ecdh_curve secp521r1:secp384r1;

# Server should determine the ciphers, not the client
#ssl_prefer_server_ciphers on;

# SSL session handling
#ssl_session_timeout 1d; 
#ssl_session_cache shared:SSL:50m; 
#ssl_session_tickets off;

# DNS resolver
#resolver 10.20.1.1;

	
#
# Header configuration
#  

# HSTS (ngx_http_headers_module is required) In order to be recoginzed by SSL test, there must be an index.hmtl in the server's root
#add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload;" always; 

    # set max upload size and increase upload timeout:
    client_max_body_size 512M;
    client_body_timeout 300s;
    fastcgi_buffers 64 4K;

    # Enable gzip but do not remove ETag headers
    gzip on;
    gzip_vary on;
    gzip_comp_level 4;
    gzip_min_length 256;
    gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
    gzip_types application/atom+xml text/javascript application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/wasm application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;

    # Pagespeed is not supported by Nextcloud, so if your server is built
    # with the `ngx_pagespeed` module, uncomment this line to disable it.
    #pagespeed off;

    # The settings allows you to optimize the HTTP2 bandwidth.
    # See https://blog.cloudflare.com/delivering-http-2-upload-speed-improvements/
    # for tuning hints
    client_body_buffer_size 512k;

    add_header Referrer-Policy no-referrer always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-Permitted-Cross-Domain-Policies none always;
    add_header X-Robots-Tag "noindex, nofollow" always;
    add_header X-XSS-Protection "1; mode=block" always;
#add_header X-Download-Options noopen always;
    # Remove X-Powered-By, which is an information leak
    fastcgi_hide_header X-Powered-By;

# Disable FLoC
#add_header Permissions-Policy "interest-cohort=()";
    
    # Add .mjs as a file extension for javascript
    # Either include it in the default mime.types list
    # or include you can include that list explicitly and add the file extension
    # only for Nextcloud like below:
    include mime.types;
    types {
        text/javascript js mjs;
	application/wasm wasm;
    }

    # Specify how to handle directories -- specifying `/index.php$request_uri`
    # here as the fallback means that Nginx always exhibits the desired behaviour
    # when a client requests a path that corresponds to a directory that exists
    # on the server. In particular, if that directory contains an index.php file,
    # that file is correctly served; if it doesn't, then the request is passed to
    # the front-end controller. This consistent behaviour means that we don't need
    # to specify custom rules for certain paths (e.g. images and other assets,
    # `/updater`, `/ocm-provider`, `/ocs-provider`), and thus
    # `try_files $uri $uri/ /index.php$request_uri`
    # always provides the desired behaviour.
    index index.php index.html /index.php$request_uri;

    # Rule borrowed from `.htaccess` to handle Microsoft DAV clients
    location = / {
        if ( $http_user_agent ~ ^DavClnt ) {
            return 302 /remote.php/webdav/$is_args$args;
        }
    }

    location = /robots.txt {
        allow all;
        log_not_found off;
        access_log off;
    }

    # Make a regex exception for `/.well-known` so that clients can still
    # access it despite the existence of the regex rule
    # `location ~ /(\.|autotest|...)` which would otherwise handle requests
    # for `/.well-known`.
    location ^~ /.well-known {
        # The rules in this block are an adaptation of the rules
        # in `.htaccess` that concern `/.well-known`.

        location = /.well-known/carddav { return 301 /remote.php/dav/; }
        location = /.well-known/caldav  { return 301 /remote.php/dav/; }

        location /.well-known/acme-challenge    { try_files $uri $uri/ =404; }
        location /.well-known/pki-validation    { try_files $uri $uri/ =404; }

        # Let Nextcloud's API for `/.well-known` URIs handle all other
        # requests by passing them to the front-end controller.
        return 301 /index.php$request_uri;
    }

    # Rules borrowed from `.htaccess` to hide certain paths from clients
    location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)(?:$|/)  { return 404; }
    location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console)                { return 404; }

    # Ensure this block, which passes PHP files to the PHP process, is above the blocks
    # which handle static assets (as seen below). If this block is not declared first,
    # then Nginx will encounter an infinite rewriting loop when it prepends `/index.php`
    # to the URI, resulting in a HTTP 500 error response.
    location ~ \.php(?:$|/) {
        # Required for legacy support
        rewrite ^/(?!index|remote|public|cron|core\/ajax\/update|status|ocs\/v[12]|updater\/.+|ocs-provider\/.+|.+\/richdocumentscode(_arm64)?\/proxy) /index.php$request_uri;

        fastcgi_split_path_info ^(.+?\.php)(/.*)$;
        set $path_info $fastcgi_path_info;

        try_files $fastcgi_script_name =404;

        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param PATH_INFO $path_info;
        fastcgi_param HTTPS on;

        fastcgi_param modHeadersAvailable true;         # Avoid sending the security headers twice
        fastcgi_param front_controller_active true;     # Enable pretty urls
        fastcgi_pass php-handler;

        fastcgi_intercept_errors on;
        fastcgi_request_buffering off;

        fastcgi_max_temp_file_size 0;
    
#    	fastcgi_read_timeout 600;
#		fastcgi_send_timeout 600;
#		fastcgi_connect_timeout 600;
#
#		fastcgi_param PHP_VALUE "upload_max_filesize = 10G
#			post_max_size = 10G
#			max_execution_time = 3600
#			output_buffering = off";
    }

#    location ~ \.(?:css|js|mjs|svg|gif|png|jpg|ico|wasm|tflite|map|ogg|flac)$ {
#        try_files $uri /index.php$request_uri;
#        add_header Cache-Control "public, max-age=15778463, $asset_immutable";
#        access_log off;     # Optional: Don't log access to assets
#
#        location ~ \.wasm$ {
#            default_type application/wasm;
#        }
#    }

    # Serve static files
    location ~ \.(?:css|js|mjs|svg|gif|png|jpg|ico|wasm|tflite|map|ogg|flac)$ {
        try_files $uri /index.php$request_uri;
        # HTTP response headers borrowed from Nextcloud `.htaccess`
        add_header Cache-Control                     "public, max-age=15778463$asset_immutable";
        add_header Referrer-Policy                   "no-referrer"       always;
        add_header X-Content-Type-Options            "nosniff"           always;
        add_header X-Frame-Options                   "SAMEORIGIN"        always;
        add_header X-Permitted-Cross-Domain-Policies "none"              always;
        add_header X-Robots-Tag                      "noindex, nofollow" always;
        add_header X-XSS-Protection                  "1; mode=block"     always;
        access_log off;     # Optional: Don't log access to assets
    }

    location ~ \.woff2?$ {
        try_files $uri /index.php$request_uri;
        expires 7d;         # Cache-Control policy borrowed from `.htaccess`
        access_log off;     # Optional: Don't log access to assets
    }

    # Rule borrowed from `.htaccess`
    location /remote {
        return 301 /remote.php$request_uri;
    }

    location / {
        try_files $uri $uri/ /index.php$request_uri;
    }
}

nginx.conf

user  www-data;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

@jtr

If the former, what does the Network tab in your browser console look like during these transactions?

Regarding the developer tools, the only interesting thing I could find in chrome dev tools was in the timing tab of the upload request:

Removing http2 from the nginx conf and presumably switch to http1 changes it to this:

Does Nextcloud perform any compression or file processing by default?

Sadly I don’t know enough about php uploads to interpret it. Help would be really appreciated.

This is btw. the result from uploading a 17.1 MB JPG via Chrome, so something that shouldn’t take 41s on a local wired connection.

Is Nextcloud Upload configured by php-fpm or php-cli?
I only ever updated the php-fpm config, could that be the problem?

[jahey]

So I increased php-fpm pm values, tested gzip and chunking disabled individually, watched upload_tmp_dir (just enabled it for testing purposes) and the server always receives the data very slowly and especially when using temp dir, the file grows by just .2 MB/s. That shouldn’t be.

I even tried just pure SMB to these disks to rule out networking etc. and the file got copied in a second instead of 39 like with Nextcloud.

Anyone got any idea? I guess its still a problem with PHP but I don’t know where to look or change settings rn.

[jahey]

Little Update on testing:

I forgot to mention, that Nextcloud is virtualized inside of Truenas.
When testing for performance I just created a SMB Share on the same disks, that form a vdisk for the NC VM. It performed perfectly and showed there is a problem with NC itself, not the hardware.

Just out of curiosity I tried external Storage in NC and mounted a Truenas SMB Share from the same Dataset, that the NC Data zvol was on and magic, it performed as expected at 20MB/s again, as long as I didn’t reach the buffer size. That still breaks uploads.

For some reason, mounting an external SMB Share in NC that resembles the same disks as the NC datadir is faster, than NC writing to its own datadir.

How can that be?