Slow web interface and upload/download speed

Support intro

Thread about performance tuning of my nextcloud instance.

Note: I’m new to Nextcloud and all the performance tuning one can do to fit the server resources. There is a bunch of stuff in the admin manual that I’ve yet to try. The thing is that I’m pressed for time to fix this issue for my users, hence the thread. Quick tips about what I should focus on first and how I can troubleshoot to get to the crux of the problem are appreciated, otherwise I’ll be researching and tuning things on my own and update the thread as I see progress.

What I’ve tried so far:

  • Changed postgresql.conf to have “shared_buffers = 1GB”

VPS Details

6 cores CPU
16GB RAM
400GB SSD storage
400 Mb/s network port

Versions
Nextcloud version: 17.0.2
Operating system and version: Arch Linux, Kernel: Linux 4.19.87-1-lts, Architecture: x86-64
Reverse-proxy nginx version: 1.17.5
Apache version: 2.4.38
PHP version: 7.3.14

Docker Setup

Here is the Base docker-compose setup that I’m using. I’m using the apache-17 image.

The settings I changed with that setup:

  • Added the https override in the config.php because of the reverse-proxy and added a redis container.

The issue I’m facing

Slow web UI
Slow upload speed of larger files (100MB and up)

Is this the first time you’ve seen this error? (Y/N):

This has been an issue for some time now.

Steps to replicate it:

My users are complaining about the slow UI, but I don’t notice it myself, so can’t replicate that.

To reproduce the slow upload:

  1. scp a large (100MB maybe) file to the VPS, note how long it took
  2. upload the same file through the web interface, note how long it took
  3. note the big difference in speed (2 to 3 times slower in the web interface for me)

Logs
I’d have added the nginx and nextcloud logs but there’s just a bunch of sensitive info in them.

The output of your config.php file:

<?php
$CONFIG = array (
  'instanceid' => 'XXX',
  'passwordsalt' => 'XXX',
  'secret' => 'XXX',
  'trusted_domains' =>
  array (
    0 => 'files.somedomain.com',
    1 => 'filesdev.somedomeain.com',
  ),
  'datadirectory' => '/var/www/html/data',
  'dbtype' => 'pgsql',
  'version' => '17.0.2.1',
  'overwrite.cli.url' => 'https://files.somedomain.com',
  'dbname' => 'nextcloud',
  'dbhost' => 'db_container:5432',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'postgres',
  'dbpassword' => 'XXX',
  'installed' => true,
  'loglevel' => 0,
  'maintenance' => false,
  'mail_smtpmode' => 'sendmail',
  'mail_smtpsecure' => 'ssl',
  'mail_sendmailmode' => 'smtp',
  'mail_from_address' => 'files',
  'mail_domain' => 'files.somedomain.com',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpauth' => 1,
  'theme' => '',
  'overwriteprotocol' => 'https',
  'apps_paths' =>
  array (
    0 =>
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 =>
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
    'host' => 'redis',
    'port' => 6379,
    'password' => false,
  ),
);

I’m thinking this might be due to the reverse proxy in front of the nextcloud container. I’m going to see if I can remove the proxy and see what happens.

From one of my users:
“It took 1.5 hours to download a video from the cloud and 6 minutes to upload to Youtube”

So I’m trying to make things work without reverse proxying, but I’m getting 404 file not found in the browser. I don’t know how I can properly debug it either. Here is what changed:

I’m using the fpm version for the nextcloud container instead of the apache version (I was always using cron). I removed the VIRTUAL_HOST env variable from the container so nginx-proxy will not generate proxy configs for it.

My docker-compose.yml looks like this:

version: '3'

services:
  
  redis:
    image: redis:5.0.7-alpine
    restart: always
    networks:
      - files-tier

  files_db:
    image: postgres:11-alpine
    #image: postgres:alpine
    restart: always
    volumes:
      - files_db:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=
      - POSTGRES_USER=postgres
    networks:
      - files-tier

      #- SMTP_HOST=mailsender
  
  files:
    build: ./files
    restart: always
    volumes:
      - nextcloud:/var/www/html
    environment:
      - POSTGRES_PASSWORD=
      - POSTGRES_USER=postgres
      - POSTGRES_HOST=files_db
      - POSTGRES_DB=nextcloud
      - REDIS_HOST=redis
      - MAIL_FROM_ADDRESS=noreply@files.somedomain.com
      - LETSENCRYPT_HOST=files.somedomain.com
      - LETSENCRYPT_EMAIL=bob@example.com
    depends_on:
      - files_db
      - redis
    networks:
      - proxy-tier
      - files-tier
  proxy:
    build: ./proxy
    restart: always
    ports:
      - 80:80
      - 443:443
    labels:
      com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy: "true"
    volumes:
      - certs:/etc/nginx/certs:ro
      - vhost.d:/etc/nginx/vhost.d
      - html:/usr/share/nginx/html
      - /var/run/docker.sock:/tmp/docker.sock:ro
    networks:
      - proxy-tier

  letsencrypt-companion:
    image: jrcs/letsencrypt-nginx-proxy-companion
    restart: always
    volumes:
      - certs:/etc/nginx/certs
      - vhost.d:/etc/nginx/vhost.d
      - html:/usr/share/nginx/html
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - proxy-tier
    depends_on:
      - proxy

volumes:
  files_db:
  nextcloud:
  certs:
  vhost.d:
  html:

networks:
  proxy-tier:
  files-tier:

I added this nginx configuration file to the proxy container:

log_format nc '$host $remote_addr - $remote_user [$time_local] '
              '"$request" $status $body_bytes_sent '
              '"$http_referer" "$http_user_agent"';

upstream php-handler {
    server files:9000;
}

server {
    server_name files.somedomain.com;
    listen 80 ;
    listen [::]:80;
    access_log /var/log/nginx/access.log nc;
    return 301 https://$host$request_uri;
}

server {

    server_name files.somedomain.com;
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    access_log /var/log/nginx/access.log nc;
    ssl_certificate /etc/nginx/certs/files.somedomain.com.crt;
    ssl_certificate_key /etc/nginx/certs/files.somedomain.com.key;

    # Add headers to serve security related headers
    # Before enabling Strict-Transport-Security headers please read into this
    # topic first.
    #add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;" always;
    #
    # WARNING: Only add the preload option once you read about
    # the consequences in https://hstspreload.org/. This option
    # will add the domain to a hardcoded list that is shipped
    # in all major browsers and getting removed from this list
    # could take several months.
    add_header Referrer-Policy "no-referrer" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-Download-Options "noopen" always;
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-Permitted-Cross-Domain-Policies "none" always;
    add_header X-Robots-Tag "none" always;
    add_header X-XSS-Protection "1; mode=block" always;

    # Remove X-Powered-By, which is an information leak
    fastcgi_hide_header X-Powered-By;

    # Path to the root of your installation
    root /var/www/html;

    include /etc/nginx/vhost.d/default;

    location = /robots.txt {
        allow all;
        log_not_found off;
        access_log off;
    }

    # The following 2 rules are only needed for the user_webfinger app.
    # Uncomment it if you're planning to use this app.
    #rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
    #rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json last;

    # The following rule is only needed for the Social app.
    # Uncomment it if you're planning to use this app.
    #rewrite ^/.well-known/webfinger /public.php?service=webfinger last;

    location = /.well-known/carddav {
        return 301 $scheme://$host:$server_port/remote.php/dav;
    }

    location = /.well-known/caldav {
        return 301 $scheme://$host:$server_port/remote.php/dav;
    }

    # set max upload size
    client_max_body_size 10G;
    fastcgi_buffers 64 4K;

    # Enable gzip but do not remove ETag headers
    gzip on;
    gzip_vary on;
    gzip_comp_level 4;
    gzip_min_length 256;
    gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
    gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;

    # Uncomment if your server is build with the ngx_pagespeed module
    # This module is currently not supported.
    #pagespeed off;

    location / {
        rewrite ^ /index.php;
    }

    location ~ ^\/(?:build|tests|config|lib|3rdparty|templates|data)\/ {
        deny all;
    }
    location ~ ^\/(?:\.|autotest|occ|issue|indie|db_|console) {
        deny all;
    }

    location ~ ^\/(?:index|remote|public|cron|core\/ajax\/update|status|ocs\/v[12]|updater\/.+|oc[ms]-provider\/.+)\.php(?:$|\/) {
        fastcgi_split_path_info ^(.+?\.php)(\/.*|)$;
        set $path_info $fastcgi_path_info;
        try_files $fastcgi_script_name =404;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param PATH_INFO $path_info;
        fastcgi_param HTTPS on;

        # Avoid sending the security headers twice
        fastcgi_param modHeadersAvailable true;

        # Enable pretty urls
        fastcgi_param front_controller_active true;
        fastcgi_pass php-handler;
        fastcgi_intercept_errors on;
        fastcgi_request_buffering off;
    }

    location ~ ^\/(?:updater|oc[ms]-provider)(?:$|\/) {
        try_files $uri/ =404;
        index index.php;
    }

    # Adding the cache control header for js, css and map files
    # Make sure it is BELOW the PHP block
    location ~ \.(?:css|js|woff2?|svg|gif|map)$ {
        try_files $uri /index.php$request_uri;
        add_header Cache-Control "public, max-age=15778463";
        # Add headers to serve security related headers (It is intended to
        # have those duplicated to the ones above)
        # Before enabling Strict-Transport-Security headers please read into
        # this topic first.
        #add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;" always;
        #
        # WARNING: Only add the preload option once you read about
        # the consequences in https://hstspreload.org/. This option
        # will add the domain to a hardcoded list that is shipped
        # in all major browsers and getting removed from this list
        # could take several months.
        add_header Referrer-Policy "no-referrer" always;
        add_header X-Content-Type-Options "nosniff" always;
        add_header X-Download-Options "noopen" always;
        add_header X-Frame-Options "SAMEORIGIN" always;
        add_header X-Permitted-Cross-Domain-Policies "none" always;
        add_header X-Robots-Tag "none" always;
        add_header X-XSS-Protection "1; mode=block" always;

        # Optional: Don't log access to assets
        access_log off;
    }

    location ~ \.(?:png|html|ttf|ico|jpg|jpeg|bcmap|mp4|webm)$ {
        try_files $uri /index.php$request_uri;
        # Optional: Don't log access to other assets
        access_log off;
    }
}

And restarted everything. I keep getting 404 errors, even if I remove the files container. I can’t see any more details about why this is happening.

Doh. Found out what was wrong.

I needed to give the proxy container readonly access to the nextcloud volume on /var/www/html because the “try_files” nginx directive checks for the files locally.