Nextcloud Desktop Client Windows Error 504 gateway time-out on "MOVE .../remote.php/dav/uploads/admin/'#number#/.file'" and very slow upload and download

Support intro

Nextcloud version (eg, 20.0.5): 26.0.2
Operating system and version (eg, Ubuntu 20.04): OMV6 aka debnian 11
Apache or nginx version (eg, Apache 2.4.25): nginx
PHP version (eg, 7.4): 8.2.7

The issue you are facing:
The Windows Desktop Client works with smal files (<500MB). Any larger File gets uploaded and then a error pops up in the client that goes like this:

Server responded Error 504 gateway time-out on MOVE -Serveraddress-/remote.php/dav/uploads/admin/'-randomnumber-/.file

But the file still shows up in the browser.
After that the client retries and fails again:

Server responded Error 423 Locken on MOVE -Serveraddress-/remote.php/dav/uploads/admin/'-randomnumber-/.file

Upload is also extremly slow at about 15Mb/s. If I open the client and look at the progressbar it drops down to a mind blowing 1Mb/s. The Server is connected via 1GiBit/s

Is this the first time you’ve seen this error? (Y/N):
N

Steps to replicate it:

  1. Setup Nextcloud using Docker (fpm + nginx + nginx proxy + redis + mariadb)
  2. Install the Windows client
  3. Upload file > 500MB

The output of your Nextcloud log in Admin > Logging:

No Errors in the Log

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'apps_paths' => 
  array (
    0 => 
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 => 
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' => 
  array (
    'host' => 'redis',
    'password' => '',
    'port' => 6379,
  ),
  'instanceid' => 'redacted',
  'passwordsalt' => 'redacted',
  'secret' => 'redacted',
  'trusted_domains' => 
  array (
    0 => 'cloud.tobi:8081',
  ),
  'datadirectory' => '/var/www/html/data',
  'dbtype' => 'mysql',
  'version' => '26.0.2.1',
  //proxy stuff
  'overwrite.cli.url' => 'https://cloud.tobi:8081',
  'overwritehost' => 'cloud.tobi:8081',
  'overwriteprotocol' => 'https',
  'trusted_proxies' => 
  array (
    0 => 'proxy',
    1 => 'nextcloud-proxy-1',
  ),
  //video thumb
  'enable_previews' => true,
  'enabledPreviewProviders' =>
  array (
    0 => 'OC\\Preview\\Movie',
    1 => 'OC\\Preview\\PNG',
    2 => 'OC\\Preview\\JPEG',
    3 => 'OC\\Preview\\GIF',
    4 => 'OC\\Preview\\BMP',
    5 => 'OC\\Preview\\XBitmap',
    6 => 'OC\\Preview\\MP3',
    7 => 'OC\\Preview\\MP4',
    8 => 'OC\\Preview\\TXT',
    9 => 'OC\\Preview\\MarkDown',
    10 => 'OC\\Preview\\PDF',
  ),
  //thumb Generation Empfehlung Memories Doku
  'preview_max_memory' => 4096,
  'preview_max_filesize_image' => 256,
  //Blöde Medlung aus Overview
  'default_phone_region' => 'DE',
  //------------------------------------------
  'dbname' => 'nextcloud',
  'dbhost' => 'db',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud',
  'dbpassword' => 'redacted',
  'installed' => true,
  'memories.exiftool' => '/var/www/html/custom_apps/memories/exiftool-bin/exiftool-amd64-glibc',
  'memories.vod.path' => '/var/www/html/custom_apps/memories/exiftool-bin/go-vod-amd64',
  'memories.gis_type' => 1,
  'maintenance' => false,
);

Nginx config:

worker_processes auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    # Prevent nginx HTTP Server Detection
    server_tokens   off;

    keepalive_timeout  65;

    #gzip  on;

    upstream php-handler {
        server app:9000;
    }

    server {
        listen 80;

        # HSTS settings
        # WARNING: Only add the preload option once you read about
        # the consequences in https://hstspreload.org/. This option
        # will add the domain to a hardcoded list that is shipped
        # in all major browsers and getting removed from this list
        # could take several months.
        #add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;" always;

        # set max upload size
	# Versuch Error 504 zu lösen, Body Size und Timeout erhöht
        client_max_body_size 10G;
        #client_body_buffer_size 400M;
        fastcgi_read_timeout 600s; #60s;
        fastcgi_buffers 64 4K;

        # Enable gzip but do not remove ETag headers
        gzip on;
        gzip_vary on;
        gzip_comp_level 4;
        gzip_min_length 256;
        gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
        gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;

        # Pagespeed is not supported by Nextcloud, so if your server is built
        # with the `ngx_pagespeed` module, uncomment this line to disable it.
        #pagespeed off;

        # HTTP response headers borrowed from Nextcloud `.htaccess`
        add_header Referrer-Policy                      "no-referrer"       always;
        add_header X-Content-Type-Options               "nosniff"           always;
        add_header X-Download-Options                   "noopen"            always;
        add_header X-Frame-Options                      "SAMEORIGIN"        always;
        add_header X-Permitted-Cross-Domain-Policies    "none"              always;
        add_header X-Robots-Tag                         "noindex, nofollow" always;
        add_header X-XSS-Protection                     "1; mode=block"     always;

        # Remove X-Powered-By, which is an information leak
        fastcgi_hide_header X-Powered-By;

        # Path to the root of your installation
        root /var/www/html;

        # Specify how to handle directories -- specifying `/index.php$request_uri`
        # here as the fallback means that Nginx always exhibits the desired behaviour
        # when a client requests a path that corresponds to a directory that exists
        # on the server. In particular, if that directory contains an index.php file,
        # that file is correctly served; if it doesn't, then the request is passed to
        # the front-end controller. This consistent behaviour means that we don't need
        # to specify custom rules for certain paths (e.g. images and other assets,
        # `/updater`, `/ocm-provider`, `/ocs-provider`), and thus
        # `try_files $uri $uri/ /index.php$request_uri`
        # always provides the desired behaviour.
        index index.php index.html /index.php$request_uri;

        # Rule borrowed from `.htaccess` to handle Microsoft DAV clients
        location = / {
            if ( $http_user_agent ~ ^DavClnt ) {
                return 302 /remote.php/webdav/$is_args$args;
            }
        }

        location = /robots.txt {
            allow all;
            log_not_found off;
            access_log off;
        }

        # Make a regex exception for `/.well-known` so that clients can still
        # access it despite the existence of the regex rule
        # `location ~ /(\.|autotest|...)` which would otherwise handle requests
        # for `/.well-known`.
        location ^~/.well-known {
            # The rules in this block are an adaptation of the rules
            # in `.htaccess` that concern `/.well-known`.

            location = /.well-known/carddav { return 301 /remote.php/dav/; }
            location = /.well-known/caldav  { return 301 /remote.php/dav/; }

            # Hab das hinzugefügt aus dieser Seite: https://github.com/nextcloud/documentation/issues/6157
            # according to the documentation these two lines are not necessary, but version 21.0.0 will produce warnings in the overview setup check
            location = /.well-known/webfinger   { return 301 /index.php$uri; }
            location = /.well-known/nodeinfo   { return 301 /index.php$uri; }

            location /.well-known/acme-challenge    { try_files $uri $uri/ =404; }
            location /.well-known/pki-validation    { try_files $uri $uri/ =404; }

            # Let Nextcloud's API for `/.well-known` URIs handle all other
            # requests by passing them to the front-end controller.
            return 301 /index.php$request_uri;
        }

        # Rules borrowed from `.htaccess` to hide certain paths from clients
        location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)(?:$|/)  { return 404; }
        location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console)                { return 404; }

        # Ensure this block, which passes PHP files to the PHP process, is above the blocks
        # which handle static assets (as seen below). If this block is not declared first,
        # then Nginx will encounter an infinite rewriting loop when it prepends `/index.php`
        # to the URI, resulting in a HTTP 500 error response.
        location ~ \.php(?:$|/) {
            # Required for legacy support
            rewrite ^/(?!index|remote|public|cron|core\/ajax\/update|status|ocs\/v[12]|updater\/.+|oc[ms]-provider\/.+|.+\/richdocumentscode\/proxy) /index.php$request_uri;

            fastcgi_split_path_info ^(.+?\.php)(/.*)$;
            set $path_info $fastcgi_path_info;

            try_files $fastcgi_script_name =404;

            include fastcgi_params;
            fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
            fastcgi_param PATH_INFO $path_info;
            #fastcgi_param HTTPS on;

            fastcgi_param modHeadersAvailable true;         # Avoid sending the security headers twice
            fastcgi_param front_controller_active true;     # Enable pretty urls
            fastcgi_pass php-handler;
            
            #Laut https://github.com/nextcloud/documentation/issues/6157 muss das aus sein:
            fastcgi_intercept_errors off;
            fastcgi_request_buffering off;
        }

        location ~ \.(?:css|js|svg|gif)$ {
            try_files $uri /index.php$request_uri;
            expires 6M;         # Cache-Control policy borrowed from `.htaccess`
            access_log off;     # Optional: Don't log access to assets
        }

        location ~ \.woff2?$ {
            try_files $uri /index.php$request_uri;
            expires 7d;         # Cache-Control policy borrowed from `.htaccess`
            access_log off;     # Optional: Don't log access to assets
        }

        # Rule borrowed from `.htaccess`
        location /remote {
            return 301 /remote.php$request_uri;
        }

        location / {
            try_files $uri $uri/ /index.php$request_uri;
        }
    }
}

Nginx Proxy config

server {
  server_name cloud.tobi;
  location / {
  proxy_set_header  Host $http_host;
  proxy_set_header  X-Real-IP $remote_addr;
  proxy_set_header  X-Forwarded-Proto https;
  proxy_set_header  X-Forwarded-Host $remote_addr;
  proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_pass http://nextcloud-web-1:80;
  proxy_buffers 16 4k;
  proxy_buffer_size 2k;
  }

  #Fix webfinger etc,
  location /.well-known/carddav {
    return 301 $scheme://$http_host/remote.php/dav;
  }

  location /.well-known/caldav {
    return 301 $scheme://$http_host/remote.php/dav;
  }

  location /.well-known/webfinger {
    return 301 $scheme://$http_host/index.php/.well-known/webfinger;
  }

  location /.well-known/nodeinfo {
    return 301 $scheme://$http_host/index.php/.well-known/nodeinfo;
  }

  fastcgi_intercept_errors off;

  # Versuch Error 504 zu lösen, Body Size und Timeout erhöht
  proxy_buffering off;
  client_max_body_size 10G;
  #client_body_buffer_size 400M;
  fastcgi_read_timeout 600s;
  fastcgi_request_buffering off;

  listen [::]:443 ssl http2;
  listen 443 ssl http2;
  ssl_certificate /etc/nginx/certs/cloud.tobi.crt;
  ssl_certificate_key /etc/nginx/certs/cloud.tobi.key;
}

server {
  if ($http_host = cloud.tobi) {
      return 301 https://$http_host$request_uri;
  }

  #Fix webfinger etc,
  fastcgi_intercept_errors off;

  location /.well-known/carddav {
    return 301 $scheme://$http_host/remote.php/dav;
  }

  location /.well-known/caldav {
    return 301 $scheme://$http_host/remote.php/dav;
  }

  location /.well-known/webfinger {
    return 301 $scheme://$http_host/index.php/.well-known/webfinger;
  }

  location /.well-known/nodeinfo {
    return 301 $scheme://$http_host/index.php/.well-known/nodeinfo;
  }

  # Versuch Error 504 zu lösen, Body Size und Timeout erhöht
  client_max_body_size 10G;
  proxy_buffering off;
  #client_body_buffer_size 400M;
  fastcgi_read_timeout 600s;
  fastcgi_request_buffering off;

  server_name cloud.tobi;
  listen [::]:80;
  listen 80;
  return 404;
}

Docker compose:

version: '2'

services:
  db:
    image: mariadb:10.6
    restart: always
    command: --transaction-isolation=READ-COMMITTED --log-bin=binlog --binlog-format=ROW
    volumes:
      - /appdata/nextcloud/db:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=redacted
      - MYSQL_PASSWORD=redacted
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
      - MARIADB_AUTO_UPGRADE=1
      - MARIADB_DISABLE_UPGRADE_BACKUP=1

  redis:
    image: redis:alpine
    restart: always

  app:
    image: nextcloud-custom:fpm-26
    restart: always
    links:
      - db
    volumes:
      - /appdata/nextcloud/app:/var/www/html:z
      - /appdata/nextcloud/tmp-app:/tmp
      - /srv/dev-disk-by-uuid-cc6e214c-17e7-40d9-9e23-f7fdc7dc5450/NextcloudData:/var/www/html/data
    environment:
      - MYSQL_PASSWORD=redacted
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
      - MYSQL_HOST=db
      - NEXTCLOUD_TRUSTED_DOMAINS=cloud.tobi
      - PHP_MEMORY_LIMIT=4G
      - PHP_UPLOAD_LIMIT=16G
      - REDIS_HOST=redis
    depends_on:
      - db
      - redis

  web:
    image: nginx
    restart: always
    links:
      - app
    volumes:
      - /appdata/nextcloud/web/nginx.conf:/etc/nginx/nginx.conf:ro
      - /appdata/nextcloud/tmp-web:/tmp
    volumes_from:
      - app
    environment:
      - VIRTUAL_HOST=cloud.tobi
    depends_on:
      - app
    networks:
      - proxy-tier
      - default

      
#  cron:
#    image: nextcloud-custom:fpm-26
#    restart: always
#    volumes:
#      - /appdata/nextcloud/app:/var/www/html:z
#    entrypoint: /cron.sh
#    depends_on:
#      - db
#      - redis

  proxy:
    image: nginxproxy/nginx-proxy #:alpine
    restart: always
    ports:
      - 8080:80
      - 8081:443
    volumes:
      - /appdata/nextcloud/proxy:/etc/nginx/conf.d
      - /appdata/nextcloud/ssl:/etc/nginx/certs:ro
#      - vhost.d:/etc/nginx/vhost.d:z
#      - html:/usr/share/nginx/html:z
      - /var/run/docker.sock:/tmp/docker.sock:z,ro
      - /appdata/nextcloud/tmp-proxy:/tmp
    networks:
      - proxy-tier

networks:
  proxy-tier:

Has anybody a idea why I get these errors? Sorry for any spelling errors its very late.

Additional info: The config file of my proxy is call just like my domain “cloud.tobi.conf”.

I found this error in the proxy log:

nginx.1 | 2023/06/23 21:29:41 [error] 33#33: *1 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.3.0.1, server: cloud.tobi, request: “MOVE /remote.php/dav/uploads/admin/4205298899/.file HTTP/1.1”, upstream: “http://172.22.0.2:80/remote.php/dav/uploads/admin/4205298899/.file”, host: “cloud.tobi:8081”

How long does it take for that error to pop-up in the client?

Can you share the associated entries from your nextcloud.log from immediately prior to the failure?

As for general performance, what is performance like if you use the Web interface rather than the client? For that matter, can you upload files >500GB via the Web interface?

Also, what is latency (ping) between the server and the client?

Thank you for the quick response.

If I upload multiple files the client seams to just upload the small files and sometimes just ignore the large files for a while. But I think the error appears after it has finished uploading the files to the server and starts reassembling the chunks.
When I uploaded a big file (12,3 GB .mp4 file) it was when the progressbar reached the end.

I have added the folowing line to my nginx reverse proxy and I was able to upload serveral files with over 1GB (max 2GB) without the error:

proxy_read_timeout 3600;

I think the nextcloud server was taking to long to process the chunks after the upload and the proxy thus responded with error 504.

A ping to my server takes less than 1ms (it is on the same network and not reachable from the internet).

I tested using a 1,32GB file:
Desktop client (Not looking a the systemtray GUI):
Upload: 1minute and 24seconds
Download: 4minutes 30seconds (according to Windows 4.63Mb/s)
Browser:
Upload: 32seconds
Download: 23seconds
I think this shows the differenz quite well.

I will look for the log tomorrow and throw a 12,3GB file at it and test if it breaks again. Maybe the chunk size has something to do with it.

I also don’t understand the RAM usage. It’s basically allways at 1GB of 8GB.

I was able to upload the 12,3GB big file over the client without the error. The performance still isn’t great. Any Ideas?