Timeouts while uploading multiple files

Hi,

Update
I forgot to mention that I’m using traefik as an entrance to my server.
If I access the docker on his IP like http://192.168.1.100:9050 I can upload files without a problem.
But I want to keep Traefik, so if anyone can still help, would be great

I’m trying to setup Nextcloud to save my photos (RAW and JPG), but I’m having an issue with drag and drop to the browser client.

If I drag and drop few images (both RAW and JPG at the same time), the images are correctly uploaded.

But if I try to upload multiple files (like 20 files), the requests start to hang and they fail, causing the images to not be sent. In the network requests, they end up after one minute with a 502 error (nothing on the response data).

After the timeouts, some files are uploaded, others are not. From around 20 files,it managed to upload some 12 for example.

Then on the logs I see the following errors which I think are caused because of the timeout:

`

{“reqId”:“l0IzJPVWLLRZrpowAAwr”,“level”:4,“time”:“August 27, 2020 10:24:38”,“remoteAddr”:“172.26.0.1”,“user”:“retronwarz”,“app”:“webdav”,“method”:“PUT”,“url”:"/remote.php/dav/uploads/retronwarz/web-file-upload-cbb12276a6ff2bdf8ef0088c8f09cf06-1598523816447/0",“message”:{“Exception”:“Sabre\DAV\Exception\BadRequest”,“Message”:“Expected filesize of 10485760 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 2555904 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.”,“Code”:0,“Trace”:[{“file”:"/var/www/html/apps/dav/lib/Connector/Sabre/Directory.php",“line”:154,“function”:“put”,“class”:“OCA\DAV\Connector\Sabre\File”,“type”:"->",“args”:[null]},{“file”:"/var/www/html/apps/dav/lib/Upload/UploadFolder.php",“line”:46,“function”:“createFile”,“class”:“OCA\DAV\Connector\Sabre\Directory”,“type”:"->",“args”:[“0”,null]},{“file”:"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php",“line”:1104,“function”:“createFile”,“class”:“OCA\DAV\Upload\UploadFolder”,“type”:"->",“args”:[“0”,null]},{“file”:"/var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php",“line”:527,“function”:“createFile”,“class”:“Sabre\DAV\Server”,“type”:"->",“args”:[“uploads/retronwarz/web-file-upload-cbb12276a6ff2bdf8ef0088c8f09cf06-1598523816447/0”,null,null]},{“file”:"/var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php",“line”:89,“function”:“httpPut”,“class”:“Sabre\DAV\CorePlugin”,“type”:"->",“args”:[{“class”:“Sabre\HTTP\Request”},{“class”:“Sabre\HTTP\Response”}]},{“file”:"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php",“line”:474,“function”:“emit”,“class”:“Sabre\DAV\Server”,“type”:"->",“args”:[“method:PUT”,[{“class”:“Sabre\HTTP\Request”},{“class”:“Sabre\HTTP\Response”}]]},{“file”:"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php",“line”:251,“function”:“invokeMethod”,“class”:“Sabre\DAV\Server”,“type”:"->",“args”:[{“class”:“Sabre\HTTP\Request”},{“class”:“Sabre\HTTP\Response”}]},{“file”:"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php",“line”:319,“function”:“start”,“class”:“Sabre\DAV\Server”,“type”:"->",“args”:[]},{“file”:"/var/www/html/apps/dav/lib/Server.php",“line”:320,“function”:“exec”,“class”:“Sabre\DAV\Server”,“type”:"->",“args”:[]},{“file”:"/var/www/html/apps/dav/appinfo/v2/remote.php",“line”:35,“function”:“exec”,“class”:“OCA\DAV\Server”,“type”:"->",“args”:[]},{“file”:"/var/www/html/remote.php",“line”:167,“args”:["/var/www/html/apps/dav/appinfo/v2/remote.php"],“function”:“require_once”}],“File”:"/var/www/html/apps/dav/lib/Connector/Sabre/File.php",“Line”:238,“CustomMessage”:"–"},“userAgent”:“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135 Safari/537.36”,“version”:“19.0.1.1”}

`

I’m running everything in docker containers in a unraid server with 10tb HD space, 16GB RAM and a Ryzen 3700x.

Here is my docker-compose.yml:

version: '3'

services:
  db:
    container_name: Nextcloud-DB
    image: mariadb
    command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
    restart: always
    volumes:
      - "${SETTINGS}/NextcloudFPM/db:/var/lib/mysql"
    env_file:
      - db.env
    ports:
      - 3306:3306
    environment:
      - PUID=99
      - PGID=100

  redis:
    container_name: Nextcloud-Redis
    image: redis:alpine
    command: redis-server --requirepass 123456789
    restart: always
    ports:
      - 6379:6379
    environment:
      - PUID=99
      - PGID=100

  app:
    container_name: Nextcloud-FPM
    image: nextcloud:fpm-alpine
    restart: always
    volumes:
      - "${SETTINGS}/NextcloudFPM/www:/var/www/html/"
      - "/mnt/user/Cloud:/var/www/html/data"
    environment:
      - MYSQL_HOST=db
      - REDIS_HOST=redis
      - REDIS_HOST_PASSWORD=123456789
      - PUID=99
      - PGID=100
    env_file:
      - db.env
    depends_on:
      - db
      - redis

  web:
    container_name: Nextcloud-Web
    build: ./web
    restart: always
    ports:
      - 9050:80
    volumes:
      - '${SETTINGS}/NextcloudFPM/www:/var/www/html:ro'
      - '${SETTINGS}/NextcloudFPM/tmp:/var/www/tmp'
      - '${SETTINGS}/NextcloudFPM/nginx/nginx.conf:/etc/nginx/nginx.conf'
    depends_on:
      - app
    environment:
      - PUID=99
      - PGID=100
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.nextcloud-http.entrypoints=http"
      - "traefik.http.routers.nextcloud-http.rule=Host(`cloud.${DOMAIN}`)"
      - "traefik.http.routers.nextcloud-http.middlewares=https-redirect@file"

      - "traefik.http.routers.nextcloud-https.entrypoints=https"
      - "traefik.http.routers.nextcloud-https.tls=true"

      - "traefik.http.routers.nextcloud-https.rule=Host(`cloud.${DOMAIN}`)"
      - "traefik.http.routers.nextcloud-https=nextcloud"
      - "traefik.http.routers.nextcloud-https.service=nextcloud"

      - "traefik.http.services.nextcloud.loadbalancer.server.port=80"

  cron:
    container_name: Nextcloud-Crons
    image: nextcloud:fpm-alpine
    restart: always
    volumes:
      - "${SETTINGS}/NextcloudFPM/www:/var/www/html/"
    entrypoint: /cron.sh
    depends_on:
      - db
      - redis
    environment:
      - PUID=99
      - PGID=100

volumes:
  db:
  nextcloud:

My config.php file:

<?php
$CONFIG = array(
    'memcache.local' => '\\OC\\Memcache\\APCu',
    'apps_paths' =>
        array(
            0 =>
                array(
                    'path' => '/var/www/html/apps',
                    'url' => '/apps',
                    'writable' => false,
                ),
            1 =>
                array(
                    'path' => '/var/www/html/custom_apps',
                    'url' => '/custom_apps',
                    'writable' => true,
                ),
        ),
    'memcache.distributed' => '\\OC\\Memcache\\Redis',
    'memcache.locking' => '\\OC\\Memcache\\Redis',
    'redis' =>
        array(
            'host' => 'redis',
            'password' => '123456789',
            'port' => 6379,
            'timeout' => 0
        ),
    'instanceid' => 'ocmmnsr8ewqo',
    'passwordsalt' => 'mENsGih2ASWuoeXF4mIT7WmN3+pyKS',
    'secret' => '+3KTbyOG2m96xHpDiK+ptVnJSdc0fuM5Y17p1lXkbu2FdWig',
    'trusted_domains' =>
        array(
            0 => 'cloud.domain.com',
            1 => '172.26.0.*',
            2 => '192.168.1.*'
        ),
    'datadirectory' => '/var/www/html/data',
    'dbtype' => 'mysql',
    'version' => '19.0.1.1',
    'overwrite.cli.url' => 'http://cloud.domain.com',
    'dbname' => 'nextcloud',
    'dbhost' => 'db',
    'dbport' => '',
    'dbtableprefix' => 'oc_',
    'mysql.utf8mb4' => true,
    'dbuser' => 'user',
    'dbpassword' => 'password',
    'installed' => true,
    'filelocking.enabled' => true,
    'debug' => true,
    "log_type" => "file",
    "logfile" => "/var/www/html/error_loggin.log",
    'logfilemode' => 0640,
    "loglevel" => "2",
    "logdateformat" => "F d, Y H:i:s"
);

And I also changed the timeout limits on the nginx

worker_processes auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  800;
    proxy_connect_timeout 600s;
    proxy_send_timeout 600s;
    proxy_read_timeout 600s;
    fastcgi_send_timeout 600s;
    fastcgi_read_timeout 600s;

    #gzip  on;

    upstream php-handler {
        server app:9000;
    }

    server {
        listen 80;

        proxy_connect_timeout       605;
        proxy_send_timeout          605;
        proxy_read_timeout          605;
        send_timeout                605;
        keepalive_timeout           605;

        # Add headers to serve security related headers
        # Before enabling Strict-Transport-Security headers please read into this
        # topic first.
        #add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;" always;
        #
        # WARNING: Only add the preload option once you read about
        # the consequences in https://hstspreload.org/. This option
        # will add the domain to a hardcoded list that is shipped
        # in all major browsers and getting removed from this list
        # could take several months.
        add_header Referrer-Policy "no-referrer" always;
        add_header X-Content-Type-Options "nosniff" always;
        add_header X-Download-Options "noopen" always;
        add_header X-Frame-Options "SAMEORIGIN" always;
        add_header X-Permitted-Cross-Domain-Policies "none" always;
        add_header X-Robots-Tag "none" always;
        add_header X-XSS-Protection "1; mode=block" always;

        # Remove X-Powered-By, which is an information leak
        fastcgi_hide_header X-Powered-By;

        # Path to the root of your installation
        root /var/www/html;

        location = /robots.txt {
            allow all;
            log_not_found off;
            access_log off;
        }

        # The following 2 rules are only needed for the user_webfinger app.
        # Uncomment it if you're planning to use this app.
        #rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
        #rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json last;

        # The following rule is only needed for the Social app.
        # Uncomment it if you're planning to use this app.
        #rewrite ^/.well-known/webfinger /public.php?service=webfinger last;

        location = /.well-known/carddav {
            return 301 $scheme://$host:$server_port/remote.php/dav;
        }

        location = /.well-known/caldav {
            return 301 $scheme://$host:$server_port/remote.php/dav;
        }

        # set max upload size
        client_max_body_size 10G;
        fastcgi_buffers 64 4K;

        # Enable gzip but do not remove ETag headers
        gzip on;
        gzip_vary on;
        gzip_comp_level 4;
        gzip_min_length 256;
        gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
        gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;

        # Uncomment if your server is build with the ngx_pagespeed module
        # This module is currently not supported.
        #pagespeed off;

        location / {
            rewrite ^ /index.php;
        }

        location ~ ^\/(?:build|tests|config|lib|3rdparty|templates|data)\/ {
            deny all;
        }
        location ~ ^\/(?:\.|autotest|occ|issue|indie|db_|console) {
            deny all;
        }

        location ~ ^\/(?:index|remote|public|cron|core\/ajax\/update|status|ocs\/v[12]|updater\/.+|oc[ms]-provider\/.+)\.php(?:$|\/) {
            fastcgi_split_path_info ^(.+?\.php)(\/.*|)$;
            set $path_info $fastcgi_path_info;
            try_files $fastcgi_script_name =404;
            include fastcgi_params;
            fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
            fastcgi_param PATH_INFO $path_info;
            # fastcgi_param HTTPS on;

            # Avoid sending the security headers twice
            fastcgi_param modHeadersAvailable true;

            # Enable pretty urls
            fastcgi_param front_controller_active true;
            fastcgi_pass php-handler;
            fastcgi_intercept_errors on;
            fastcgi_request_buffering off;
        }

        location ~ ^\/(?:updater|oc[ms]-provider)(?:$|\/) {
            try_files $uri/ =404;
            index index.php;
        }

        # Adding the cache control header for js, css and map files
        # Make sure it is BELOW the PHP block
        location ~ \.(?:css|js|woff2?|svg|gif|map)$ {
            try_files $uri /index.php$request_uri;
            add_header Cache-Control "public, max-age=15778463";
            # Add headers to serve security related headers (It is intended to
            # have those duplicated to the ones above)
            # Before enabling Strict-Transport-Security headers please read into
            # this topic first.
            #add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;" always;
            #
            # WARNING: Only add the preload option once you read about
            # the consequences in https://hstspreload.org/. This option
            # will add the domain to a hardcoded list that is shipped
            # in all major browsers and getting removed from this list
            # could take several months.
            add_header Referrer-Policy "no-referrer" always;
            add_header X-Content-Type-Options "nosniff" always;
            add_header X-Download-Options "noopen" always;
            add_header X-Frame-Options "SAMEORIGIN" always;
            add_header X-Permitted-Cross-Domain-Policies "none" always;
            add_header X-Robots-Tag "none" always;
            add_header X-XSS-Protection "1; mode=block" always;

            # Optional: Don't log access to assets
            access_log off;
        }

        location ~ \.(?:png|html|ttf|ico|jpg|jpeg|bcmap|mp4|webm)$ {
            try_files $uri /index.php$request_uri;
            # Optional: Don't log access to other assets
            access_log off;
        }
    }
}

And I changed the following in my .user.ini:

mbstring.func_overload=0
always_populate_raw_post_data=-1
default_charset='UTF-8'
output_buffering=0
max_input_time=3600
max_execution_time=6000
request_terminate_timeout=6000
sys_temp_dir="/var/www/tmp"

memory_limit=8000M
upload_max_filesize=16G
client_max_body_size=16G
post_max_size=16G

I’ve been looking for a solution all night long and couldn’t find anything or why this is happening… Has anyone had this issue, or knows where I should look to search for more information on the error?

Thank you

Unfortunately I have a very similar problem and not a solution. In my case:

ha-proxy (on pfsense) -> nginx -> php-fpm

I can upload individual files (also very big ones) without a problem. However if I upload many files in one go, some work, others result in the BadException problem and in some cases I have to restart the nextcloud instance to get things going again (sometimes even with lock problems etc.).

If I access the nginx directly internally without WAN -> ha-proxy, all is fine. So this at least looks as if it is somehow dependent on the loadbalancer/proxy in front of the nginx. I cannot see anything in the debug logs on the ha-proxy other than that after a while the nginx seems to give back a 5xx server error (after quite a long time).

Any idea would be incredibly helpful as this is driving me nuts and is limiting the functionality very much. Oh and if I am not using the web interface but a nextcloud sync client, all is well (also through ha-proxy)…

I read somewhere that disabling http2 might solve the problem. Guess what: It did for me. I disabled http2 in nginx and ha-proxy and all is fine now!

1 Like

I have the same problem…