504 Gateway Timeout Error When Uploading 2GB+ Files

Mathews411 · August 10, 2020, 12:19am

Nextcloud version (eg, 18.0.2): 18.0.7
Operating system and version (eg, Ubuntu 20.04): UnRAID with linuxserver/nextcloud docker container
Apache or nginx version (eg, Apache 2.4.25):
PHP version (eg, 7.1): 7.3.20

The issue you are facing:

So I am running NextCloud within a docker container on my UnRAID system, which utilizes MariaDB to supply a mysql database, along with using Redis (if that helps). So far, everything with the app is working perfectly. It’s accessible from my reverse proxy (Using NGINX Proxy Manager Container) , I can upload/download files from it, and I have no Security/Setup Warnings. However, I am constantly running into an issue whenever I try and upload files larger than 2GB from the client.

The strange thing about all of this, is that after I receive my “504 Gateway Timeout on MOVE” (or PUT) error, if I wait around 15 minutes, the file will eventually be synced to NextCloud. Though I am happy the file is uploaded, I would like to prevent these errors from popping up.

I have tried to compile as much information as I can below. I originally implemented this 2 months ago, and fixing this error has been on my “to do” list for quite some time. I apologize if there is anything missing or if I have missed something blindly obvious. I am still really new to NextCloud .

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:

Open either Client or Web Portal
Upload any file larger than 2.0GB
Wait until the error starts
Wait another 15 minutes and the file uploads

The output of your Nextcloud log in Admin > Logging:

[no app in context] Error: Sabre\DAV\Exception\ServiceUnavailable: Could not open file at <<closure>>

 0. /config/www/nextcloud/apps/dav/lib/Upload/AssemblyStream.php line 293
    OCA\DAV\Connector\Sabre\File->get()
 1. /config/www/nextcloud/apps/dav/lib/Upload/AssemblyStream.php line 163
    OCA\DAV\Upload\AssemblyStream->getStream(OCA\DAV\Connector\Sabre\File {})
 2. <<closure>>
    OCA\DAV\Upload\AssemblyStream->stream_read(8192)
 3. /config/www/nextcloud/3rdparty/icewind/streams/src/Wrapper.php line 91
    fread(null, 8192)
 4. /config/www/nextcloud/3rdparty/icewind/streams/src/CallbackWrapper.php line 98
    Icewind\Streams\Wrapper->stream_read(8192)
 5. <<closure>>
    Icewind\Streams\CallbackWrapper->stream_read(8192)
 6. /config/www/nextcloud/lib/private/legacy/helper.php line 256
    fread(null, 8192)
 7. /config/www/nextcloud/lib/private/Files/Storage/Wrapper/Encryption.php line 1038
    OC_Helper::streamCopy(null, null)
 8. /config/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php line 630
    OC\Files\Storage\Wrapper\Encryption->writeStream("files/Batman Un ... t", null, null)
 9. /config/www/nextcloud/apps/dav/lib/Connector/Sabre/File.php line 192
    OC\Files\Storage\Wrapper\Wrapper->writeStream("files/Batman Un ... t", null)
10. /config/www/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php line 156
    OCA\DAV\Connector\Sabre\File->put(null)
11. /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Tree.php line 316
    OCA\DAV\Connector\Sabre\Directory->createFile("Batman Under th ... v", null)
12. /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Tree.php line 130
    Sabre\DAV\Tree->copyNode(OCA\DAV\Upload\FutureFile {}, OCA\DAV\Files\FilesHome {}, "Batman Under th ... v")
13. /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Tree.php line 161
    Sabre\DAV\Tree->copy("uploads/rmathews/2589610512/.file", "files/rmathews/ ... v")
14. /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php line 642
    Sabre\DAV\Tree->move("uploads/rmathews/2589610512/.file", "files/rmathews/ ... v")
15. <<closure>>
    Sabre\DAV\CorePlugin->httpMove(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
16. /config/www/nextcloud/3rdparty/sabre/event/lib/EventEmitterTrait.php line 105
    call_user_func_array([Sabre\DAV\CorePlugin {},"httpMove"], [Sabre\HTTP\Requ ... }])
17. /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 479
    Sabre\Event\EventEmitter->emit("method:MOVE", [Sabre\HTTP\Requ ... }])
18. /config/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 254
    Sabre\DAV\Server->invokeMethod(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
19. /config/www/nextcloud/apps/dav/lib/Server.php line 319
    Sabre\DAV\Server->exec()
20. /config/www/nextcloud/apps/dav/appinfo/v2/remote.php line 35
    OCA\DAV\Server->exec()
21. /config/www/nextcloud/remote.php line 165
    require_once("/config/www/nex ... p")

MOVE /remote.php/dav/uploads/rmathews/2589610512/.file
from 10.10.150.1 by rmathews at 2020-08-09T23:45:45+00:00

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

$CONFIG = array (
  'memcache.local' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' => 
  array (
    'host' => '10.10.150.5',
    'port' => 6379,
  ),
  'datadirectory' => '/data',
  'instanceid' => **Removed**,
  'passwordsalt' => **Removed**,
  'secret' => **Removed**,
  'trusted_domains' => 
  array (
    0 => '10.10.150.5:444',
    1 => '**Removed**',
  ),
  'dbtype' => 'mysql',
  'version' => '18.0.7.1',
  'overwrite.cli.url' => **Removed**,
  'overwritehost' => **Removed**,
  'overwriteprotocol' => 'https',
  'dbname' => 'nextcloud',
  'dbhost' => '10.10.150.5:3306',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => **Removed**
  'dbpassword' => **Removed**
  'installed' => true,
  'mail_smtpmode' => 'smtp',
  'mail_sendmailmode' => 'pipe',
  'maintenance' => false,
  'theme' => '',
  'loglevel' => 2,
);

The output of your Apache/nginx/system log in /var/log/____:

*I am not sure where to find this within UnRAID, but the errors.log I found within \var\logs\nginx\ the console of my Container*

simonspa · August 10, 2020, 3:50pm

Hi,

your posted error log seems to be cut at the end of the post - I guess that would have been the most interesting part. From what you describe it very likely has nothing to do with Nextcloud but lower layers of the stack. It could be

PHP that runs out of memory (what is your PHP memory setting?)
PHP that runs into a timeout
Webserver that runs into timeout for client connection

Could you check for your PHP and webserver settings?

/S

Mathews411 · August 12, 2020, 1:31am

Hello Simonspa,

I have made a few edits to my PHP file. I have increased the Memory usage from 512MB to 2GB. I cannot seem to find my file within UNRAID for my PHP Config (sorry about that)

I did find my Webserver settings under the NGINX Default Site Configs:

upstream php-handler {
server 127.0.0.1:9000;
}
server {
listen 80;
listen [::]:80;
server_name _;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name _;
ssl_certificate /config/keys/cert.crt;
ssl_certificate_key /config/keys/cert.key;

# Add headers to serve security related headers
# Before enabling Strict-Transport-Security headers please read into this
# topic first.
add_header Strict-Transport-Security "max-age=15552000; includeSubDomains" always;
#
# WARNING: Only add the preload option once you read about
# the consequences in https://hstspreload.org/. This option
# will add the domain to a hardcoded list that is shipped
# in all major browsers and getting removed from this list
# could take several months.

add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
add_header X-Robots-Tag none;
add_header X-Download-Options noopen;
add_header X-Frame-Options "SAMEORIGIN";
add_header X-Permitted-Cross-Domain-Policies none;
add_header Referrer-Policy no-referrer;
fastcgi_hide_header X-Powered-By;
root /config/www/nextcloud/;
# display real ip in nginx logs when connected through reverse proxy via docker network
set_real_ip_from 172.0.0.0/8;
real_ip_header X-Forwarded-For;
location = /robots.txt {
    allow all;
    log_not_found off;
    access_log off;
}
location = /.well-known/carddav {
  return 301 $scheme://$host:$server_port/remote.php/dav;
}
location = /.well-known/caldav {
  return 301 $scheme://$host:$server_port/remote.php/dav;
}
client_max_body_size 10G;
fastcgi_buffers 64 4K;
gzip on;
gzip_vary on;
gzip_comp_level 4;
gzip_min_length 256;
gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;
location / {
    rewrite ^ /index.php;
}
location ~ ^\/(?:build|tests|config|lib|3rdparty|templates|data)\/ {
    deny all;
}
location ~ ^\/(?:\.|autotest|occ|issue|indie|db_|console) {
    deny all;
}
location ~ ^\/(?:index|remote|public|cron|core\/ajax\/update|status|ocs\/v[12]|updater\/.+|oc[ms]-provider\/.+)\.php(?:$|\/) {
    fastcgi_split_path_info ^(.+?\.php)(\/.*|)$;
    set $path_info $fastcgi_path_info;
    try_files $fastcgi_script_name =404;
    include /etc/nginx/fastcgi_params;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_param PATH_INFO $path_info;
    fastcgi_param HTTPS on;
    fastcgi_param modHeadersAvailable true;
    fastcgi_param front_controller_active true;
    fastcgi_pass php-handler;
    fastcgi_intercept_errors on;
    fastcgi_request_buffering off;
}
location ~ ^\/(?:updater|oc[ms]-provider)(?:$|\/) {
    try_files $uri/ =404;
    index index.php;
}
location ~ \.(?:css|js|woff2?|svg|gif|map)$ {
    try_files $uri /index.php$request_uri;
    add_header Cache-Control "public, max-age=15778463";
    # Add headers to serve security related headers
    # Before enabling Strict-Transport-Security headers please read into this
    # topic first.
    #add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;" always;
    #
    # WARNING: Only add the preload option once you read about
    # the consequences in https://hstspreload.org/. This option
    # will add the domain to a hardcoded list that is shipped
    # in all major browsers and getting removed from this list
    # could take several months.
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";
    add_header X-Robots-Tag none;
    add_header X-Download-Options noopen;
    add_header X-Frame-Options "SAMEORIGIN";
    add_header X-Permitted-Cross-Domain-Policies none;
    add_header Referrer-Policy no-referrer;
    access_log off;
}
location ~ \.(?:png|html|ttf|ico|jpg|jpeg|bcmap)$ {
    try_files $uri /index.php$request_uri;
    access_log off;
}

}

I’m not sure if there is anything else I can provide that could be of much more help

simonspa · August 12, 2020, 10:32am

This all looks okay, maybe hunt for the PHP config some more?