Problem uploading large files

xokia · November 6, 2023, 2:07am

Nextcloud version : Nextcloud Hub 6 27.1.3
Operating system and version : Unraid 6.12.4 nextcloud in docker
Apache or nginx version : nginx via swag 2.7.4
PHP version (eg, 7.4): php8.2

The issue you are facing:
I can upload 900MB fine. If I try to upload 2GB the upload completes but then get unknown error message. No error in logfile.

Is this the first time you’ve seen this error? (Y/N): N

Steps to replicate it:

upload a file larger then 1GB

The output of your Nextcloud log in Admin > Logging:

PASTE HERE

The output of your config.php file in /path/to/nextcloud :

<?php
$CONFIG = array (
  'datadirectory' => '/data',
  'instanceid' => 'xxxxxxxxxxxxxx',
  'passwordsalt' => 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
  'secret' => 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
  'trusted_domains' =>
  array (
    0 => '192.168.3.12:444',
    1 => 'xxxxxxxxxxxxxx.com',
  ),
  'trusted_proxies' =>
  array (
    0 => 'swag',
  ),
  'default_phone_region' => 'US',
  'dbtype' => 'mysql',
  'version' => '27.1.3.2',
  'overwrite.cli.url' => 'https://xxxxxxxx.com',
  'overwritehost' => 'xxxxxxxxxxxx.com',
  'overwriteprotocol' => 'https',
  'dbname' => 'nextcloud',
  'dbhost' => '192.168.3.12:3306',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud',
  'dbpassword' => 'xxxxxxxxxxxxxx',
  'installed' => true,
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'filelocking.enabled' => true,
  'memcache.locking' => '\\OC\\Memcache\\APCu',
  'upgrade.disable-web' => true,
  'mail_from_address' => 'xxxxxxx',
  'mail_smtpmode' => 'smtp',
  'mail_sendmailmode' => 'smtp',
  'mail_domain' => 'yahoo.com',
  'mail_smtphost' => 'xxxxxxxxxxx',
  'mail_smtpport' => '465',
  'mail_smtpauth' => 1,
  'mail_smtpname' => 'xxxxxxxxx',
  'mail_smtppassword' => 'xxxxxxxxxxxxxxxxx',
  'maintenance' => false,
  'app_install_overwrite' =>
  array (
    0 => 'flowupload',
  ),
);
~

The output of your Apache/nginx/system log in /var/log/____:

empty


Output errors in nextcloud.log in /var/www/ or as admin user in top right menu, filtering for errors. Use a pastebin service if necessary.

empty

I modified /etc/php82/conf.d/nextcloud.ini
memory_limit=2048M
upload_max_filesize=70G
post_max_size=70G
max_input_time=6400
max_execution_time=6400

dugite-code · November 6, 2023, 3:04am

Probably nginx and the client_max_body_size setting

There was the flowupload app that added a js based uploader to improve large file uploads but that hasn’t seen updates for the latest Nextcloud version.

xokia · November 6, 2023, 3:22am

flowupload no longer works with the latest nextcloud last supported version was 23. I am on version 27 of nextcloud. I tried installing it. It expected an older version of nextcloud.

I changed the client_max_body_size
/mnt/user/appdata/swag/nginx/proxy-confs/nextcloud.subdomain.conf

in my experience if its an nginx issue the file will get rejected at the point of upload. Not after its finished uploading.

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name nextcloud.* opencloud.*;

    include /config/nginx/ssl.conf;
    add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";
    client_max_body_size 70G;
    fastcgi_buffers 64 4k;
    #client_body_temp_path /tmp/nginx_client_bodies;

    location / {
        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;
        set $upstream_app 192.168.3.12;
        set $upstream_port 444;
        set $upstream_proto https;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

        # Hide proxy response headers from Nextcloud that conflict with ssl.conf
        # Uncomment the Optional additional headers in SWAG's ssl.conf to pass Nextcloud's security scan
        proxy_hide_header Referrer-Policy;
        proxy_hide_header X-Content-Type-Options;
        proxy_hide_header X-Frame-Options;
        proxy_hide_header X-XSS-Protection;

        # Disable proxy buffering
        proxy_buffering off;
        fastcgi_intercept_errors on;
        fastcgi_request_buffering off;
        fastcgi_connect_timeout 60;
        fastcgi_send_timeout 1800;
        fastcgi_read_timeout 1800;
        #proxy_request_buffering off;

        #proxy_max_temp_file_size 0;
    }
}

For me it looks as if it uploads the file fine. It takes the appropriate amount of time and I can see the network traffic I expect when its uploading. What it looks like to me is when it tries to create the file entry in the database it’s somehow failing? I get unknown error and no entry in the log file so almost impossible to debug.

dugite-code · November 6, 2023, 5:20am

Which nextcloud docker container are you using? AIO Linux-server or the community Nextcloud docker container?

The only suggestion I can think of is the your config doesn’t have a loglevel set, try setting it to debug (level 0) and see if you get any useful info.

"loglevel" => 0,

https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/logging_configuration.html#log-level

I feel like I remember having a similar issue once and it had to do with temporary files, but I can’t find it in my notes.

xokia · November 6, 2023, 6:13am

I am using the linux-server.io not the community version

namespace OCA\FirstRunWizard\Settings;

use OCP\AppFramework\Http\TemplateResponse;
use OCP\IConfig;
use OCP\Settings\ISettings;
use OCP\IURLGenerator;

class Personal implements ISettings {

        /** @var IConfig */
        private $config;
        /** @var \OC_Defaults */
        private $defaults;

        public function __construct(IConfig $config, IURLGenerator $urlGenerator, \OC_Defaults $defaults) {
                $this->config = $config;
                $this->urlGenerator = $urlGenerator;
                $this->defaults = $defaults;
        }

for proxy temp I have tried to add that but no luck
/mnt/user/appdata/swag/nginx/proxy-confs/nextcloud.subdomain.conf

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name nextcloud.* opencloud.*;

    include /config/nginx/ssl.conf;
    add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";
    client_max_body_size 0;
    #fastcgi_buffers 64 4k;
    #client_body_temp_path client_body_temp;
    #client_body_in_file_only clean;
    #client_body_in_single_buffer on;
    #proxy_request_buffering off;
    #proxy_temp_path proxy_temp;
    #proxy_max_temp_file_size 5000m;

    location / {
        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;
        set $upstream_app 192.168.3.12;
        set $upstream_port 444;
        set $upstream_proto https;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

        # Hide proxy response headers from Nextcloud that conflict with ssl.conf
        # Uncomment the Optional additional headers in SWAG's ssl.conf to pass Nextcloud's security scan
        proxy_hide_header Referrer-Policy;
        proxy_hide_header X-Content-Type-Options;
        proxy_hide_header X-Frame-Options;
        proxy_hide_header X-XSS-Protection;

        # Disable proxy buffering
        #proxy_buffering off;
        #fastcgi_intercept_errors on;
        #fastcgi_request_buffering off;
        #fastcgi_connect_timeout 60;
        #fastcgi_send_timeout 1800;
        #fastcgi_read_timeout 1800;
       # proxy_request_buffering off;
        proxy_temp_path proxy_temp;
        proxy_max_temp_file_size 0;
    }
}

one thing I noticed client_body_temp_path is getting used when I am logged into the console I can see this space getting the file size I am sending it.

When I check proxy_temp this space is always 0. It does not appear to be using this space

xokia · November 7, 2023, 4:20am

bump anyone? Dont know where to go to get help and find what is preventing the file from showing up in nextcloud. All nextcloud gives me is unknown error. I’ve tried every setting I can think of and nothing seems to make a file > 1GB show up in nextcloud. Even though the file fully uploads.

xokia · November 7, 2023, 5:34pm

bump please help. Would really like to upload >1GB and use as cloud storage.

Skyhooker · November 7, 2023, 11:21pm

I am very much a noob at this, but you mentioned “flowupload” - that sounds like the browser-based drag-and-drop method, is that right? I stopped using that method of uploading quite a while back because of errors and performance issues. Instead, I created a folder within the Nextcloud folder that was created when I installed the desktop client, and named it “Upload temp.” I paste files into that, and once uploaded by the client, move them to the correct destination folder using the web UI. I hope this helps.

xokia · November 8, 2023, 1:48am

I dont use flowupload since it does not work with version 27 of nextcloud.

I have a folder called upload.

I am trying to upload via chrome web browser by selecting new then browsing to the file I want to upload. I can see the file fully upload. But then when it gets to the point where nextcloud should update the database and then show the file in the directory it fails with unknown error. Files under 1GB upload fine this way. The limit seems to be 1GB for some strange reason. I can even see the file get fully uploaded to nginx. The issues seems to be when nextcloud needs to create the database entry almost as if they are somehow limiting the filesize to 1GB or less by default.

Skyhooker · November 8, 2023, 2:13am

What I’m saying is don’t use a browser at all for uploads - drop files directly into the Nextcloud client’s sync folder (I created a sub-folder within that), then move them where they need to go once uploaded. The only things I use the browser for are for downloads and to move files around between folders after uploading.

xokia · November 8, 2023, 8:10am

using the app from what I understand bypasses the limit. I’d like to solve this so I can upload files from any PC without downloading the APP. I.e. a machine that maybe you cant install apps on.

The only thing I have found is this
https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/big_file_upload_configuration.html

I’ve tried all of that and still no luck. I created a share that all 3 have access to
nextcloud_tmp/ I can see the files in all 3 consoles.

/mnt/user/appdata/nextcloud/php/php-local.ini

memory_limit=2048M
upload_max_filesize=70G
post_max_size=70G
max_input_time=6400
max_execution_time=6400
output_buffering=0
upload_tmp_dir =/nextcloud_tmp
file_uploads =On

/mnt/user/appdata/nextcloud/www/nextcloud/config/config.php

  'tempdirectory' => '/nextcloud_tmp',

/mnt/user/appdata/swag/nginx/proxy-confs/nextcloud.subdomain.conf

        add_header X-Accel-Buffering no;
        proxy_buffering off;
        client_max_body_size 0;
        client_body_in_file_only on;
        client_body_in_single_buffer on;
        client_body_temp_path /nextcloud_tmp;
        fastcgi_request_buffering off;
        fastcgi_max_temp_file_size 0;

I can see the file get written out to the temp directory. The 2.7GB file is the file I uploaded.

root@fe21f75dab1f:/# ls -l nextcloud_tmp/
total 2666004
-rw------- 1 abc abc         17 Nov  8 00:56 0000000069
-rw------- 1 abc abc        114 Nov  8 00:56 0000000070
-rw------- 1 abc abc        129 Nov  8 00:57 0000000071
-rw------- 1 abc abc         92 Nov  8 00:57 0000000072
-rw------- 1 abc abc        323 Nov  8 00:57 0000000073
-rw------- 1 abc abc        198 Nov  8 00:57 0000000074
-rw------- 1 abc abc        341 Nov  8 00:57 0000000075
-rw------- 1 abc abc         19 Nov  8 00:57 0000000076
-rw------- 1 abc abc        198 Nov  8 00:57 0000000077
-rw------- 1 abc abc 2729873091 Nov  8 00:57 0000000078
-rw------- 1 abc abc        198 Nov  8 00:57 0000000079
-rw------- 1 abc abc         17 Nov  8 00:57 0000000080
-rw------- 1 abc abc        198 Nov  8 00:57 0000000081
-rw------- 1 abc abc        198 Nov  8 00:57 0000000082
-rw------- 1 abc abc        323 Nov  8 00:57 0000000083
-rw------- 1 abc abc        198 Nov  8 00:57 0000000084
-rw------- 1 abc abc        198 Nov  8 00:57 0000000085
-rw------- 1 abc abc        198 Nov  8 00:57 0000000086
-rw------- 1 abc abc        323 Nov  8 00:57 0000000087
-rw------- 1 abc abc        323 Nov  8 00:57 0000000088
-rw------- 1 abc abc        198 Nov  8 00:57 0000000089
-rw------- 1 abc abc        114 Nov  8 00:57 0000000090
-rw------- 1 abc abc        198 Nov  8 00:57 0000000091
-rw------- 1 abc abc        198 Nov  8 00:58 0000000092
-rw------- 1 abc abc        198 Nov  8 00:58 0000000093
-rw------- 1 abc abc        323 Nov  8 00:58 0000000094
-rw------- 1 abc abc        198 Nov  8 00:58 0000000095
-rw------- 1 abc abc        323 Nov  8 00:58 0000000096
-rw------- 1 abc abc        323 Nov  8 00:58 0000000097

xokia · November 10, 2023, 8:26pm

bump Would really like to upload >1GB and use as cloud storage.

xokia · November 10, 2023, 11:00pm

I think I finally figured it out. But maybe someone can explain it to me because I do not understand it. I have been trying to modify /mnt/user/appdata/swag/nginx/proxy-confs/nextcloud.subdomain.conf to make this work because swag is the proxy server.

however after I modified
/mnt/user/appdata/nextcloud/nginx/site-confs/default.conf

    # display real ip in nginx logs when connected through reverse proxy via docker network
    set_real_ip_from 172.18.0.0/12;
    real_ip_header X-Forwarded-For;

    # https://docs.nextcloud.com/server/latest/admin_manual/installation/nginx.html#nextcloud-in-the-webroot-of-nginx

    # set max upload size and increase upload timeout:
    client_max_body_size 0;
    client_body_temp_path /nextcloud_tmp;
    client_body_timeout 6400s;
    fastcgi_buffers 64 4K;
    proxy_buffering off;
    fastcgi_request_buffering off;
    fastcgi_max_temp_file_size 0;
    add_header X-Accel-Buffering no;

It started working as expect. How does modifying the nginx file within the nextcloud container solve this? I assumed this was just an example file?

Knutowskie1 · July 7, 2024, 11:37pm

I was ripping my hair out over this. Thank you very much! I tried swag changes and everything… the DEFAULT.CONF… dude!