Large files will not upload, no matter what

Starfish · July 21, 2023, 9:00am

Nextcloud version: 27.0.1
Operating system and version: CentOS Stream 8
Apache or nginx version: Apache/2.4.37 (centos)
PHP version: PHP 8.1.20

The issue you are facing:

I cannot upload any large files. A 3.9GB iso errors out.

Is this the first time you’ve seen this error? (Y/N):
Y
Steps to replicate it:

Set all the PHP settings and occ settings as per documentation
Open Nextcloud in browser and upload a >3GB file.
Check logs for errors.

The output of your Nextcloud log in Admin > Logging:

{"reqId":"ZLpE8A-lrTmRExoewFy4YQAAAAY","level":0,"time":"2023-07-21T10:43:24+02:00","remoteAddr":"192.168.10.137","user":"hanse","app":"webdav","method":"PUT","url":"/remote.php/dav/uploads/hanse/web-file-upload-4bb525aacfcc298b321cd35b3f4741c5-1689928943994/1","message":"Expected filesize of 20971520 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 0 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0","version":"27.0.0.8","exception":{"Exception":"Sabre\\DAV\\Exception\\BadRequest","Message":"Expected filesize of 20971520 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 0 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.","Code":0,"Trace":[{"file":"/var/www/html/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php","line":149,"function":"put","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->"},{"file":"/var/www/html/nextcloud/apps/dav/lib/Upload/UploadFolder.php","line":50,"function":"createFile","class":"OCA\\DAV\\Connector\\Sabre\\Directory","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":1098,"function":"createFile","class":"OCA\\DAV\\Upload\\UploadFolder","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":504,"function":"createFile","class":"Sabre\\DAV\\Server","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpPut","class":"Sabre\\DAV\\CorePlugin","type":"->"},{"file":"/var/www/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":253,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":321,"function":"start","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/html/nextcloud/apps/dav/lib/Server.php","line":364,"function":"exec","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/html/nextcloud/apps/dav/appinfo/v2/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server","type":"->"},{"file":"/var/www/html/nextcloud/remote.php","line":172,"args":["/var/www/html/nextcloud/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/html/nextcloud/apps/dav/lib/Connector/Sabre/File.php","Line":297,"message":"Expected filesize of 20971520 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 0 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.","exception":{},"CustomMessage":"Expected filesize of 20971520 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 0 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side."}}

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'app.mail.imap.timeout' => 25,
  'passwordsalt' => 'u287Zy1aQhtvY6mEpxW7U3cb1Ru2cx',
  'secret' => 'SCRUBBED',
  'trusted_domains' => 
  array (
    0 => '192.168.20.15',
    1 => '192.168.10.4',
    2 => 'SCRUBBED',
    3 => 'SCRUBBED',
  ),
  'datadirectory' => '/var/www/html/synology/nextcloud',
  'dbtype' => 'mysql',
  'version' => '27.0.0.8',
  'overwrite.cli.url' => 'http://SCRUBBED',
  'htaccess.RewriteBase' => '/',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '3306',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud',
  'dbpassword' => 'SCRUBBED',
  'installed' => true,
  'instanceid' => 'ociu92qehwsl',
  'logtimezone' => 'Africa/Johannesburg',
  'memcache.local' => '\\OC\\Memcache\\Redis',
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'filelocking.enabled' => true,
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' => 
  array (
    'host' => '127.0.0.1',
    'port' => 6379,
  ),
  'mail_domain' => 'SCRUBBED',
  'mail_from_address' => 'nextcloud',
  'mail_smtpmode' => 'smtp',
  'mail_sendmailmode' => 'smtp',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpauth' => 1,
  'mail_smtphost' => 'SCRUBBED',
  'mail_smtpport' => '465',
  'mail_smtpname' => 'SCRUBBED',
  'mail_smtppassword' => 'SCRUBBED',
  'mail_smtpsecure' => 'ssl',
  'maintenance' => false,
  'onlyoffice' => 
  array (
    'verify_peer_off' => true,
    'jwt_header' => 'AuthorizationJwt',
  ),
  'allow_local_remote_servers' => true,
  'trusted_proxies' => 
  array (
    0 => '192.168.10.11',
    1 => '127.0.0.1',
  ),
  'theme' => '',
  'loglevel' => 0,
  'app_install_overwrite' => 
  array (
    0 => 'breezedark',
    1 => 'drawio',
    2 => 'fulltextsearch',
    3 => 'fulltextsearch_elasticsearch',
    4 => 'files_fulltextsearch',
    5 => 'onlyoffice',
    6 => 'groupfolders',
    7 => 'ransomware_protection',
    8 => 'spreed',
    9 => 'rocketchat_nextcloud',
    10 => 'passwords',
    11 => 'electronicsignatures',
    12 => 'impersonate',
    13 => 'files_downloadactivity',
  ),
  'default_phone_region' => 'ISO 3166-2:ZA',
  'ldapProviderFactory' => 'OCA\\User_LDAP\\LDAPProviderFactory',
  'preview_max_memory' => 4096,
  'preview_max_filesize_image' => 256,
  'memories.ffmpeg_path' => '/bin/ffmpeg',
  'memories.ffprobe_path' => '/bin/ffprobe',
  'memories.transcoder' => '/var/www/html/nextcloud/apps/memories/exiftool-bin/go-vod-amd64',
  'memories.no_transcode' => false,
  'memories.qsv' => false,
  'memories.exiftool_no_local' => true,
  'memories.vod.path' => '/var/www/html/nextcloud/apps/memories/exiftool-bin/go-vod-amd64',
  'memories.vod.ffmpeg' => '/usr/bin/ffmpeg',
  'memories.vod.ffprobe' => '/usr/bin/ffprobe',
);

The output of your Apache/nginx/system log in /var/log/____:

Nothing funny

Apache modules enabled:

Loaded Modules:
 core_module (static)
 so_module (static)
 http_module (static)
 access_compat_module (shared)
 actions_module (shared)
 alias_module (shared)
 allowmethods_module (shared)
 auth_basic_module (shared)
 auth_digest_module (shared)
 authn_anon_module (shared)
 authn_core_module (shared)
 authn_dbd_module (shared)
 authn_dbm_module (shared)
 authn_file_module (shared)
 authn_socache_module (shared)
 authz_core_module (shared)
 authz_dbd_module (shared)
 authz_dbm_module (shared)
 authz_groupfile_module (shared)
 authz_host_module (shared)
 authz_owner_module (shared)
 authz_user_module (shared)
 autoindex_module (shared)
 brotli_module (shared)
 cache_module (shared)
 cache_disk_module (shared)
 cache_socache_module (shared)
 data_module (shared)
 dbd_module (shared)
 deflate_module (shared)
 dir_module (shared)
 dumpio_module (shared)
 echo_module (shared)
 env_module (shared)
 expires_module (shared)
 ext_filter_module (shared)
 filter_module (shared)
 headers_module (shared)
 include_module (shared)
 info_module (shared)
 log_config_module (shared)
 logio_module (shared)
 macro_module (shared)
 mime_magic_module (shared)
 mime_module (shared)
 negotiation_module (shared)
 remoteip_module (shared)
 request_module (shared)
 rewrite_module (shared)
 setenvif_module (shared)
 slotmem_plain_module (shared)
 slotmem_shm_module (shared)
 socache_dbm_module (shared)
 socache_memcache_module (shared)
 socache_shmcb_module (shared)
 status_module (shared)
 substitute_module (shared)
 suexec_module (shared)
 unique_id_module (shared)
 unixd_module (shared)
 userdir_module (shared)
 version_module (shared)
 vhost_alias_module (shared)
 watchdog_module (shared)
 dav_module (shared)
 dav_fs_module (shared)
 dav_lock_module (shared)
 lua_module (shared)
 mpm_prefork_module (shared)
 proxy_module (shared)
 lbmethod_bybusyness_module (shared)
 lbmethod_byrequests_module (shared)
 lbmethod_bytraffic_module (shared)
 lbmethod_heartbeat_module (shared)
 proxy_ajp_module (shared)
 proxy_balancer_module (shared)
 proxy_connect_module (shared)
 proxy_express_module (shared)
 proxy_fcgi_module (shared)
 proxy_fdpass_module (shared)
 proxy_ftp_module (shared)
 proxy_http_module (shared)
 proxy_hcheck_module (shared)
 proxy_scgi_module (shared)
 proxy_uwsgi_module (shared)
 proxy_wstunnel_module (shared)
 systemd_module (shared)
 cgi_module (shared)
 http2_module (shared)
 proxy_http2_module (shared)
 php_module (shared)

Relevant php.ini values:

php -i | grep -i max
Zend Max Execution Timers => disabled
max_execution_time => 0 => 0
max_file_uploads => 20 => 20
max_input_nesting_level => 64 => 64
max_input_time => -1 => -1
max_input_vars => 1000 => 1000
max_multipart_body_parts => -1 => -1
post_max_size => 20G => 20G
upload_max_filesize => 16G => 16G
zend.exception_string_param_max_len => 15 => 15
ldap.max_links => Unlimited => Unlimited
mysql.max_links => Unlimited => Unlimited
mysql.max_persistent => Unlimited => Unlimited
mysqli.max_links => Unlimited => Unlimited
mysqli.max_persistent => Unlimited => Unlimited
session.gc_maxlifetime => 1440 => 1440
...
unserialize_max_depth => 4096 => 4096
Max keys => 16229
opcache.jit_max_exit_counters => 8192 => 8192
opcache.jit_max_loop_unrolls => 8 => 8
opcache.jit_max_polymorphic_calls => 2 => 2
opcache.jit_max_recursive_calls => 2 => 2
opcache.jit_max_recursive_returns => 2 => 2
opcache.jit_max_root_traces => 1024 => 1024
opcache.jit_max_side_traces => 128 => 128
opcache.max_accelerated_files => 10000 => 10000
opcache.max_file_size => 0 => 0
opcache.max_wasted_percentage => 5 => 5

max_execution_time = 3600
max_input_time = 3600

Apache config:

<VirtualHost *:80>
  DocumentRoot /var/www/html/nextcloud
  ServerName '192.168.20.15'
  <IfModule mod_headers.c>
      Header always set Referrer-Policy "strict-origin"
      Header always set Strict-Transport-Security "max-age=15552000; includeSubDomains"
  </IfModule>
  <Directory "/var/www/html/">
       Require all granted
       AllowOverride All
       Options FollowSymLinks MultiViews
  </Directory>
</VirtualHost>

I have set up a temp upload folder on my NAS to ensure there is enough space. The temp folder has 777 access (yeah yeah, I know) and is owned by apache user and group.

As you will see I have disabled the mod_reqtimeout for apache as per the documentation. I am running out of ideas. Anyone willing to help?I am not using php-fpm and not sure if it will help.

Thank you in advance for any help! I appreciate it a lot.

jtr · July 22, 2023, 11:40pm

What does disk usage on your NC server look like while you’re doing these uploads? I saw you mentioned a “temp upload folder” on your NAS, but the one that matters is the /tmp on your NC server.

The web UI will use chunking so the file size should basically be irrelevant as long as there is disk space.

Also, can you confirm you are getting this behavior when bypassing your reverse proxy?

Unfortunately this particular error can be caused by many things ranging from misconfigurations to resource problems to client device power management settings to antivirus programs to bad hardware/switch configurations (like a bad Ethernet cable or switch port).

Starfish · July 24, 2023, 7:00am

@jtr thank you for your response.

I have set the temp foider in /etc/php.ini as well, so it will not use /tmp on the server at all. PHP should use the Synology space.

Yes, have tested and confirmed I get this error by directly accessing Nextcloud. So it is not a reverse proxy issue.

Doubt it is resources. A VM running on a Dell R720 PowerEdge, VM has 8 cores and 16 GB of memory, sufficient disk space. Server has dual 1Gbps uplinks in LACP, running through a proper network device. CAT6 cables all tested. So, this brings me back to Nextcloud + PHP config, and my problem still being: cannot upload large files. Not sure what other configs anyone will need to troubleshoot this.

TIA for any help!

Starfish · July 24, 2023, 9:46am

Increased local /tmp space. Reconfigured php to use this local tmp storage. The file does not even reach this tmp space during upload. This all points to Nextcloud not behaving like it should.

I get this, every time:

{“reqId”:“ZL5HP3D34pb-Y9rKCmJ8vwAAAAQ”,“level”:3,“time”:“2023-07-24T11:42:19+02:00”,“remoteAddr”:“192.168.10.137”,“user”:“hanse”,“app”:“no app in context”,“method”:“PUT”,“url”:“/remote.php/dav/uploads/hanse/web-file-upload-4bb525aacfcc298b321cd35b3f4741c5-1690191679677/1”,“message”:“Expected filesize of 20971520 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 0 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.”,“userAgent”:“Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0”,“version”:“27.0.0.8”,“exception”:{“Exception”:“Sabre\DAV\Exception\BadRequest”,“Message”:“Expected filesize of 20971520 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 0 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.”,“Code”:0,“Trace”:[{“file”:“/var/www/html/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php”,“line”:149,“function”:“put”,“class”:“OCA\DAV\Connector\Sabre\File”,“type”:“->”},{“file”:“/var/www/html/nextcloud/apps/dav/lib/Upload/UploadFolder.php”,“line”:50,“function”:“createFile”,“class”:“OCA\DAV\Connector\Sabre\Directory”,“type”:“->”,“args”:[“*** sensitive parameters replaced “]},{“file”:”/var/www/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php",“line”:1098,“function”:“createFile”,“class”:“OCA\DAV\Upload\UploadFolder”,“type”:“->”,“args”:[" sensitive parameters replaced “]},{“file”:”/var/www/html/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php",“line”:504,“function”:“createFile”,“class”:“Sabre\DAV\Server”,“type”:“->”,“args”:[" sensitive parameters replaced “]},{“file”:”/var/www/html/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php",“line”:89,“function”:“httpPut”,“class”:“Sabre\DAV\CorePlugin”,“type”:“->”},{“file”:“/var/www/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php”,“line”:472,“function”:“emit”,“class”:“Sabre\DAV\Server”,“type”:“->”},{“file”:“/var/www/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php”,“line”:253,“function”:“invokeMethod”,“class”:“Sabre\DAV\Server”,“type”:“->”},{“file”:“/var/www/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php”,“line”:321,“function”:“start”,“class”:“Sabre\DAV\Server”,“type”:“->”},{“file”:“/var/www/html/nextcloud/apps/dav/lib/Server.php”,“line”:364,“function”:“exec”,“class”:“Sabre\DAV\Server”,“type”:“->”},{“file”:“/var/www/html/nextcloud/apps/dav/appinfo/v2/remote.php”,“line”:35,“function”:“exec”,“class”:“OCA\DAV\Server”,“type”:“->”},{“file”:“/var/www/html/nextcloud/remote.php”,“line”:172,“args”:[“/var/www/html/nextcloud/apps/dav/appinfo/v2/remote.php”],“function”:“require_once”}],“File”:“/var/www/html/nextcloud/apps/dav/lib/Connector/Sabre/File.php”,“Line”:297,“message”:“Expected filesize of 20971520 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 0 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.”,“exception”:{},“CustomMessage”:“Expected filesize of 20971520 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 0 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.”}}
{“reqId”:“ZL5HP3D34pb-Y9rKCmJ8vwAAAAQ”,“level”:0,“time”:“2023-07-24T11:42:19+02:00”,“remoteAddr”:“192.168.10.137”,“user”:“hanse”,“app”:“webdav”,“method”:“PUT”,“url”:“/remote.php/dav/uploads/hanse/web-file-upload-4bb525aacfcc298b321cd35b3f4741c5-1690191679677/1”,“message”:“Expected filesize of 20971520 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 0 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.”,“userAgent”:“Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0”,“version”:“27.0.0.8”,“exception”:{“Exception”:“Sabre\DAV\Exception\BadRequest”,“Message”:“Expected filesize of 20971520 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 0 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.”,“Code”:0,“Trace”:[{“file”:“/var/www/html/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php”,“line”:149,“function”:“put”,“class”:“OCA\DAV\Connector\Sabre\File”,“type”:“->”},{“file”:“/var/www/html/nextcloud/apps/dav/lib/Upload/UploadFolder.php”,“line”:50,“function”:“createFile”,“class”:“OCA\DAV\Connector\Sabre\Directory”,“type”:“->”,“args”:[" sensitive parameters replaced “]},{“file”:”/var/www/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php",“line”:1098,“function”:“createFile”,“class”:“OCA\DAV\Upload\UploadFolder”,“type”:“->”,“args”:[" sensitive parameters replaced “]},{“file”:”/var/www/html/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php",“line”:504,“function”:“createFile”,“class”:“Sabre\DAV\Server”,“type”:“->”,“args”:[" sensitive parameters replaced ***”]},{“file”:“/var/www/html/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php”,“line”:89,“function”:“httpPut”,“class”:“Sabre\DAV\CorePlugin”,“type”:“->”},{“file”:“/var/www/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php”,“line”:472,“function”:“emit”,“class”:“Sabre\DAV\Server”,“type”:“->”},{“file”:“/var/www/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php”,“line”:253,“function”:“invokeMethod”,“class”:“Sabre\DAV\Server”,“type”:“->”},{“file”:“/var/www/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php”,“line”:321,“function”:“start”,“class”:“Sabre\DAV\Server”,“type”:“->”},{“file”:“/var/www/html/nextcloud/apps/dav/lib/Server.php”,“line”:364,“function”:“exec”,“class”:“Sabre\DAV\Server”,“type”:“->”},{“file”:“/var/www/html/nextcloud/apps/dav/appinfo/v2/remote.php”,“line”:35,“function”:“exec”,“class”:“OCA\DAV\Server”,“type”:“->”},{“file”:“/var/www/html/nextcloud/remote.php”,“line”:172,“args”:[“/var/www/html/nextcloud/apps/dav/appinfo/v2/remote.php”],“function”:“require_once”}],“File”:“/var/www/html/nextcloud/apps/dav/lib/Connector/Sabre/File.php”,“Line”:297,“message”:“Expected filesize of 20971520 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 0 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.”,“exception”:{},“CustomMessage”:“Expected filesize of 20971520 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 0 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.”}}

This is from Firefox browser, on my local LAN to the Nextcloud server (same LAN, different VLAN), all connected via Ethernet. So no wifi or other interference that can be the cause. This is configuration issue, but not sure what else I can still look at. It is a really irritating problem to have.

Starfish · July 24, 2023, 10:07am

Update: i have added the following two settings to the apache config:

  LimitRequestBody 0
  Timeout 300

My config now looks like this:

<VirtualHost *:80>
  DocumentRoot /var/www/html/nextcloud
  ServerName '192.168.20.15'
  LimitRequestBody 0
  Timeout 300
  <IfModule mod_headers.c>
      Header always set Referrer-Policy "strict-origin"
      Header always set Strict-Transport-Security "max-age=15552000; includeSubDomains"
  </IfModule>
  <Directory "/var/www/html/">
       Require all granted
       AllowOverride All
       Options FollowSymLinks MultiViews
  </Directory>
</VirtualHost>

I am now able to upload a 1.5GB file without any issues to my nextcloud, but, the 3.9GB file still errors out with the same error. So which settings are not being honoured at the moment?