Nextcloud fails to upload large files

defaria · March 31, 2022, 9:06pm

Nextcloud version: 23.0.0
Operating system and version: Synology DSM 7.x
Apache version: 2.4.52 (Debian)
PHP version : 8.0.14

The issue you are facing:

Uploads are very slow, even on local LAN. But far worse is after uploading NC attempts to assemble some parts together and fails with a 504. Sometimes the file appears on refresh, other times it doesn’t. I set up this Nextcloud server to allow my non-techie friends to be able to upload and share with me large files. As such it needs to be quick and reliable. But it’s slow and unreliable.

Is this the first time you’ve seen this error? N

Steps to replicate it:

Log into Nextcloud
Attempt to upload a large file (2 Gig or greater)
Wait, wait, wait - fails with 504

The output of your Nextcloud log in Admin > Logging:

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => 'ocr9s4iyfz5x',
  'passwordsalt' => 'redacted',
  'secret' => 'redacted+',
  'trusted_domains' => 
  array (
    0 => 'redacted',
    1 => 'redacted',
    2 => 'jupiter',
    3 => '192.168.0.251',
  ),
  'default_phone_region' => 'US',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'datadirectory' => '/var/www/html/data',
  'dbtype' => 'mysql',
  'version' => '23.0.0.10',
  'overwrite.cli.url' => 'redacted',
  'overwritehost' => 'redacted',
  'overwriteprotocol' => 'https',
  'dbname' => 'nextcloud',
  'dbhost' => '192.168.0.251',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud',
  'dbpassword' => 'nextcloud',
  'installed' => true,
  'mail_from_address' => 'Andrew',
  'mail_smtpmode' => 'smtp',
  'mail_sendmailmode' => 'smtp',
  'mail_domain' => 'DeFaria.com',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpauth' => 1,
  'mail_smtphost' => 'defaria.com',
  'mail_smtpport' => '',
  'mail_smtpname' => 'andrew',
  'mail_smtppassword' => 'redacted',
  'maintenance' => false,
  'enable_previews' => true,
  'enabledPreviewProviders' => 
  array (
    0 => 'OC\\Preview\\Movie',
    1 => 'OC\\Preview\\PNG',
    2 => 'OC\\Preview\\JPEG',
    3 => 'OC\\Preview\\GIF',
    4 => 'OC\\Preview\\BMP',
    5 => 'OC\\Preview\\XBitmap',
    6 => 'OC\\Preview\\MP3',
    7 => 'OC\\Preview\\MP4',
    8 => 'OC\\Preview\\TXT',
    9 => 'OC\\Preview\\MarkDown',
    10 => 'OC\\Preview\\PDF',
  ),
);

The output of your Apache/nginx/system log in /var/log/____:

PASTE HERE

stratege1401 · March 31, 2022, 9:29pm

I dont know much of DSM but,
I think it’s your php.ini value witch is wrong…

My /etc/php/8.0/apache2/php.ini

; Maximum allowed size for uploaded files.
; http://php.net/upload-max-filesize
upload_max_filesize = 999GB

; Maximum number of files that can be uploaded via a single request
max_file_uploads = 20

defaria · April 1, 2022, 12:14am

DSM is Synology’s DiskStation Manager and I believe it’s derived from Debian.

I running Nextcloud in a Docker container. As such I don’t have an /etc/php at all. Under Docker, they seem to put php under /usr/local. Looking at /usr/local I find only:

root@nextcloud:cd /usr/local
root@nextcloud:/usr/local# find . -type f -exec grep upload_max_filesize {} +
grep: ./bin/php: binary file matches
grep: ./bin/phpdbg: binary file matches
./etc/php/conf.d/nextcloud.ini:upload_max_filesize=${PHP_UPLOAD_LIMIT}
./etc/php/php.ini-development:upload_max_filesize = 2M
./etc/php/php.ini-production:upload_max_filesize = 2M
./include/php/main/php_globals.h:	zend_long upload_max_filesize;
grep: ./lib/php/extensions/no-debug-non-zts-20200930/imagick.so: binary file matches
root@nextcloud:/usr/local#

Uploading big files > 512MB says

The default maximum file size for uploads is 512MB". It seems to allude to setting values in php.ini but doesn’t tell you where that php.ini is.

So I created a phpinfo.php in the Docker container and found that my php.ini should be:

Configuration File (php.ini) Path	/usr/local/etc/php

Yet there is no /usr/local/etc/php/php.ini. There are php.ini-development and php.ini-production but as we can see above the upload_max_filesize in both of them are a poultry 2M. Are you saying go into the Docker container and set the parms as you indicate by creating php.ini in the Docker container? If so then what happens when the Docker container updates? Reapply the tweaks?

And there’s also this on that same page:

The upload_max_filesize and post_max_size settings may not apply to file uploads through WebDAV single file PUT requests or Chunked file uploads For those, PHP and webserver timeouts are the limiting factor on the upload size.

Nextcloud should really provide a UI in the settings interface to configure this stuff! Uploading to store and as they like to say have a “Safe place for all your data”, one would think that seamless and fast uploads of large data files would be a core piece of functionality.

devnull · April 1, 2022, 5:48am

Nextcloud will probably not provide for this, since they are not Nextcloud configurations, but PHP configurations and partially web server configurations. But I agree that many of the configurations mentioned are not optimal for Nextcloud.

defaria · April 1, 2022, 6:09am

Care to take a guess as to where I’m supposed to make these modifications and how do I get them to stick in the context of a Docker container?

defaria · April 2, 2022, 12:03am

OK this still isn’t working for me. Here’s where I’m at. Bear in mind I’m running NC in a docker container on Synology (a Debian derived OS).

I have managed to map a config file into the Docker container that NC seems to be using. My goal is to set the following:

; Must set post_max_size to be larger than upload_max_filesize
post_max_size = 5GB

; Maximum allowed size for uploaded files.
; http://php.net/upload-max-filesize
upload_max_filesize = 4GB

; Test to see if I can set a variable for PHP through this php.ini file
max_execution_time = 60

This shows up in the docker container:

$ docker exec -it nextcloud cat /usr/local/etc/php/php.ini
; Must set post_max_size to be larger than upload_max_filesize
post_max_size = 5GB

; Maximum allowed size for uploaded files.
; http://php.net/upload-max-filesize
upload_max_filesize = 4GB

; Test to see if I can set a variable for PHP through this php.ini file
max_execution_time = 60
$

And I have a phpinfo.php file set up. Looking at the output I see that php.ini seems to be correct:

2022-04-01@16:51:36

And my max_execution_time has taken effect (default is 30, not 60) (Sorry, image uploading not working, you’re gonna have to trust me on this), However, neither post_max_size nor upload_max_filesize have been changed. They still say 512 Meg whereas I was trying to set them to 5 and 4 GB.

How do you set these values for NC in a docker container?

defaria · April 15, 2022, 1:07am

More info!

Turns out there is a /usr/local/etc/php/conf.d/nextcloud.ini that contains

memory_limit=${PHP_MEMORY_LIMIT}
upload_max_filesize=${PHP_UPLOAD_LIMIT}
post_max_size=${PHP_UPLOAD_LIMIT}

and the docker container is supposed to set these environment variables.

Even with these settings uploads still fail with “Error when assembling chunks, status code 504” which seems to indicate timeout yet refreshing the page shows the file was uploaded apparently in full.

Oh I added max_execution_time and set it to 1 hour (3600). Didn’t help.

defaria · May 16, 2022, 10:05pm

I was hoping that updating to NextCloud 24 would address these problems. It says it’s a lot quicker. I’m not seeing it. NC tries to add all kinds of functionality but the underlying filesystem is just way quicker! For example, copying a 2.2 gig file from my desktop to my NAS (which is also where NC is running) mounted via NFS takes 10 minutes and 19 seconds of wall time. However, dragging and dropping the same file into NC going through WebDAV takes <just past the 11-minute mark and it’s bouncing between 15 - 30 minutes…> an additional 11 minutes - twice as long!

Plus the indicator is all over the map! It’ll say 1 hour and then a second or two later that drops to 20 minutes, climbs up to 36 minutes, and back down. It’s sampling way too quickly and reporting all kinds of incorrect times.

Finally, the copy of this file into NC results in the same old error “Error when assembling chunks, status code 504” and yet the file appears intact with a simple refresh.

svigerske · July 25, 2022, 3:12pm

For us, we got a similar issue together with logmessage

Sabre\DAV\Exception\BadRequest: Expected filesize of 3065896288 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 0 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.

after upgrading from NextCloud 23 to 24 last weekend.
The Docker images for NextCloud 24 were just declared as “stable”, so I thought it was time to upgrade.

We use the Docker image that runs Apache in the container and it turns out that the culprit is LimitRequestBody in the Apache config (found this because its mentioned at Uploading big files > 512MB — Nextcloud latest Administration Manual latest documentation). Apparently the default changed from unlimited to 1GB with Apache 2.4.54 (core - Apache HTTP Server Version 2.4), and this is the version running in the Docker container now. I presume that with NextCloud 23, it was still Apache 2.4.53. Adding a line LimitRequestBody 0 to .htaccess helped. Would be nice if the NextCloud image would be setup to do this automatically.

Robin_Hofmann · August 21, 2022, 8:20pm

Weißt du, ich habe jetzt Stunden damit verbracht Configs anzupassen. Die kleine Zeile in der .htaccess war endlich die Lösung. 1000 Dank für deinen Post.

defaria · August 22, 2022, 2:36pm

Didn’t work for me. Same error.

jordant2 · November 4, 2022, 5:19am

This worked for me after upgrading to v23. I hadn’t been automatically syncing files for a while and didn’t realize I couldn’t upload anything over 1gb until I tried to upload a video yesterday. Even though my max upload size was set and showing in the admin panel as 16GB, I still couldn’t upload a 2GB video. Adding LimitRequestBody 0 to htaccess did the trick. Thanks!

defaria · November 8, 2022, 3:24am

Well, a small test of a 3-gig file seems to have worked. But man, if you cannot properly estimate how long a process will take please don’t try to!

A test of the much larger file (The Last Duel (2021) - 23.2 gigs)

No bueno
2022-11-07@19:23:11

Rico_la_terreur · May 10, 2023, 7:06pm

Hello! I have same issue on an openmediavault installation, swag and nextcloud running in a container…

I can’t find any htaccess file (where is this file normally?), or maybe is hidden?……

Who had the same problem and a solution please ? Running crazy here…

Bellusterra · May 19, 2023, 10:42am

Should be in your Nextcloud folder /var/www/html. You are correct it is a hidden file. Use

ls -la

command to view hidden files.