File takes long time to appear after upload finished // File Upload larger 500MB

Nextcloud version : 13.0.4
Operating system and version : 18.04
Apache or nginx version : Apache/2.4.29 (Ubuntu)
PHP version : 7.1 *

  • Had 7.2 Installd before tought it might help to switch to recommended version, but that did not help.

Is this the first time you’ve seen this error? (Y/N): N

The issue you are facing:

So I decided to setup the Nextcloud 13 on a new Ubuntu 18.04 server.
Everything looked fine until I encountered an odd behaviour, while uploading files. Depending on the size of a file at the end of the upload when the upload “completes” the Files doesn’t show up for a couple of seconds the larger the file the longer the time for the uploaded file to appear.For a 700MB file it takes about 20 seconds

At First I thought it may be an Antivirus issue and that it just takes some time to Scan the file.
So I deactivated the Apps for Malware/Virus detection and also stopped ClamAV on the Server…Did not help

Since the user-data is stored on another server which is mounted over nfs, I moved the user-data location to the server on which nextcloud is installed just to be sure it isn’t a network problem…no improvement

I Also tried to increase the php values drastically, just to see any changes in the behaviour…Still nothing

PHP-Values:

upload_max_filesize = 20GB
max_file_uploads = 20
post_max_size = 40GB
memory_limit = 20GB
max_input_vars = 1000
max_execution_time = 3600
max_input_time = 3600

I also deactivated the mod_reqtimeout since it was mentioned in the documentation
that it could lead to the problem I have…well that also did not help

So yeah I really can not think of anything else that could lead to this issue and the only solution I can think of right now is to wipe the system clean and start from the beginning…

But before I do that I really would like to find out what caused the issue or what I might have done wrong or overlooked…

LOGS:

The output of your Nextcloud log in Admin > Logging:

{"reqId":"W8hLzgMBR5HSLB3aSmIhoAAAAAo","level":4,"time":"2018-10-18T09:01:02+00:00","remoteAddr":"192.168.10.184","user":"kfpnx-admin","app":"webdav","method":"PUT","url":"\/remote.php\/dav\/uploads\/kfpnx-admin\/web-file-upload-db89ac72c24fc80c5d44fa3$
{"reqId":"W8hhmchvmt6EWM4Q@msYrwAAAAE","level":3,"time":"2018-10-18T10:34:02+00:00","remoteAddr":"192.168.10.184","user":"--","app":"base","method":"GET","url":"\/apps\/files\/?dir=\/&fileid=7","message":"Exception: {\"Exception\":\"ErrorException\",\"$
{"reqId":"W8hhmchvmt6EWM4Q@msYrwAAAAE","level":3,"time":"2018-10-18T10:34:02+00:00","remoteAddr":"192.168.10.184","user":"--","app":"index","method":"GET","url":"\/apps\/files\/?dir=\/&fileid=7","message":"Exception: {\"Exception\":\"ErrorException\",\$
{"reqId":"W8hhmchvmt6EWM4Q@msYrwAAAAE","level":3,"time":"2018-10-18T10:34:02+00:00","remoteAddr":"192.168.10.184","user":"--","app":"PHP","method":"GET","url":"\/apps\/files\/?dir=\/&fileid=7","message":"ErrorException: Cannot modify header information$

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => '*****',
  'passwordsalt' => '*****',
  'secret' => '*****',
  'trusted_domains' =>
  array (
    0 => 'nx.domain.de',
    1 => 'nx.domain.de',
  ),
  'datadirectory' => '/var/www/html/nextcloud/ldata',
  'overwrite.cli.url' => 'https://nx.domain.de',
  'htaccess.RewriteBase' => '/',
  'dbtype' => 'mysql',
  'version' => '13.0.4.0',
  'dbname' => 'db',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'dbu',
  'dbpassword' => '*****',
  'installed' => true,
  'mail_smtpmode' => 'smtp',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_from_address' => 'cloud',
  'mail_domain' => 'mail.de',
  'mail_smtphost' => '0.0.0.0',
  'mail_smtpport' => '25',
  'ldapIgnoreNamingRules' => false,
  'ldapProviderFactory' => '\\OCA\\User_LDAP\\LDAPProviderFactory',
  'memcache.distributed' => '\OC\Memcache\Redis',
  'filelocking.enabled' => true,
  'memcache.locking' => '\OC\Memcache\Redis',
  'memcache.local' => '\OC\Memcache\APCu',
  'redis' => array(
     'host' => 'localhost',
     'port' => 6379,
  ),
  'maintenance' => false,
  'theme' => 'default',
  'default_language' => 'en',
);

The output of your Apache/nginx/system log in /var/log/____:

[Thu Oct 18 12:44:22.594548 2018] [:warn] [pid 3305] ModSecurity: Loaded PCRE do not match with compiled!
[Thu Oct 18 12:44:22.594564 2018] [:notice] [pid 3305] ModSecurity: LUA compiled version="Lua 5.1"
[Thu Oct 18 12:44:22.594579 2018] [:notice] [pid 3305] ModSecurity: YAJL compiled version="2.1.0"
[Thu Oct 18 12:44:22.594595 2018] [:notice] [pid 3305] ModSecurity: LIBXML compiled version="2.9.4"
[Thu Oct 18 12:44:22.594610 2018] [:notice] [pid 3305] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
[Thu Oct 18 12:44:22.678874 2018] [http2:warn] [pid 3318] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected m$
[Thu Oct 18 12:44:22.773374 2018] [mpm_prefork:notice] [pid 3318] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.0h configured -- resuming normal operations
[Thu Oct 18 12:44:22.773436 2018] [core:notice] [pid 3318] AH00094: Command line: '/usr/sbin/apache2'
[Thu Oct 18 14:11:04.581613 2018] [mpm_prefork:notice] [pid 3318] AH00169: caught SIGTERM, shutting down
[Thu Oct 18 14:11:04.851422 2018] [:notice] [pid 3819] ModSecurity for Apache/2.9.2 (http://www.modsecurity.org/) configured.
[Thu Oct 18 14:11:04.851504 2018] [:notice] [pid 3819] ModSecurity: APR compiled version="1.6.2"; loaded version="1.6.3"
[Thu Oct 18 14:11:04.851517 2018] [:warn] [pid 3819] ModSecurity: Loaded APR do not match with compiled!
[Thu Oct 18 14:11:04.851530 2018] [:notice] [pid 3819] ModSecurity: PCRE compiled version="8.39 "; loaded version="8.41 2017-07-05"
[Thu Oct 18 14:11:04.851542 2018] [:warn] [pid 3819] ModSecurity: Loaded PCRE do not match with compiled!
[Thu Oct 18 14:11:04.851553 2018] [:notice] [pid 3819] ModSecurity: LUA compiled version="Lua 5.1"
[Thu Oct 18 14:11:04.851563 2018] [:notice] [pid 3819] ModSecurity: YAJL compiled version="2.1.0"
[Thu Oct 18 14:11:04.851574 2018] [:notice] [pid 3819] ModSecurity: LIBXML compiled version="2.9.4"
[Thu Oct 18 14:11:04.851586 2018] [:notice] [pid 3819] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
[Thu Oct 18 14:11:04.971062 2018] [http2:warn] [pid 3834] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected m$
[Thu Oct 18 14:11:05.089718 2018] [mpm_prefork:notice] [pid 3834] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.0h configured -- resuming normal operations
[Thu Oct 18 14:11:05.089776 2018] [core:notice] [pid 3834] AH00094: Command line: '/usr/sbin/apache2'
[Thu Oct 18 14:11:45.804969 2018] [authz_core:error] [pid 3867] [client 192.168.10.184:59296] AH01630: client denied by server configuration: /var/www/html/nextcloud/localdata/.ocdata
[Thu Oct 18 14:11:48.616155 2018] [authz_core:error] [pid 3874] [client 192.168.10.184:59302] AH01630: client denied by server configuration: /var/www/html/nextcloud/localdata/.ocdata
[Thu Oct 18 14:11:48.865816 2018] [evasive20:error] [pid 3853] [client 192.168.10.184:59294] client denied by server configuration: /var/www/html/nextcloud/apps/files/
[Thu Oct 18 14:13:00.106328 2018] [authz_core:error] [pid 3873] [client 192.168.10.184:59429] AH01630: client denied by server configuration: /var/www/html/nextcloud/localdata/.ocdata
[Thu Oct 18 14:13:02.405390 2018] [authz_core:error] [pid 3867] [client 192.168.10.184:59435] AH01630: client denied by server configuration: /var/www/html/nextcloud/localdata/.ocdata
[Thu Oct 18 14:13:02.430730 2018] [evasive20:error] [pid 3898] [client 192.168.10.184:59427] client denied by server configuration: /var/www/html/nextcloud/apps/files/
[Thu Oct 18 14:13:02.638617 2018] [evasive20:error] [pid 3898] [client 192.168.10.184:59427] client denied by server configuration: /var/www/html/nextcloud/apps/files/
[Thu Oct 18 14:14:11.882002 2018] [mpm_prefork:notice] [pid 3834] AH00169: caught SIGTERM, shutting down
[Thu Oct 18 14:14:12.180130 2018] [:notice] [pid 3953] ModSecurity for Apache/2.9.2 (http://www.modsecurity.org/) configured.
[Thu Oct 18 14:14:12.180209 2018] [:notice] [pid 3953] ModSecurity: APR compiled version="1.6.2"; loaded version="1.6.3"
[Thu Oct 18 14:14:12.180223 2018] [:warn] [pid 3953] ModSecurity: Loaded APR do not match with compiled!
[Thu Oct 18 14:14:12.180236 2018] [:notice] [pid 3953] ModSecurity: PCRE compiled version="8.39 "; loaded version="8.41 2017-07-05"
[Thu Oct 18 14:14:12.180249 2018] [:warn] [pid 3953] ModSecurity: Loaded PCRE do not match with compiled!
[Thu Oct 18 14:14:12.180260 2018] [:notice] [pid 3953] ModSecurity: LUA compiled version="Lua 5.1"
[Thu Oct 18 14:14:12.180272 2018] [:notice] [pid 3953] ModSecurity: YAJL compiled version="2.1.0"
[Thu Oct 18 14:14:12.180283 2018] [:notice] [pid 3953] ModSecurity: LIBXML compiled version="2.9.4"
[Thu Oct 18 14:14:12.180295 2018] [:notice] [pid 3953] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
[Thu Oct 18 14:14:12.303021 2018] [http2:warn] [pid 3967] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected m$
[Thu Oct 18 14:14:12.431275 2018] [mpm_prefork:notice] [pid 3967] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.0h configured -- resuming normal operations
[Thu Oct 18 14:14:12.431336 2018] [core:notice] [pid 3967] AH00094: Command line: '/usr/sbin/apache2'
[Thu Oct 18 14:16:38.949062 2018] [mpm_prefork:notice] [pid 3967] AH00169: caught SIGTERM, shutting down
[Thu Oct 18 14:16:39.203393 2018] [:notice] [pid 4094] ModSecurity for Apache/2.9.2 (http://www.modsecurity.org/) configured.
[Thu Oct 18 14:16:39.203485 2018] [:notice] [pid 4094] ModSecurity: APR compiled version="1.6.2"; loaded version="1.6.3"
[Thu Oct 18 14:16:39.203499 2018] [:warn] [pid 4094] ModSecurity: Loaded APR do not match with compiled!
[Thu Oct 18 14:16:39.203512 2018] [:notice] [pid 4094] ModSecurity: PCRE compiled version="8.39 "; loaded version="8.41 2017-07-05"
[Thu Oct 18 14:16:39.203524 2018] [:warn] [pid 4094] ModSecurity: Loaded PCRE do not match with compiled!
[Thu Oct 18 14:16:39.203535 2018] [:notice] [pid 4094] ModSecurity: LUA compiled version="Lua 5.1"
[Thu Oct 18 14:16:39.203547 2018] [:notice] [pid 4094] ModSecurity: YAJL compiled version="2.1.0"
[Thu Oct 18 14:16:39.203558 2018] [:notice] [pid 4094] ModSecurity: LIBXML compiled version="2.9.4"
[Thu Oct 18 14:16:39.203570 2018] [:notice] [pid 4094] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
[Thu Oct 18 14:16:39.283987 2018] [http2:warn] [pid 4109] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected m$
[Thu Oct 18 14:16:39.366762 2018] [mpm_prefork:notice] [pid 4109] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.0h configured -- resuming normal operations
[Thu Oct 18 14:16:39.366820 2018] [core:notice] [pid 4109] AH00094: Command line: '/usr/sbin/apache2'
[Thu Oct 18 14:21:11.917484 2018] [authz_core:error] [pid 4120] [client 192.168.10.184:60159] AH01630: client denied by server configuration: /var/www/html/nextcloud/ldata/.ocdata
[Thu Oct 18 14:21:15.209774 2018] [authz_core:error] [pid 4168] [client 192.168.10.184:60156] AH01630: client denied by server configuration: /var/www/html/nextcloud/ldata/.ocdata
[Thu Oct 18 14:22:06.419716 2018] [authz_core:error] [pid 4115] [client 192.168.10.184:60272] AH01630: client denied by server configuration: /var/www/html/nextcloud/ldata/.ocdata
[Thu Oct 18 14:22:08.380816 2018] [authz_core:error] [pid 4168] [client 192.168.10.184:60278] AH01630: client denied by server configuration: /var/www/html/nextcloud/ldata/.ocdata
[Thu Oct 18 14:22:19.575371 2018] [authz_core:error] [pid 4217] [client 192.168.10.184:60296] AH01630: client denied by server configuration: /var/www/html/nextcloud/ldata/.ocdata


Can you check on the server what is happening during these 20s? Just use top, iotop and a network monitor.