Webdav requests very slow with haproxy as proxy

ℹ️ Support
1
Support intro

Sorry to hear you’re facing problems :slightly_frowning_face:

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can :heart:

Nextcloud version (eg, 20.0.5): 26.0.1
Operating system and version (eg, Ubuntu 20.04): debian 11
Apache or nginx version (eg, Apache 2.4.25): 2.4.56
PHP version (eg, 7.4): 8.1

The issue you are facing: webdav requests (https://domain.tld/public.php/webdav/) trigger HAProxy server timeout (so the web interface freezes during the timeout)

Is this the first time you’ve seen this error? (Y/N): Yes

Steps to replicate it:

  1. Create a public share
  2. Access it via web browser
  3. Wait the interface show up

The output of your Nextcloud log in Admin > Logging:

Nothing

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

$CONFIG = array (
  'passwordsalt' => '',
  'secret' => '',
  'trusted_domains' => 
  array (
    0 => 'domain.com',
    1 => 'node7',
  ),
  'datadirectory' => '/var/www/html/data',
  'dbtype' => 'mysql',
  'version' => '26.0.1.1',
  'overwrite.cli.url' => 'https://domain.com',
  'dbname' => 'nextcloud',
  'dbhost' => '127.0.0.1',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => 'true',
  'dbuser' => 'nicolas',
  'dbpassword' => '',
  'installed' => true,
  'instanceid' => '',
  'trusted_proxies' => 
  array (
    0 => '192.168.1.39',
    1 => '192.168.1.40',
  ),
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'open_basedir' => '/dev/urandom',
  'updater.release.channel' => 'production',
  'default_phone_region' => 'FR',
  'redis' => 
  array (
    'host' => '/var/run/redis/redis.sock',
    'port' => '0',
  ),
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'mail_smtpmode' => 'smtp',
  'mail_smtpsecure' => 'ssl',
  'mail_sendmailmode' => 'smtp',
  'mail_from_address' => 'contact',
  'mail_domain' => 'domain.com ',
  'mail_smtphost' => 'smpt',
  'mail_smtpauth' => 1,
  'mail_smtpport' => '465',
  'mail_smtpname' => 'contact@domain.com',
  'mail_smtppassword' => '',
);

The output of your Apache/nginx/system log in /var/log/____:

[Wed May 24 11:41:40.515634 2023] [proxy:debug] [pid 88077:tid 140618361726720] mod_proxy.c(1506): [client 192.168.1.39:58420] AH01143: Running scheme unix handler (attempt 0)
        [Wed May 24 11:41:40.515653 2023] [proxy_fcgi:debug] [pid 88077:tid 140618361726720] mod_proxy_fcgi.c(1054): [client 192.168.1.39:58420] AH01076: url: fcgi://localhost/var/www/html/index.php proxyname: (null) proxyport: 0
        [Wed May 24 11:41:40.515655 2023] [proxy_fcgi:debug] [pid 88077:tid 140618361726720] mod_proxy_fcgi.c(1063): [client 192.168.1.39:58420] AH01078: serving URL fcgi://localhost/var/www/html/index.php
        [Wed May 24 11:41:40.515657 2023] [proxy:debug] [pid 88077:tid 140618361726720] proxy_util.c(2538): AH00942: FCGI: has acquired connection for (*:80)
        [Wed May 24 11:41:40.515660 2023] [proxy:debug] [pid 88077:tid 140618361726720] proxy_util.c(2596): [client 192.168.1.39:58420] AH00944: connecting fcgi://localhost/var/www/html/index.php to localhost:8000
        [Wed May 24 11:41:40.515662 2023] [proxy:debug] [pid 88077:tid 140618361726720] proxy_util.c(2632): [client 192.168.1.39:58420] AH02545: fcgi: has determined UDS as /run/php/php8.1-fpm.sock
        [Wed May 24 11:41:40.515706 2023] [proxy:debug] [pid 88077:tid 140618361726720] proxy_util.c(2819): [client 192.168.1.39:58420] AH00947: connected /var/www/html/index.php to httpd-UDS:0
        [Wed May 24 11:41:40.515722 2023] [proxy:debug] [pid 88077:tid 140618361726720] proxy_util.c(3188): AH02823: FCGI: connection established with Unix domain socket /run/php/php8.1-fpm.sock (*:80)
        [Wed May 24 11:41:42.149683 2023] [proxy:debug] [pid 88077:tid 140618361726720] proxy_util.c(2554): AH00943: FCGI: has released connection for (*:80)
        [Wed May 24 11:41:42.282234 2023] [deflate:debug] [pid 88077:tid 140618143516416] mod_deflate.c(869): [client 192.168.1.39:58424] AH01384: Zlib: Compressed 1649 to 800 : URL /core/js/public/publicpage.js
        [Wed May 24 11:41:42.302534 2023] [deflate:debug] [pid 88077:tid 140618447001344] mod_deflate.c(869): [client 192.168.1.39:58446] AH01384: Zlib: Compressed 64469 to 19599 : URL /core/l10n/fr.js
        [Wed May 24 11:41:42.303010 2023] [deflate:debug] [pid 87564:tid 140618143516416] mod_deflate.c(869): [client 192.168.1.39:58440] AH01384: Zlib: Compressed 159509 to 51758 : URL /dist/core-main.js
        [Wed May 24 11:41:42.305179 2023] [proxy:debug] [pid 87565:tid 140618447001344] mod_proxy.c(1506): [client 192.168.1.39:58476] AH01143: Running scheme unix handler (attempt 0)
        [Wed May 24 11:41:42.305197 2023] [proxy_fcgi:debug] [pid 87565:tid 140618447001344] mod_proxy_fcgi.c(1054): [client 192.168.1.39:58476] AH01076: url: fcgi://localhost/var/www/html/index.php proxyname: (null) proxyport: 0
        [Wed May 24 11:41:42.305199 2023] [proxy_fcgi:debug] [pid 87565:tid 140618447001344] mod_proxy_fcgi.c(1063): [client 192.168.1.39:58476] AH01078: serving URL fcgi://localhost/var/www/html/index.php
        [Wed May 24 11:41:42.305721 2023] [deflate:debug] [pid 87565:tid 140618455402240] mod_deflate.c(869): [client 192.168.1.39:58484] AH01384: Zlib: Compressed 12400 to 4222 : URL /dist/core-files_client.js
        [Wed May 24 11:41:42.306692 2023] [deflate:debug] [pid 87565:tid 140618463794944] mod_deflate.c(869): [client 192.168.1.39:58460] AH01384: Zlib: Compressed 928 to 503 : URL /dist/core-files_fileinfo.js
        [Wed May 24 11:41:42.307039 2023] [proxy:debug] [pid 87565:tid 140618447001344] proxy_util.c(2538): AH00942: FCGI: has acquired connection for (*:80)
        [Wed May 24 11:41:42.307062 2023] [proxy:debug] [pid 87565:tid 140618447001344] proxy_util.c(2596): [client 192.168.1.39:58476] AH00944: connecting fcgi://localhost/var/www/html/index.php to localhost:8000
        [Wed May 24 11:41:42.307065 2023] [proxy:debug] [pid 87565:tid 140618447001344] proxy_util.c(2632): [client 192.168.1.39:58476] AH02545: fcgi: has determined UDS as /run/php/php8.1-fpm.sock
        [Wed May 24 11:41:42.307794 2023] [proxy:debug] [pid 87565:tid 140618447001344] proxy_util.c(2819): [client 192.168.1.39:58476] AH00947: connected /var/www/html/index.php to httpd-UDS:0
        [Wed May 24 11:41:42.307812 2023] [proxy:debug] [pid 87565:tid 140618447001344] proxy_util.c(3188): AH02823: FCGI: connection established with Unix domain socket /run/php/php8.1-fpm.sock (*:80)
        [Wed May 24 11:41:42.310368 2023] [deflate:debug] [pid 88077:tid 140618210658048] mod_deflate.c(869): [client 192.168.1.39:58424] AH01384: Zlib: Compressed 22786 to 6135 : URL /apps/files_sharing/l10n/fr.js
        [Wed May 24 11:41:42.327483 2023] [proxy:debug] [pid 87565:tid 140618447001344] proxy_util.c(2554): AH00943: FCGI: has released connection for (*:80)
        [Wed May 24 11:41:42.333379 2023] [deflate:debug] [pid 87565:tid 140618361726720] mod_deflate.c(869): [client 192.168.1.39:58460] AH01384: Zlib: Compressed 397 to 248 : URL /dist/files_sharing-main.js
        [Wed May 24 11:41:42.334980 2023] [deflate:debug] [pid 87565:tid 140618353334016] mod_deflate.c(869): [client 192.168.1.39:58476] AH01384: Zlib: Compressed 16055 to 5246 : URL /apps/files_sharing/js/public.js
        [Wed May 24 11:41:42.336673 2023] [deflate:debug] [pid 88077:tid 140618168694528] mod_deflate.c(869): [client 192.168.1.39:58424] AH01384: Zlib: Compressed 3785 to 814 : URL /apps/files_sharing/js/templates.js
        [Wed May 24 11:41:42.353606 2023] [deflate:debug] [pid 87565:tid 140618344941312] mod_deflate.c(869): [client 192.168.1.39:58460] AH01384: Zlib: Compressed 1190 to 664 : URL /apps/files_sharing/js/public_note.js
        [Wed May 24 11:41:42.363172 2023] [deflate:debug] [pid 87565:tid 140618336548608] mod_deflate.c(869): [client 192.168.1.39:58476] AH01384: Zlib: Compressed 20728 to 6030 : URL /apps/files/l10n/fr.js
        [Wed May 24 11:41:42.366622 2023] [deflate:debug] [pid 88077:tid 140618455402240] mod_deflate.c(869): [client 192.168.1.39:58424] AH01384: Zlib: Compressed 773 to 388 : URL /apps/files/js/semaphore.js
        [Wed May 24 11:41:42.368796 2023] [deflate:debug] [pid 87565:tid 140618328155904] mod_deflate.c(869): [client 192.168.1.39:58460] AH01384: Zlib: Compressed 40232 to 11352 : URL /apps/files/js/file-upload.js
        [Wed May 24 11:41:42.939863 2023] [proxy:debug] [pid 88077:tid 140618135123712] mod_proxy.c(1506): [client 192.168.1.39:58488] AH01143: Running scheme unix handler (attempt 0)
        [Wed May 24 11:41:42.939877 2023] [proxy_fcgi:debug] [pid 88077:tid 140618135123712] mod_proxy_fcgi.c(1054): [client 192.168.1.39:58488] AH01076: url: fcgi://localhost/var/www/html/index.php proxyname: (null) proxyport: 0
        [Wed May 24 11:41:42.939879 2023] [proxy_fcgi:debug] [pid 88077:tid 140618135123712] mod_proxy_fcgi.c(1063): [client 192.168.1.39:58488] AH01078: serving URL fcgi://localhost/var/www/html/index.php
        [Wed May 24 11:41:42.939881 2023] [proxy:debug] [pid 88077:tid 140618135123712] proxy_util.c(2538): AH00942: FCGI: has acquired connection for (*:80)
        [Wed May 24 11:41:42.939884 2023] [proxy:debug] [pid 88077:tid 140618135123712] proxy_util.c(2596): [client 192.168.1.39:58488] AH00944: connecting fcgi://localhost/var/www/html/index.php to localhost:8000
        [Wed May 24 11:41:42.939886 2023] [proxy:debug] [pid 88077:tid 140618135123712] proxy_util.c(2632): [client 192.168.1.39:58488] AH02545: fcgi: has determined UDS as /run/php/php8.1-fpm.sock
        [Wed May 24 11:41:42.939929 2023] [proxy:debug] [pid 88077:tid 140618135123712] proxy_util.c(2819): [client 192.168.1.39:58488] AH00947: connected /var/www/html/index.php to httpd-UDS:0
        [Wed May 24 11:41:42.939944 2023] [proxy:debug] [pid 88077:tid 140618135123712] proxy_util.c(3188): AH02823: FCGI: connection established with Unix domain socket /run/php/php8.1-fpm.sock (*:80)
        [Wed May 24 11:41:42.953605 2023] [proxy:debug] [pid 88077:tid 140618135123712] proxy_util.c(2554): AH00943: FCGI: has released connection for (*:80)
        [Wed May 24 11:41:47.151585 2023] [proxy:debug] [pid 88077:tid 140618185479936] mod_proxy.c(1506): [client 192.168.1.39:58420] AH01143: Running scheme unix handler (attempt 0)
        [Wed May 24 11:41:47.151602 2023] [proxy_fcgi:debug] [pid 88077:tid 140618185479936] mod_proxy_fcgi.c(1054): [client 192.168.1.39:58420] AH01076: url: fcgi://localhost/var/www/html/public.php proxyname: (null) proxyport: 0
        [Wed May 24 11:41:47.151604 2023] [proxy_fcgi:debug] [pid 88077:tid 140618185479936] mod_proxy_fcgi.c(1063): [client 192.168.1.39:58420] AH01078: serving URL fcgi://localhost/var/www/html/public.php
        [Wed May 24 11:41:47.151613 2023] [proxy:debug] [pid 88077:tid 140618185479936] proxy_util.c(2538): AH00942: FCGI: has acquired connection for (*:80)
        [Wed May 24 11:41:47.151617 2023] [proxy:debug] [pid 88077:tid 140618185479936] proxy_util.c(2596): [client 192.168.1.39:58420] AH00944: connecting fcgi://localhost/var/www/html/public.php to localhost:8000
        [Wed May 24 11:41:47.151618 2023] [proxy:debug] [pid 88077:tid 140618185479936] proxy_util.c(2632): [client 192.168.1.39:58420] AH02545: fcgi: has determined UDS as /run/php/php8.1-fpm.sock
        [Wed May 24 11:41:47.151665 2023] [proxy:debug] [pid 88077:tid 140618185479936] proxy_util.c(2819): [client 192.168.1.39:58420] AH00947: connected /var/www/html/public.php to httpd-UDS:0
        [Wed May 24 11:41:47.151682 2023] [proxy:debug] [pid 88077:tid 140618185479936] proxy_util.c(3188): AH02823: FCGI: connection established with Unix domain socket /run/php/php8.1-fpm.sock (*:80)
        [Wed May 24 11:41:47.151913 2023] [proxy:debug] [pid 88077:tid 140618185479936] proxy_util.c(2554): AH00943: FCGI: has released connection for (*:80)
        [Wed May 24 11:41:47.151929 2023] [deflate:debug] [pid 88077:tid 140618185479936] mod_deflate.c(869): [client 192.168.1.39:58420] AH01384: Zlib: Compressed 4 to 6 : URL /public.php/webdav/
        [Wed May 24 11:41:47.304669 2023] [deflate:debug] [pid 87565:tid 140618319763200] mod_deflate.c(869): [client 192.168.1.39:58476] AH01384: Zlib: Compressed 5117 to 1951 : URL /apps/files_sharing/js/files_drop.js

Output errors in nextcloud.log in /var/www/ or as admin user in top right menu, filtering for errors. Use a pastebin service if necessary.

EMPTY

As mentionned, webdav requests such as https://domain.tld/public.php/webdav/ always wait for the timeout server of haproxy. But if I do the same request with curl on y laptop, there is not problems at.
Into the logs, the interesting part is here :

[Wed May 24 11:41:42.953605 2023] [proxy:debug] [pid 88077:tid 140618135123712] proxy_util.c(2554): AH00943: FCGI: has released connection for (*:80)
[Wed May 24 11:41:47.151585 2023] [proxy:debug] [pid 88077:tid 140618185479936] mod_proxy.c(1506): [client 192.168.1.39:58420] AH01143: Running scheme unix handler (attempt 0)
[Wed May 24 11:41:47.151602 2023] [proxy_fcgi:debug] [pid 88077:tid 140618185479936] mod_proxy_fcgi.c(1054): [client 192.168.1.39:58420] AH01076: url: fcgi://localhost/var/www/html/public.php proxyname: (null) proxyport: 0

We clearly see the haproxy timeout between the first and second line. I think there could be a problem with headers sent by next cloud interface ?

2

I’m a bit unclear about your topology. Please provide your HAProxy configuration.

It looks like you’re using HAProxy as an app (FastCGI) serve, but have it behind another proxy as well (192.168.1.39). What’s that proxy consist of? And where are your static assets originating from?

To clarify, you’re saying that URLs like /public.php/webdav/[...] work with cURL, but not in your web browser?

I presume you’ve tried from another computer/browser?

There isn’t much difference between cURL and a web browser connecting in terms of the NC side of things.

What do the entries in your nextcloud.log look like during these transactions?

P.S. Unrelated, but open_basedir does not belong in your config.php.

3

Thanks for your help.
So I have 5 nexctclouds instances behind 2 haproxy servers. And you are right, the problem is not related to nextcloud, this morning l cleared the cache and I see that this is an other file that blocks (an asset) for 50 seconds which the timeout server of haproxy :

global
    chroot /var/lib/haproxy
    log 127.0.0.1:514 local0
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

defaults
    log global
    mode http
    option httplog
    option dontlognull
    timeout connect 5000
    timeout client 50000
    timeout server 50000
    errorfile 400 /etc/haproxy/errors/400.http

So I think this is a problem server like a restriction or something.
You can see the problem here : https://encelade.eros-cloud.com/index.php/s/88ieWpsrRqsoySm