Low performance on WebDAV requests

The Basics

  • Nextcloud Server version (e.g., 29.x.x):
    • nextcloud:28.0.14-fpm (Docker image)
  • Operating system and version (e.g., Ubuntu 24.04):
    • Debian GNU/Linux 12 (Bookworm)
  • Web server and version (e.g, Apache 2.4.25):
    • nginx:1.27.3-alpine3.20
  • Reverse proxy and version _(e.g. nginx 1.27.2)
    • nginxproxy/nginx-proxy:1.6.4-alpine
  • PHP version (e.g, 8.3):
    • PHP 8.2.27
  • Is this the first time you’ve seen this error? (Yes / No):
    • Yes
  • When did this problem seem to first start?
    • Since the beginning of new installation
  • Installation method (e.g. AlO, NCP, Bare Metal/Archive, etc.)
    • Docker
  • Are you using CloudfIare, mod_security, or similar? (Yes / No)
    • No

Summary of the issue you are facing:

Hello!

I have deployed a new instance of Nextcloud in my LAN. I’m using davfs2 to browse files remotelly from another host within the same LAN where the Nextcloud instance is operating.

File operation is very slow on that interface. An ncdu on the root of the WebDAV mount takes several hours to complete (interactive browsing is very slow too). A native du on the server takes several seconds. The server is currently hosting about 128K files (~300GiB) of data.

I’ve inspected the traffic on the reverse proxy with allinurl/goaccess (full command line in [1], requires Nginx configuration) during the ncdu, and I see that requests to the /remote.php/dav/files/<user>/<path> end point take about 500ms to complete. The same analysis on the web server yields similar results.

I’ve also profiled the server with xdebug during the ncdu, but I’m unsure of how to interpret the results:

50% of the time seems to go into the verify_password function. I think that the execution time in that picture is in tens of nanoseconds. So there doesn’t seem to be any bottleneck there.

I’ve seen many posts linking WebDAV performance problems with too many entries in oc_authtoken table. I’ve checked the table and there are only 9 entries.

Finally, I’ve checked the database performance:

nextcloud=# select left(regexp_replace(query, E'[\\n\\r]+', ' ', 'g' ),75),total_plan_time,mean_plan_time,calls,total_exec_time,mean_exec_time from pg_stat_statements order by calls desc limit 10;
									left                                     | total_plan_time | mean_plan_time | calls |  total_exec_time   |    mean_exec_time     
-----------------------------------------------------------------------------+-----------------+----------------+-------+--------------------+-----------------------
 SELECT "filecache"."fileid", "storage", "path", "path_hash", "filecache"."p |               0 |              0 |  2277 |         840.951808 |   0.36932446552481346
 SELECT "fileid" FROM "oc_filecache" WHERE ("storage" = $1) AND ("path_hash" |               0 |              0 |   967 |  248.9673529999998 |   0.25746365356773543
 SELECT "id", "numeric_id", "available", "last_checked" FROM "oc_storages" W |               0 |              0 |   886 |  23.80663599999999 |  0.026869792325056395
 COMMIT                                                                      |               0 |              0 |   779 | 0.9037710000000002 | 0.0011601681643132221
 BEGIN                                                                       |               0 |              0 |   779 | 0.7942910000000001 | 0.0010196290115532744
 SELECT * FROM "oc_appconfig"                                                |               0 |              0 |   509 | 133.63458700000007 |    0.2625433929273084
 SELECT "filecache"."fileid", "storage", "path", "path_hash", "filecache"."p |               0 |              0 |   509 |  2067.002205999999 |     4.060908066797643
 SET SESSION CHARACTERISTICS AS TRANSACTION ISOLATION LEVEL READ COMMITTED   |               0 |              0 |   509 |  13.46800199999999 |  0.026459728880157178
 SELECT "uid", "displayname", "password" FROM "oc_users" WHERE "uid_lower" = |               0 |              0 |   507 | 16.277062000000008 |   0.03210465877712035
 SELECT "appid", "configkey", "configvalue" FROM "oc_preferences" WHERE "use |               0 |              0 |   505 | 24.907069000000018 |   0.04932092871287128
(10 rows)

According to DB documentation [1], execution times are shown in milliseconds, so the database seems to be performing well.

At this point I’ve run out of ideas. Any hint will be very welcome! :slight_smile:

Thank you in advance,

Guillermo

[1] ssh root@nextcloud.local docker logs nextcloud_proxy.1.186ffpa58hdbdplbfhkb6v4u6 --follow | sed 's/\x1B\[[0-9;]\{1,\}[A-Za-z]//g' | docker run -p 7890:7890 --rm -i allinurl/goaccess -a -o html --real-time-html --log-format='nginx.1 | %h - %e [%x] "%r" %s %b "%R" "%u" %T' --datetime-format='%d/%b/%Y:%H:%M:%S %z' - - > report.html
[2] PostgreSQL: Documentation: 17: F.30. pg_stat_statements — track statistics of SQL planning and execution

Steps to replicate it

  1. Mount the WebDAV share.
  2. Issue ncdu on the mount point.

Log entries

Nextcloud

I don’t see any log entry related to this problem in the Nextcloud admin panel.

Web Browser

N.A.

Web server / Reverse Proxy

Already described.

Configuration

Nextcloud

occ config:list system:

{
    "system": {
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "upgrade.disable-web": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "pgsql",
        "version": "28.0.14.1",
        "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_sendmailmode": "smtp",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "default_phone_region": "ES",
        "loglevel": 2,
        "maintenance": false,
        "maintenance_window_start": 1,
        "memories.db.triggers.fcu": true,
        "memories.exiftool": "\/var\/www\/html\/custom_apps\/memories\/bin-ext\/exiftool-amd64-glibc",
        "memories.vod.path": "\/var\/www\/html\/custom_apps\/memories\/bin-ext\/go-vod-amd64",
        "memories.gis_type": 2,
        "overwritehost": "***REMOVED SENSITIVE VALUE***",
        "overwriteprotocol": "https",
        "overwritewebroot": "\/",
        "overwritecondaddr": "172.29.0.*",
        "opcache.enable": 1,
        "opcache.memory_consumption": 128,
        "opcache.interned_strings_buffer": 8,
        "opcache.max_accelerated_files": 10000,
        "opcache.validate_timestamps": 1
    }
}

Apps

occ app:list:

Enabled:
  - activity: 2.20.0
  - calendar: 4.7.16
  - cloud_federation_api: 1.11.0
  - comments: 1.18.0
  - contacts: 5.5.3
  - cookbook: 0.11.2
  - dashboard: 7.8.0
  - dav: 1.29.2
  - federatedfilesharing: 1.18.0
  - files: 2.0.0
  - files_pdfviewer: 2.9.0
  - files_reminders: 1.1.0
  - files_trashbin: 1.18.0
  - firstrunwizard: 2.17.0
  - groupfolders: 16.0.12
  - logreader: 2.13.0
  - lookup_server_connector: 1.16.0
  - maps: 1.4.0
  - memories: 7.4.1
  - music: 2.1.1
  - nextcloud_announcements: 1.17.0
  - notifications: 2.16.0
  - oauth2: 1.16.4
  - password_policy: 1.18.0
  - privacy: 1.12.0
  - provisioning_api: 1.18.0
  - recommendations: 2.0.0
  - related_resources: 1.3.0
  - richdocuments: 8.3.13
  - serverinfo: 1.18.0
  - settings: 1.10.1
  - sharebymail: 1.18.0
  - support: 1.11.1
  - survey_client: 1.16.0
  - systemtags: 1.18.0
  - tasks: 0.16.1
  - text: 3.9.2
  - theming: 2.3.0
  - twofactor_backupcodes: 1.17.0
  - updatenotification: 1.18.0
  - viewer: 2.2.0
  - workflowengine: 2.10.0
Disabled:
  - admin_audit: 1.18.0
  - bruteforcesettings: 2.8.0
  - circles: 28.0.0 (installed 28.0.0)
  - contactsinteraction: 1.9.0 (installed 1.9.0)
  - encryption: 2.16.0
  - federation: 1.18.0 (installed 1.18.0)
  - files_external: 1.20.0
  - files_sharing: 1.20.0 (installed 1.20.0)
  - files_versions: 1.21.0 (installed 1.21.0)
  - photos: 2.4.0 (installed 2.4.0)
  - suspicious_login: 6.0.0
  - twofactor_totp: 10.0.0-beta.2
  - user_ldap: 1.19.0
  - user_status: 1.8.1 (installed 1.8.1)
  - weather_status: 1.8.0 (installed 1.8.0)