Occasionally very slow root directory PROPFIND

Nextcloud version (eg, 20.0.5): 23.0.0
Operating system and version (eg, Ubuntu 20.04): debian something
Apache or nginx version (eg, Apache 2.4.25): nginx something
PHP version (eg, 7.4): 8.something

Sometimes the PROPFIND https://mydomain/remote.php/dav/files/myuser/ will take a very long time (20s+) instead of ~100ms normally. This results in the web interface (/apps/files) loading slowly and also affects the desktop clients, usually it will work fine but sometimes it gets stuck until the proxy aborts the connection (timeout set to 5 minutes).

I found this issue but it doesn’t apply, I only have 8 devices.

Is this a common problem? Does someone have any hints as to what might cause it or how to troubleshoot?

Installed apps
  - accessibility: 1.9.0
  - admin_audit: 1.13.0
  - breezedark: 23.1.0
  - calendar: 3.0.4
  - cloud_federation_api: 1.6.0
  - contacts: 4.0.7
  - dav: 1.21.0
  - extract: 1.3.3
  - federatedfilesharing: 1.13.0
  - files: 1.18.0
  - files_external: 1.15.0
  - files_pdfviewer: 2.4.0
  - files_rightclick: 1.2.0
  - files_sharing: 1.15.0
  - files_trashbin: 1.13.0
  - files_versions: 1.16.0
  - files_videoplayer: 1.12.0
  - forms: 2.4.0
  - keeweb: 0.6.8
  - logreader: 2.8.0
  - lookup_server_connector: 1.11.0
  - notifications: 2.11.1
  - notify_push: 0.3.0
  - oauth2: 1.11.0
  - phonetrack: 0.6.9
  - photos: 1.5.0
  - provisioning_api: 1.13.0
  - settings: 1.5.0
  - sharebymail: 1.13.0
  - text: 3.4.0
  - theming: 1.14.0
  - twofactor_backupcodes: 1.12.0
  - twofactor_totp: 6.2.0
  - updatenotification: 1.13.0
  - viewer: 1.7.0
  - workflowengine: 2.5.0