Extremly slow nextcloud

The Basics

  • Nextcloud Server version (e.g., 29.x.x):
    • 31.0.5
  • Operating system and version (e.g., Ubuntu 24.04):
    • Debian 12.10
  • Web server and version (e.g, Apache 2.4.25):
    • Apache 2.4.63
  • Reverse proxy and version _(e.g. nginx 1.27.2)
    • Caddy 2.10.0
  • PHP version (e.g, 8.3):
    • 8.3.21
  • Is this the first time you’ve seen this error? (Yes / No):
    • No
  • When did this problem seem to first start?
    • NC 30
  • Installation method (e.g. AlO, NCP, Bare Metal/Archive, etc.)
    • AIO
  • Are you using CloudfIare, mod_security, or similar? (Yes / No)
    • No

Summary of the issue you are facing:

Hello,
we have been having problems with NC lately, everything takes a long time to load, whether it is opening a folder in the web or via webdav, then moving a folder with data from one place to another, things move after a long time but there is an empty folder in the original place and countless other similar strange problems. I even made up my mind yesterday and did an update to 31.0.5 but unfortunately that didn’t help. We have about 80 users and 50 of them active in the last 24 hours. Nextcloud is in a Proxmox VM with 32 AMD EPYC 7713 cores, 64GB RAM, NVME disk for OS and containers and NVME disks for user data. If any additional data/logs/information is needed I will be happy to supply it. Any help will be appreciated.

Steps to replicate it (hint: details matter!):

    1. Sorry, i do not know how to reproduce this issue

Log entries

Nextcloud

Just a few logs, I’ll be happy to add more, it’s just a lot and I didn’t find anything interesting in them

:"Invalid token for WOPI access","exception":{},"CustomMessage":"Invalid token for WOPI access"}}
{"reqId":"EWNPe6mImRJEpCLYatQe","level":1,"time":"2025-05-28T08:55:55+00:00","remoteAddr":"127.0.0.1","user":"--","app":"richdocuments","method":"GET","url":"/index.php/apps/richdocuments/wopi/files/5759290_oci6ys2z3zqh?access_token=3Yi77QduMCPw8qUhvBScXOPhxRRHba2k&access_token_ttl=0&permission=edit","message":"Invalid token for WOPI access","userAgent":"COOLWSD HTTP Agent 24.04.13.3","version":"31.0.5.1","exception":{"Exception":"OCA\\Richdocuments\\Exceptions\\UnknownTokenException","Message":"Could not find token.","Code":0,"Trace":[{"file":"/var/www/html/custom_apps/richdocuments/lib/Middleware/WOPIMiddleware.php","line":66,"function":"getWopiForToken","class":"OCA\\Richdocuments\\Db\\WopiMapper","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Middleware/MiddlewareDispatcher.php","line":73,"function":"beforeController","class":"OCA\\Richdocuments\\Middleware\\WOPIMiddleware","type":"->","args":[{"__class__":"OCA\\Richdocuments\\Controller\\WopiController"},"checkFileInfo"]},{"file":"/var/www/html/lib/private/AppFramework/Http/Dispatcher.php","line":106,"function":"beforeController","class":"OC\\AppFramework\\Middleware\\MiddlewareDispatcher","type":"->","args":[{"__class__":"OCA\\Richdocuments\\Controller\\WopiController"},"checkFileInfo"]},{"file":"/var/www/html/lib/private/AppFramework/App.php","line":161,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OCA\\Richdocuments\\Controller\\WopiController"},"checkFileInfo"]},{"file":"/var/www/html/lib/private/Route/Router.php","line":307,"function":"main","class":"OC\\AppFramework\\App","type":"::","args":["OCA\\Richdocuments\\Controller\\WopiController","checkFileInfo",{"__class__":"OC\\AppFramework\\DependencyInjection\\DIContainer"},{"fileId":"5759290_oci6ys2z3zqh","_route":"richdocuments.wopi.checkfileinfo"}]},{"file":"/var/www/html/lib/base.php","line":1040,"function":"match","class":"OC\\Route\\Router","type":"->","args":["/apps/richdocuments/wopi/files/5759290_oci6ys2z3zqh"]},{"file":"/var/www/html/index.php","line":24,"function":"handleRequest","class":"OC","type":"::","args":[]}],"File":"/var/www/html/custom_apps/richdocuments/lib/Db/WopiMapper.php","Line":149,"message":"Invalid token for WOPI access","exception":{},"CustomMessage":"Invalid token for WOPI access"}}
{"reqId":"al6RetyJmaCV0gx9RlQo","level":1,"time":"2025-05-28T08:55:55+00:00","remoteAddr":"176.74.142.247","user":"jansychra","app":"no app in context","method":"GET","url":"/ocs/v2.php/apps/notifications/api/v2/notifications","message":"Notification was claimed to be parsed, but was not fully parsed by OCA\\Deck\\Notification\\Notifier [app: deck, subject: card_user_assign]","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36","version":"31.0.5.1","data":[]}
{"reqId":"isO6HkGVMaTT54bfaO1C","level":1,"time":"2025-05-28T08:53:55+00:00","remoteAddr":"176.74.142.247","user":"jansychra","app":"no app in context","method":"GET","url":"/ocs/v2.php/apps
/notifications/api/v2/notifications","message":"Notification was claimed to be parsed, but was not fully parsed by OCA\\Deck\\Notification\\Notifier [app: deck, subject: card_user_unassign]"
,"userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36","version":"31.0.5.1","data":[]}

Web Browser

Screenshot of opening random folders in UI.

Web server / Reverse Proxy

Caddy logs

{"level":"warn","ts":1748409137.0103688,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"localhost:11000","duration":0.001444589,"request":{"remote_ip":"81.19.0.62","remote_port":"48354","client_ip":"81.19.0.62","proto":"HTTP/3.0","method":"GET","host":"work.faster.cz","uri":"/core/l10n/cs.js?v=411cd5a2-11","headers":{"Dnt":["1"],"Alt-Used":["work.faster.cz"],"Sec-Fetch-Dest":["script"],"Accept-Encoding":["gzip, deflate, br, zstd"],"X-Forwarded-For":["81.19.0.62"],"X-Forwarded-Host":["work.faster.cz"],"User-Agent":["Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:138.0) Gecko/20100101 Firefox/138.0"],"Sec-Fetch-Site":["same-origin"],"Accept":["*/*"],"X-Forwarded-Proto":["https"],"Accept-Language":["cs,en-US;q=0.7,en;q=0.3"],"Sec-Gpc":["1"],"Cookie":["REDACTED"],"Sec-Fetch-Mode":["no-cors"],"Via":["3.0 Caddy"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"work.faster.cz"}},"error":"writing: H3_REQUEST_CANCELLED"}
{"level":"warn","ts":1748409137.010383,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"localhost:11000","duration":0.001717247,"request":{"remote_ip":"81.19.0.62","remote_port":"48354","client_ip":"81.19.0.62","proto":"HTTP/3.0","method":"GET","host":"work.faster.cz","uri":"/custom_apps/calendar/l10n/cs.js?v=411cd5a2-11","headers":{"Accept":["*/*"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Alt-Used":["work.faster.cz"],"User-Agent":["Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:138.0) Gecko/20100101 Firefox/138.0"],"Cookie":["REDACTED"],"Sec-Fetch-Mode":["no-cors"],"Sec-Gpc":["1"],"Accept-Language":["cs,en-US;q=0.7,en;q=0.3"],"Sec-Fetch-Site":["same-origin"],"X-Forwarded-For":["81.19.0.62"],"X-Forwarded-Host":["work.faster.cz"],"Via":["3.0 Caddy"],"Sec-Fetch-Dest":["script"],"Dnt":["1"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"work.faster.cz"}},"error":"writing: H3_REQUEST_CANCELLED"}
{"level":"warn","ts":1748409137.0103796,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"localhost:11000","duration":0.001716777,"request":{"remote_ip":"81.19.0.62","remote_port":"48354","client_ip":"81.19.0.62","proto":"HTTP/3.0","method":"GET","host":"work.faster.cz","uri":"/dist/core-unified-search.js?v=411cd5a2-11","headers":{"Accept-Language":["cs,en-US;q=0.7,en;q=0.3"],"Via":["3.0 Caddy"],"Sec-Fetch-Mode":["no-cors"],"User-Agent":["Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:138.0) Gecko/20100101 Firefox/138.0"],"Sec-Fetch-Dest":["script"],"Sec-Fetch-Site":["same-origin"],"Dnt":["1"],"X-Forwarded-Proto":["https"],"Sec-Gpc":["1"],"Accept":["*/*"],"Cookie":["REDACTED"],"X-Forwarded-Host":["work.faster.cz"],"Alt-Used":["work.faster.cz"],"Accept-Encoding":["gzip, deflate, br, zstd"],"X-Forwarded-For":["81.19.0.62"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"work.faster.cz"}},"error":"writing: H3_REQUEST_CANCELLED"}
{"level":"warn","ts":1748413303.8095448,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"localhost:11000","duration":0.001514177,"request":{"remote_ip":"81.19.0.62","remote_port":"54937","client_ip":"81.19.0.62","proto":"HTTP/3.0","method":"GET","host":"work.faster.cz","uri":"/browser/b7ba9a23ba/l10n/uno-localizations.json","headers":{"Sec-Fetch-Site":["same-origin"],"Cookie":["REDACTED"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["work.faster.cz"],"Dnt":["1"],"Sec-Fetch-Mode":["cors"],"User-Agent":["Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:138.0) Gecko/20100101 Firefox/138.0"],"Accept":["*/*"],"X-Forwarded-For":["81.19.0.62"],"Via":["3.0 Caddy"],"Accept-Language":["en-US,en;q=0.5"],"Sec-Fetch-Dest":["empty"],"Alt-Used":["work.faster.cz"],"Accept-Encoding":["gzip, deflate, br, zstd"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"work.faster.cz"}},"error":"writing: H3_REQUEST_CANCELLED"}
{"level":"warn","ts":1748422793.7497492,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"localhost:11000","duration":0.1143928,"request":{"remote_ip":"81.19.0.62","remote_port":"34569","client_ip":"81.19.0.62","proto":"HTTP/3.0","method":"GET","host":"work.faster.cz","uri":"/apps/activity/js/preload-helper-CfzTJz-5.chunk.mjs.map","headers":{"Sec-Fetch-Mode":["no-cors"],"Sec-Fetch-Site":["same-origin"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"],"Cookie":["REDACTED"],"Priority":["u=4, i"],"Accept-Language":["cs-CZ,cs;q=0.9"],"Pragma":["no-cache"],"Via":["3.0 Caddy"],"Cache-Control":["no-cache"],"Sec-Fetch-Dest":["empty"],"Accept-Encoding":["gzip, deflate, br, zstd"],"X-Forwarded-For":["81.19.0.62"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["work.faster.cz"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"work.faster.cz"}},"error":"writing: H3_REQUEST_CANCELLED"}
{"level":"warn","ts":1748422793.7506092,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"localhost:11000","duration":0.114862037,"request":{"remote_ip":"81.19.0.62","remote_port":"34569","client_ip":"81.19.0.62","proto":"HTTP/3.0","method":"GET","host":"work.faster.cz","uri":"/apps/activity/js/_commonjsHelpers-BdswosdL.chunk.mjs.map","headers":{"Pragma":["no-cache"],"Sec-Fetch-Site":["same-origin"],"Priority":["u=4, i"],"Cache-Control":["no-cache"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["work.faster.cz"],"Cookie":["REDACTED"],"Sec-Fetch-Dest":["empty"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"],"Sec-Fetch-Mode":["no-cors"],"X-Forwarded-For":["81.19.0.62"],"Via":["3.0 Caddy"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Accept-Language":["cs-CZ,cs;q=0.9"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"work.faster.cz"}},"error":"writing: H3_REQUEST_CANCELLED"}
{"level":"warn","ts":1748422793.7529721,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"localhost:11000","duration":0.116256049,"request":{"remote_ip":"81.19.0.62","remote_port":"34569","client_ip":"81.19.0.62","proto":"HTTP/3.0","method":"GET","host":"work.faster.cz","uri":"/apps/activity/js/translation-CD_FiYBO-CsihNwbo.chunk.mjs.map","headers":{"Cookie":["REDACTED"],"X-Forwarded-For":["81.19.0.62"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["work.faster.cz"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Accept-Language":["cs-CZ,cs;q=0.9"],"Sec-Fetch-Mode":["no-cors"],"Priority":["u=4, i"],"Via":["3.0 Caddy"],"Sec-Fetch-Dest":["empty"],"Cache-Control":["no-cache"],"Pragma":["no-cache"],"Sec-Fetch-Site":["same-origin"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"work.faster.cz"}},"error":"writing: H3_REQUEST_CANCELLED"}
{"level":"warn","ts":1748422793.7622297,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"localhost:11000","duration":0.118580964,"request":{"remote_ip":"81.19.0.62","remote_port":"34569","client_ip":"81.19.0.62","proto":"HTTP/3.0","method":"GET","host":"work.faster.cz","uri":"/custom_apps/bookmarks/js/bookmarks-references.js.map?v=fa69c36421929ea9cbad","headers":{"X-Forwarded-For":["81.19.0.62"],"Via":["3.0 Caddy"],"Cache-Control":["no-cache"],"Sec-Fetch-Site":["same-origin"],"Pragma":["no-cache"],"Sec-Fetch-Mode":["no-cors"],"Accept-Encoding":["gzip, deflate, br, zstd"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["work.faster.cz"],"Cookie":["REDACTED"],"Priority":["u=4, i"],"Sec-Fetch-Dest":["empty"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"],"Accept-Language":["cs-CZ,cs;q=0.9"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"work.faster.cz"}},"error":"writing: H3_REQUEST_CANCELLED"}

Configuration

Nextcloud

occ config:list system

{
    "system": {
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "overwritehost": "work.faster.cz",
        "overwriteprotocol": "https",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost",
            "work.faster.cz"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "pgsql",
        "version": "31.0.5.1",
        "overwrite.cli.url": "https:\/\/work.faster.cz\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "loglevel": 1,
        "log_type": "file",
        "logfile": "\/var\/www\/html\/data\/nextcloud.log",
        "log_rotate_size": "10485760",
        "skeletondirectory": "",
        "lost_password_link": "disabled",
        "log.condition": {
            "apps": [
                "admin_audit"
            ]
        },
        "preview_max_x": "2048",
        "preview_max_y": "2048",
        "jpeg_quality": "60",
        "enabledPreviewProviders": {
            "1": "OC\\Preview\\Image",
            "2": "OC\\Preview\\MarkDown",
            "3": "OC\\Preview\\MP3",
            "4": "OC\\Preview\\TXT",
            "5": "OC\\Preview\\OpenDocument",
            "6": "OC\\Preview\\Movie",
            "0": "OC\\Preview\\Imaginary",
            "23": "OC\\Preview\\ImaginaryPDF"
        },
        "enable_previews": true,
        "upgrade.disable-web": true,
        "mail_smtpmode": "smtp",
        "trashbin_retention_obligation": "auto, 30",
        "versions_retention_obligation": "auto, 30",
        "activity_expire_days": "30",
        "simpleSignUpLink.shown": false,
        "share_folder": "\/Shared",
        "one-click-instance": true,
        "one-click-instance.user-limit": 100,
        "one-click-instance.link": "https:\/\/nextcloud.com\/all-in-one\/",
        "htaccess.RewriteBase": "\/",
        "files_external_allow_create_new_local": false,
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "allow_local_remote_servers": true,
        "preview_imaginary_url": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpsecure": "ssl",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "allow_user_to_change_display_name": false,
        "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory",
        "updatedirectory": "\/nc-updater",
        "twofactor_enforced": "true",
        "twofactor_enforced_groups": [],
        "twofactor_enforced_excluded_groups": [],
        "onlyoffice": {
            "jwt_secret": "***REMOVED SENSITIVE VALUE***",
            "jwt_header": "AuthorizationJwt"
        },
        "upgrade.cli-upgrade-link": "https:\/\/github.com\/nextcloud\/all-in-one\/discussions\/2726",
        "davstorage.request_timeout": 3600,
        "dbpersistent": false,
        "appsallowlist": false,
        "no_unsupported_browser_warning": "true",
        "maintenance_window_start": 100,
        "preview_imaginary_key": "***REMOVED SENSITIVE VALUE***",
        "auth.bruteforce.protection.enabled": true,
        "ratelimit.protection.enabled": true,
        "default_phone_region": "CZ",
        "documentation_url.server_logs": "https:\/\/github.com\/nextcloud\/all-in-one\/discussions\/5425",
        "data-fingerprint": "b89f23100c6059bba95951dac41ab3d2",
        "app_install_overwrite": [
            "sharelisting",
            "nextcloud-aio",
            "workspace"
        ],
        "forbidden_filename_basenames": [
            "con",
            "prn",
            "aux",
            "nul",
            "com0",
            "com1",
            "com2",
            "com3",
            "com4",
            "com5",
            "com6",
            "com7",
            "com8",
            "com9",
            "com\u00b9",
            "com\u00b2",
            "com\u00b3",
            "lpt0",
            "lpt1",
            "lpt2",
            "lpt3",
            "lpt4",
            "lpt5",
            "lpt6",
            "lpt7",
            "lpt8",
            "lpt9",
            "lpt\u00b9",
            "lpt\u00b2",
            "lpt\u00b3"
        ],
        "forbidden_filename_characters": [
            "<",
            ">",
            ":",
            "\"",
            "|",
            "?",
            "*",
            "\\",
            "\/"
        ],
        "forbidden_filename_extensions": [
            " ",
            ".",
            ".filepart",
            ".part"
        ],
        "DOMAIN": "work.faster.cz"
    }
}

Apps

occ app:list

Enabled:
  - activity: 4.0.0
  - admin_audit: 1.21.0
  - appointments: 2.4.4
  - bookmarks: 15.1.0
  - bruteforcesettings: 4.0.0
  - calendar: 5.2.4
  - cfg_share_links: 7.0.1
  - circles: 31.0.0
  - cloud_federation_api: 1.14.0
  - comments: 1.21.0
  - contacts: 7.1.1
  - contactsinteraction: 1.12.0
  - dashboard: 7.11.0
  - dav: 1.33.0
  - deck: 1.15.1
  - drawio: 3.1.0
  - end_to_end_encryption: 1.17.0
  - external: 6.0.2
  - federatedfilesharing: 1.21.0
  - files: 2.3.1
  - files_accesscontrol: 2.0.0
  - files_antivirus: 6.0.2
  - files_downloadlimit: 4.0.0
  - files_fulltextsearch: 31.0.0
  - files_pdfviewer: 4.0.0
  - files_reminders: 1.4.0
  - files_sharing: 1.23.1
  - files_trashbin: 1.21.0
  - files_versions: 1.24.0
  - forms: 5.1.0
  - fulltextsearch: 31.0.0
  - fulltextsearch_elasticsearch: 31.0.0
  - groupfolders: 19.0.4
  - impersonate: 2.0.0
  - integration_mattermost: 2.1.0
  - logreader: 4.0.0
  - lookup_server_connector: 1.19.0
  - mail: 5.1.1
  - nextcloud-aio: 0.7.0
  - notes: 4.12.0
  - notifications: 4.0.0
  - notify_push: 1.1.0
  - oauth2: 1.19.1
  - photos: 4.0.0-dev.1
  - previewgenerator: 5.8.0
  - profile: 1.0.0
  - provisioning_api: 1.21.0
  - recommendations: 4.0.0
  - related_resources: 2.0.0
  - richdocuments: 8.6.5
  - serverinfo: 3.0.0
  - settings: 1.14.0
  - sharebymail: 1.21.0
  - sharelisting: 1.2.0
  - support: 3.0.0
  - suspicious_login: 9.0.1
  - tasks: 0.16.1
  - text: 5.0.0
  - theming: 2.6.1
  - theming_customcss: 1.18.0
  - twofactor_backupcodes: 1.20.0
  - twofactor_totp: 13.0.0-dev.0
  - updatenotification: 1.21.0
  - user_ldap: 1.22.0
  - user_oidc: 7.2.0
  - viewer: 4.0.0
  - whiteboard: 1.0.5
  - workflowengine: 2.13.0
  - workspace: 4.0.3
Disabled:
  - app_api: 5.0.2 (installed 4.0.5)
  - encryption: 2.19.0 (installed 2.15.0)
  - federation: 1.21.0 (installed 1.20.0)
  - files_external: 1.23.0 (installed 1.19.0)
  - firstrunwizard: 4.0.0 (installed 2.17.0)
  - nextcloud_announcements: 3.0.0 (installed 1.17.0)
  - password_policy: 3.0.0 (installed 2.0.0)
  - privacy: 3.0.0 (installed 2.0.0)
  - survey_client: 3.0.0 (installed 1.16.0)
  - systemtags: 1.21.1 (installed 1.20.0)
  - twofactor_nextcloud_notification: 5.0.0
  - user_status: 1.11.0 (installed 1.8.1)
  - weather_status: 1.11.0 (installed 1.8.0)
  - webhook_listeners: 1.2.0 (installed 1.1.0-dev)

Screenshots

Hi
I have a laptop that the cpu clock has dropped to low power mode and I can’t seem to unlock it.

Do you have the expected clock rate running?

cat /proc/cpuinfo | grep "MHz"

Hi, sure, it’s running on a Supermicro server in a datacenter. I don’t think this problem is HW related.

root@work:~# cat /proc/cpuinfo | grep "MHz"

cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000

You could ask your provider if VMs are allowed to scale above the base clock, or if there are any restrictions in place on their platform. I asked my own provider, and they confirmed that they don’t limit CPU frequency, but your still seeing your VM’s CPUs fixed at the base clock rate.

This makes me wonder if it’s related to the way the hypervisor or virtualization layer handles CPU scaling. It’s actually unusual for data centre CPUs to run strictly at base frequency, since most platforms support some level of dynamic scaling.

Let us know what you find out!

I am the provider, there are no limits, boost works as it should, several other VMs are running there without any problems.

Hypervisor proxmox:

Hypervisor Mhz:

cpu MHz		: 3095.616
cpu MHz		: 3099.194
cpu MHz		: 3099.418
cpu MHz		: 3092.771
cpu MHz		: 3099.666
cpu MHz		: 3099.119
cpu MHz		: 3099.451
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2975.592
cpu MHz		: 2000.000
cpu MHz		: 3039.862
cpu MHz		: 3092.335
cpu MHz		: 3100.057
cpu MHz		: 3098.155
cpu MHz		: 3100.094
cpu MHz		: 3099.277
cpu MHz		: 3099.214
cpu MHz		: 3099.218
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 3097.676
cpu MHz		: 3098.980
cpu MHz		: 3098.445
cpu MHz		: 3096.000
cpu MHz		: 3102.191
cpu MHz		: 2700.433
cpu MHz		: 2788.355
cpu MHz		: 3019.727
cpu MHz		: 3028.412
cpu MHz		: 3097.751
cpu MHz		: 3099.361
cpu MHz		: 3097.733
cpu MHz		: 3096.295
cpu MHz		: 3095.717
cpu MHz		: 3099.503
cpu MHz		: 3099.689
cpu MHz		: 3098.572
cpu MHz		: 3099.672
cpu MHz		: 3106.605
cpu MHz		: 3095.287
cpu MHz		: 3068.333
cpu MHz		: 3099.278
cpu MHz		: 3094.391
cpu MHz		: 3094.262
cpu MHz		: 3099.425
cpu MHz		: 3096.720
cpu MHz		: 2000.000
cpu MHz		: 3100.117
cpu MHz		: 3100.435
cpu MHz		: 3099.414
cpu MHz		: 3094.265
cpu MHz		: 2000.000
cpu MHz		: 3099.701
cpu MHz		: 3098.562
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 3100.103
cpu MHz		: 3099.760
cpu MHz		: 2957.746
cpu MHz		: 3098.957
cpu MHz		: 3100.221
cpu MHz		: 3099.973
cpu MHz		: 3105.298
cpu MHz		: 3099.483
cpu MHz		: 3096.044
cpu MHz		: 3099.647
cpu MHz		: 3099.485
cpu MHz		: 3099.532
cpu MHz		: 3099.389
cpu MHz		: 3099.250
cpu MHz		: 3099.614
cpu MHz		: 3103.952
cpu MHz		: 2000.000
cpu MHz		: 3100.095
cpu MHz		: 3100.533
cpu MHz		: 3100.391
cpu MHz		: 3098.995
cpu MHz		: 3099.427
cpu MHz		: 3098.099
cpu MHz		: 3100.771
cpu MHz		: 3098.324
cpu MHz		: 2000.000
cpu MHz		: 3099.293
cpu MHz		: 3098.671
cpu MHz		: 3099.537
cpu MHz		: 2947.215
cpu MHz		: 3100.080
cpu MHz		: 3093.148
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 3099.449
cpu MHz		: 3100.002
cpu MHz		: 3097.670
cpu MHz		: 3098.095
cpu MHz		: 3099.896
cpu MHz		: 3100.960
cpu MHz		: 3092.083
cpu MHz		: 3098.245
cpu MHz		: 3099.506
cpu MHz		: 3104.269
cpu MHz		: 3099.181
cpu MHz		: 3100.047
cpu MHz		: 3092.460
cpu MHz		: 3099.355
cpu MHz		: 3099.703
cpu MHz		: 3096.086
cpu MHz		: 2944.679
cpu MHz		: 3099.025
cpu MHz		: 3097.469
cpu MHz		: 3100.391
cpu MHz		: 3099.237
cpu MHz		: 3099.243
cpu MHz		: 3098.709
cpu MHz		: 2824.881
cpu MHz		: 2823.703
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 3097.320
cpu MHz		: 3100.062
cpu MHz		: 2000.000
cpu MHz		: 2000.000
cpu MHz		: 3100.013

Do you have a test/staging environment you can attempt to reproduce the behavior in (even if just to confirm it doesn’t happen there)?

Other than that some ideas:

  • set loglevel back to the default (2) to reduce noise in the logs. Then look specifically at the log period specific to some test transactions (at Caddy level too)
  • Are there any transactions that don’t seem to be impacted?
  • Are you able to confirm it did start specifically when you when to v30 or is that just a best guess at this point?
  • Maybe confirm both network and storage performance from the container but outside of Nextcloud itself (e.g. curl via command-line from the nextcloud app container)?
  • Is behavior the same for both groupfolders/workspace and normal folders?
  • Anything special about your Caddy configuration?

Hi, we don’t have a test environment.

  • I changed the log level to 2 and tried everything that is slow but there is nothing in the log at the time of the test.
  • I’d say calendar, contacts, notes. Actually anything that isn’t file related at all is pretty good, although it could be better. The files, decks, and photos are the worst.
  • It started pretty much gradually and it’s getting worse and worse. It wasn’t right after the upgrade.
  • As for the network, each node in the proxmox cluster has 4x25Gbe and with iperf I get gigabit(laptop has only gigabit).
  • In terms of storage, you can see on the benchmarks
fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=read --size=500m --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.33
Starting 1 process
TEST: Laying out IO file (1 file / 500MiB)
Jobs: 1 (f=1): [R(1)][80.0%][r=2758MiB/s][r=2758 IOPS][eta 00m:01s]
TEST: (groupid=0, jobs=1): err= 0: pid=1907494: Thu May 29 07:30:49 2025
  read: IOPS=2735, BW=2736MiB/s (2869MB/s)(10.0GiB/3743msec)
    slat (usec): min=7, max=2592, avg=15.81, stdev=46.65
    clat (usec): min=1947, max=26740, avg=11624.86, stdev=3445.40
     lat (usec): min=1962, max=26757, avg=11640.66, stdev=3442.96
    clat percentiles (usec):
     |  1.00th=[ 4752],  5.00th=[ 6587], 10.00th=[ 7635], 20.00th=[ 8717],
     | 30.00th=[ 9634], 40.00th=[10552], 50.00th=[11338], 60.00th=[12125],
     | 70.00th=[13173], 80.00th=[14353], 90.00th=[16319], 95.00th=[17695],
     | 99.00th=[20317], 99.50th=[21890], 99.90th=[26084], 99.95th=[26608],
     | 99.99th=[26870]
   bw (  MiB/s): min= 2676, max= 2786, per=99.97%, avg=2734.86, stdev=44.07, samples=7
   iops        : min= 2676, max= 2786, avg=2734.86, stdev=44.07, samples=7
  lat (msec)   : 2=0.02%, 4=0.43%, 10=33.45%, 20=64.78%, 50=1.33%
  cpu          : usr=0.53%, sys=6.20%, ctx=4638, majf=0, minf=28
  IO depths    : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.6%, 16=3.3%, 32=93.6%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
     issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=2736MiB/s (2869MB/s), 2736MiB/s-2736MiB/s (2869MB/s-2869MB/s), io=10.0GiB (10.7GB), run=3743-3743msec

Disk stats (read/write):
  vdb: ios=9764/0, merge=0/0, ticks=111442/0, in_queue=111443, util=91.66%
fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=write --size=500m --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [W(1)][70.0%][w=1134MiB/s][w=1134 IOPS][eta 00m:03s]
Jobs: 1 (f=1): [W(1)][100.0%][w=988MiB/s][w=987 IOPS][eta 00m:00s]  
TEST: (groupid=0, jobs=1): err= 0: pid=1909009: Thu May 29 07:33:07 2025
  write: IOPS=1016, BW=1016MiB/s (1066MB/s)(10.0GiB/10074msec); 0 zone resets
    slat (usec): min=17, max=767, avg=52.20, stdev=15.47
    clat (usec): min=1199, max=449656, avg=30569.97, stdev=28277.07
     lat (usec): min=1234, max=449703, avg=30622.17, stdev=28277.01
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[   13], 10.00th=[   17], 20.00th=[   24],
     | 30.00th=[   26], 40.00th=[   27], 50.00th=[   28], 60.00th=[   30],
     | 70.00th=[   32], 80.00th=[   34], 90.00th=[   40], 95.00th=[   47],
     | 99.00th=[   72], 99.50th=[  249], 99.90th=[  443], 99.95th=[  443],
     | 99.99th=[  447]
   bw (  KiB/s): min=239616, max=1513472, per=99.49%, avg=1035571.20, stdev=228976.52, samples=20
   iops        : min=  234, max= 1478, avg=1011.30, stdev=223.61, samples=20
  lat (msec)   : 2=0.23%, 4=0.42%, 10=2.68%, 20=9.92%, 50=82.83%
  lat (msec)   : 100=3.21%, 250=0.33%, 500=0.37%
  fsync/fdatasync/sync_file_range:
    sync (nsec): min=277485k, max=277485k, avg=277485270.00, stdev= 0.00
    sync percentiles (msec):
     |  1.00th=[  279],  5.00th=[  279], 10.00th=[  279], 20.00th=[  279],
     | 30.00th=[  279], 40.00th=[  279], 50.00th=[  279], 60.00th=[  279],
     | 70.00th=[  279], 80.00th=[  279], 90.00th=[  279], 95.00th=[  279],
     | 99.00th=[  279], 99.50th=[  279], 99.90th=[  279], 99.95th=[  279],
     | 99.99th=[  279]
  cpu          : usr=3.50%, sys=2.50%, ctx=9687, majf=0, minf=12
  IO depths    : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.6%, 16=3.3%, 32=93.6%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=1016MiB/s (1066MB/s), 1016MiB/s-1016MiB/s (1066MB/s-1066MB/s), io=10.0GiB (10.7GB), run=10074-10074msec

Disk stats (read/write):
  vdb: ios=0/10494, merge=0/2, ticks=0/314176, in_queue=315103, util=90.75%

  • Is behavior the same for both groupfolders/workspace and normal folders? Yes
  • Caddy configuration is the same as in the documentation

You might want to stop guessing and profile Nextcloud to see where exactly time is spent. I wrote a blog post about this if you want more details but the TL; DR is, assuming you have the xdebug extension installed, to add to php.ini something like this:

xdebug.mode = profile;
xdebug.trigger_value = debug_me;
xdebug.output_dir = /var/log/xdebug;
xdebug.start_with_request = trigger;

This will create a cachegrind.out.XXXXXX file under the /var/log/xdebug directory for each request having the cookie XDEBUG_PROFILE=debug_me set. You can then use kcachegrind for example to see those profiles.

1 Like

Hi,
sorry for the delay, I have nextcloud aio and I don’t know how to edit php.ini because it always gets overwritten. However, if I try time curl -X “PROPFIND” -H “Depth: 3” -u “USER:PASS” 'https://NC/remote.php/dav/files/USER/' I get a real value of 1,358s. But if I go into that path via app/web/webdav it’s much more.
Any further help would be appreciated :slight_smile:

The log you posted indicates a problem with WOPI/Collabora. Let me guess: As soon as you deactivate the richdocuments app, Nextcloud is back to normal speed?

If so, this is a known issue:

Hi, that’s not the problem, unfortunately. And even after deactivating richdocuments the problem still exists.

I think, like @ibizaman, that debugging with Xdebug would be the easiest way to get further here. Or activate PHP Slow Error Log - there may be something visible in it.

Just a few more wild ideas:

  • Nextcloud has brute force protection build-in. In combination with reverse proxies, it is possible that Nextcloud sees only on IP (of the reverse proxy) and thus rate limit all requests. However, this should normally be visible in the log.
  • You can deactivate apps until you can’t deactivate any more. Maybe there is an app that is causing the problem.

Hello sparkycz1,

I had the same issue long time ago and the reason was a DNS issue for the docker container and for the underlying server. Check, if the docker AIO main container has correct DNS resolution.

Cheers
Ranki

Hi, DNS resolution is working fine.

Hi,
I’ll try turning off all apps one by one and let you know.
As for brute force protection, caddy forwards a real IP address and I have our subnet in whitelist. So we can check it out.