Multiple problems with NC

The Basics

  • Nextcloud Server version (e.g., 29.x.x):
    • 30.0.5
  • Operating system and version (e.g., Ubuntu 24.04):
    • Debian 11.11
  • Reverse proxy and version _(e.g. nginx 1.27.2)
    • NPM 2.12.1
  • PHP version (e.g, 8.3):
    • 8.3.15
  • Is this the first time you’ve seen this error? (Yes / No):
    • no
  • When did this problem seem to first start?
    • around NC 29
  • Installation method (e.g. AlO, NCP, Bare Metal/Archive, etc.)
    • AIO
  • Are you using CloudfIare, mod_security, or similar? (Yes / No)
    • no

Summary of the issue you are facing:

Hi, I would like to start by thanking you for the great work and also for any help you can offer me now.
Since about version 29 we have been having strange problems with nextcloud. In particular, it eats a lot of memory and then just freezes. We also have problems like not being able to open a photo or pdf file, not loading thumbnails and more. It always helps to shut down all containers, restart the VM and boot everything back up, but it lasts about 4-14 days.
Nextcloud runs as a virtual server, Debian 11.11, has 64GB RAM, 32 AMD EPYC 7713 threads and all data on NVMe disks. So I don’t think there’s a performance issue.
I have attached the memory usage of the nextcloud-aio-nextcloud container, the memory usage of the whole VM and the system from nextcloud.
Further, I don’t think whatever is in the logs is related to my problem because there is nothing in the lots during or before the problem.

Steps to replicate it (hint: details matter!):

  1. Sorry, i do not know how to reproduce this issue

Log entries

Nextcloud

These are the only problems I found in the log, I can supply more logs. These are all logs, I have all logs redirected to journald.

Jan 28 08:15:44 work 5d7a0a7d71a1[862]: frk-00013-00013 2025-01-28 07:15:44.310939 +0000 [ forkit ] WRN  The systemplate directory [/opt/cool/systemplate] is read-only, and at least [/opt/cool/systemplate//etc/hosts] is out-of-date. Will have to clone dynamic elements of systemplate to the jails. To restore optimal performance, make sure the files in [/opt/cool/systemplate/etc] >

Jan 28 08:07:37 work 5d7a0a7d71a1[862]: wsd-00007-00029 2025-01-28 07:07:37.143796 +0000 [ websrv_poll ] ERR  #48: Socket write returned -1 (EPIPE: Broken pipe)| net/Socket.hpp:1612

Jan 28 07:59:30 work 5d7a0a7d71a1[862]: wsd-00007-00029 2025-01-28 06:59:30.789294 +0000 [ websrv_poll ] ERR  #21: WOPI::CheckFileInfo failed for URI [abcd.efg.cz/index.php/apps/richdocuments/wopi/files/5676840_oci6ys2z3zqh?access_token=TP93zGEfRi5OSs8WSMxmixc7q1v4xgWl&access_token_ttl=0&permission=edit]: 403 (Forbidden) Forbidden. Headers: Server: openresty / Date: T>

Jan 28 07:59:30 work 5d7a0a7d71a1[862]: wsd-00007-00029 2025-01-28 06:59:30.789319 +0000 [ websrv_poll ] ERR  #21: Access denied to [https://abcd.efg.cz/index.php/apps/richdocuments/wopi/files/5676840_oci6ys2z3zqh?access_token=TP93zGEfRi5OSs8WSMxmixc7q1v4xgWl&access_token_ttl=0&permission=edit]| wsd/wopi/CheckFileInfo.cpp:112

Configuration

Nextcloud

{
    "system": {
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "overwritehost": "abcd.efg.cz",
        "overwriteprotocol": "https",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost",
            "abcd.efg.cz"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "pgsql",
        "version": "30.0.5.1",
        "overwrite.cli.url": "https:\/\/abcd.efg.cz\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "loglevel": 1,
        "log_type": "file",
        "logfile": "\/var\/www\/html\/data\/nextcloud.log",
        "log_rotate_size": "10485760",
        "skeletondirectory": "",
        "lost_password_link": "disabled",
        "log.condition": {
            "apps": [
                "admin_audit"
            ]
        },
        "preview_max_x": "2048",
        "preview_max_y": "2048",
        "jpeg_quality": "60",
        "enabledPreviewProviders": {
            "1": "OC\\Preview\\Image",
            "2": "OC\\Preview\\MarkDown",
            "3": "OC\\Preview\\MP3",
            "4": "OC\\Preview\\TXT",
            "5": "OC\\Preview\\OpenDocument",
            "6": "OC\\Preview\\Movie",
            "0": "OC\\Preview\\Imaginary",
            "23": "OC\\Preview\\ImaginaryPDF"
        },
        "enable_previews": true,
        "upgrade.disable-web": true,
        "mail_smtpmode": "smtp",
        "trashbin_retention_obligation": "auto, 30",
        "versions_retention_obligation": "auto, 30",
        "activity_expire_days": "30",
        "simpleSignUpLink.shown": false,
        "share_folder": "\/Shared",
        "one-click-instance": true,
        "one-click-instance.user-limit": 100,
        "one-click-instance.link": "https:\/\/nextcloud.com\/all-in-one\/",
        "htaccess.RewriteBase": "\/",
        "files_external_allow_create_new_local": false,
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "allow_local_remote_servers": true,
        "preview_imaginary_url": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpsecure": "ssl",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "allow_user_to_change_display_name": false,
        "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory",
        "updatedirectory": "\/nc-updater",
        "twofactor_enforced": "true",
        "twofactor_enforced_groups": [],
        "twofactor_enforced_excluded_groups": [],
        "onlyoffice": {
            "jwt_secret": "***REMOVED SENSITIVE VALUE***",
            "jwt_header": "AuthorizationJwt"
        },
        "upgrade.cli-upgrade-link": "https:\/\/github.com\/nextcloud\/all-in-one\/discussions\/2726",
        "davstorage.request_timeout": 3600,
        "dbpersistent": false,
        "appsallowlist": false,
        "no_unsupported_browser_warning": "true",
        "maintenance_window_start": 100,
        "preview_imaginary_key": "***REMOVED SENSITIVE VALUE***",
        "auth.bruteforce.protection.enabled": true,
        "ratelimit.protection.enabled": true,
        "default_phone_region": "CZ",
        "documentation_url.server_logs": "https:\/\/github.com\/nextcloud\/all-in-one\/discussions\/5425",
        "data-fingerprint": "b2fafa8f875d7424f32388049b2922b3",
        "DOMAIN": "abcd.efg.cz"
    }
}

Apps

Enabled:
  - activity: 3.0.0
  - admin_audit: 1.20.0
  - announcementcenter: 7.1.0
  - appointments: 2.3.5
  - assistant: 2.3.0
  - bookmarks: 15.0.5
  - bruteforcesettings: 3.0.0
  - calendar: 5.0.9
  - cfg_share_links: 6.1.1
  - circles: 30.0.0
  - cloud_federation_api: 1.13.0
  - comments: 1.20.1
  - contacts: 6.1.3
  - contactsinteraction: 1.11.0
  - dashboard: 7.10.0
  - dav: 1.31.1
  - deck: 1.14.3
  - drawio: 3.0.3
  - end_to_end_encryption: 1.16.2
  - external: 5.5.2
  - federatedfilesharing: 1.20.0
  - federation: 1.20.0
  - files: 2.2.0
  - files_accesscontrol: 1.20.1
  - files_downloadlimit: 3.0.0
  - files_fulltextsearch: 30.0.0
  - files_pdfviewer: 3.0.0
  - files_reminders: 1.3.0
  - files_sharing: 1.22.0
  - files_trashbin: 1.20.1
  - files_versions: 1.23.0
  - forms: 4.3.5
  - fulltextsearch: 30.0.0
  - fulltextsearch_elasticsearch: 30.0.0
  - groupfolders: 18.0.9
  - impersonate: 1.17.1
  - integration_mattermost: 2.1.0
  - integration_openai: 3.4.0
  - logreader: 3.0.0
  - lookup_server_connector: 1.18.0
  - mail: 4.1.2
  - nextcloud-aio: 0.7.0
  - notes: 4.11.0
  - notifications: 3.0.0
  - notify_push: 1.0.0
  - oauth2: 1.18.1
  - password_policy: 2.0.0
  - photos: 3.0.2
  - previewgenerator: 5.7.0
  - privacy: 2.0.0
  - provisioning_api: 1.20.0
  - recommendations: 3.0.0
  - related_resources: 1.5.0
  - richdocuments: 8.5.3
  - serverinfo: 2.0.0
  - settings: 1.13.0
  - sharebymail: 1.20.0
  - support: 2.0.0
  - suspicious_login: 8.0.0
  - systemtags: 1.20.0
  - tasks: 0.16.1
  - text: 4.1.0
  - theming: 2.5.0
  - theming_customcss: 1.18.0
  - twofactor_backupcodes: 1.19.0
  - twofactor_totp: 12.0.0-dev
  - user_ldap: 1.21.0
  - user_oidc: 6.2.1
  - viewer: 3.0.0
  - webhook_listeners: 1.1.0-dev
  - whiteboard: 1.0.5
  - workflowengine: 2.12.0
  - workspace: 4.0.0
Disabled:
  - app_api: 4.0.5 (installed 4.0.5)
  - encryption: 2.18.0 (installed 2.15.0)
  - files_external: 1.22.0 (installed 1.19.0)
  - firstrunwizard: 3.0.0 (installed 2.17.0)
  - nextcloud_announcements: 2.0.0 (installed 1.17.0)
  - passwords: 2024.12.21 (installed 2024.12.21)
  - sharelisting: 1.2.0 (installed 1.2.0)
  - survey_client: 2.0.0 (installed 1.16.0)
  - twofactor_nextcloud_notification: 4.0.0
  - user_status: 1.10.0 (installed 1.8.1)
  - weather_status: 1.10.0 (installed 1.8.0)

Screenshots


I am attaching more logs.

[Tue Jan 28 13:03:22.094216 2025] [mpm_event:error] [pid 61:tid 61] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

Jan 28 14:16:44 work 5d7a0a7d71a1[862]: kit-31448-31448 2025-01-28 13:16:44.858096 +0000 [ kitbroker_0c4 ] WRN  Unusual extremely large tile combine of size 5120x512| common/RenderTiles.hpp:115

Jan 28 14:16:22 work 5d7a0a7d71a1[862]: kit-31370-31370 2025-01-28 13:16:22.935817 +0000 [ kitbroker_0c2 ] ERR  ToMaster-f38: error: cmd=getclipboard kind=nodocloaded| common/Session.hpp:175
Jan 28 14:16:22 work 5d7a0a7d71a1[862]: wsd-00007-31388 2025-01-28 13:16:22.935906 +0000 [ docbroker_0c2 ] ERR  ToClient-f38: getclipboard error failure: nodocloaded| wsd/ClientSession.cpp:2021
Jan 28 14:16:23 work 5d7a0a7d71a1[862]: Document [https://abcd.efg.cz:443/index.php/apps/richdocuments/wopi/files/5754936_oci6ys2z3zqh] has no more views, exiting bluntly.
Jan 28 14:16:23 work 5d7a0a7d71a1[862]: kit-31370-31370 2025-01-28 13:16:23.162528 +0000 [ kitbroker_0c2 ] FTL  Document [https://abcd.efg.cz:443/index.php/apps/richdocuments/wopi/files/5754936_oci6ys2z3zqh] has no more views, exiting bluntly.| kit/Kit.cpp:898
Jan 28 14:16:23 work 5d7a0a7d71a1[862]: wsd-00007-31388 2025-01-28 13:16:23.171960 +0000 [ docbroker_0c2 ] WRN  #29: DocBroker [https%3A%2F%2Fabcd.efg.cz%3A443%2Findex.php%2Fapps%2Frichdocuments%2Fwopi%2Ffiles%2F5754936_oci6ys2z3zqh] got disconnected from its Kit (31370) unexpectedly. Closing| wsd/COOLWSD.cpp:2735
Jan 28 14:16:25 work 5d7a0a7d71a1[862]: frk-00013-00013 2025-01-28 13:16:25.599683 +0000 [ forkit ] WRN  The systemplate directory [/opt/cool/systemplate] is read-only, and at least [/opt/cool/systemplate//etc/hosts] is out-of-date. Will have to clone dynamic elements of systemplate to the jails. To restore optimal performance, make sure the files in [/opt/cool/systemplate/etc] >
Jan 28 14:16:28 work 5d7a0a7d71a1[862]: kit-31389-31389 2025-01-28 13:16:28.615660 +0000 [ kitbroker_0c3 ] ERR  ToMaster-f40: error: cmd=getclipboard kind=nodocloaded| common/Session.hpp:175
Jan 28 14:16:28 work 5d7a0a7d71a1[862]: wsd-00007-31447 2025-01-28 13:16:28.615769 +0000 [ docbroker_0c3 ] ERR  ToClient-f40: getclipboard error failure: nodocloaded| wsd/ClientSession.cpp:2021
Jan 28 14:16:29 work 5d7a0a7d71a1[862]: Document [https://abcd.efg.cz:443/index.php/apps/richdocuments/wopi/files/5754936_oci6ys2z3zqh] has no more views, exiting bluntly.
Jan 28 14:16:29 work 5d7a0a7d71a1[862]: kit-31389-31389 2025-01-28 13:16:29.279269 +0000 [ kitbroker_0c3 ] FTL  Document [https://abcd.efg.cz:443/index.php/apps/richdocuments/wopi/files/5754936_oci6ys2z3zqh] has no more views, exiting bluntly.| kit/Kit.cpp:898
Jan 28 14:16:29 work 5d7a0a7d71a1[862]: wsd-00007-31447 2025-01-28 13:16:29.289094 +0000 [ docbroker_0c3 ] WRN  #37: DocBroker [https%3A%2F%2Fabcd.efg.cz%3A443%2Findex.php%2Fapps%2Frichdocuments%2Fwopi%2Ffiles%2F5754936_oci6ys2z3zqh] got disconnected from its Kit (31389) unexpectedly. Closing| wsd/COOLWSD.cpp:2735
Jan 28 14:16:29 work 5d7a0a7d71a1[862]: frk-00013-00013 2025-01-28 13:16:29.824698 +0000 [ forkit ] WRN  The systemplate directory [/opt/cool/systemplate] is read-only, and at least [/opt/cool/systemplate//etc/hosts] is out-of-date. Will have to clone dynamic elements of systemplate to the jails. To restore optimal performance, make sure the files in [/opt/cool/systemplate/etc] >
Jan 28 14:16:31 work 5d7a0a7d71a1[862]: wsd-00007-00029 2025-01-28 13:16:31.998857 +0000 [ websrv_poll ] ERR  FileServerRequestHandler: File not found: Invalid URI request (hash): [/browser/b6712dc6c4/images/dark/lc_convertformulatovalue.svg].| wsd/FileServer.cpp:779

Jan 28 14:12:59 work 7e71c2aa0306[862]: Connection to nextcloud-aio-nextcloud (172.18.0.8) 9000 port [tcp/*] succeeded!
Jan 28 14:13:01 work 7e71c2aa0306[862]: [Tue Jan 28 13:13:01.020764 2025] [mpm_event:notice] [pid 44:tid 44] AH00489: Apache/2.4.62 (Unix) configured -- resuming normal operations
Jan 28 14:13:01 work 7e71c2aa0306[862]: [Tue Jan 28 13:13:01.020830 2025] [core:notice] [pid 44:tid 44] AH00094: Command line: '/usr/local/apache2/bin/httpd -D FOREGROUND'
Jan 28 14:13:01 work 7e71c2aa0306[862]: {"level":"info","ts":1738069981.0390694,"msg":"using config from file","file":"/tmp/Caddyfile"}
Jan 28 14:13:01 work 7e71c2aa0306[862]: {"level":"info","ts":1738069981.0411556,"msg":"adapted config to JSON","adapter":"caddyfile"}
Jan 28 14:13:12 work 5d7a0a7d71a1[862]: frk-00013-00013 2025-01-28 13:13:12.885259 +0000 [ forkit ] WRN  The systemplate directory [/opt/cool/systemplate] is read-only, and at least [/opt/cool/systemplate//etc/hosts] is out-of-date. Will have to clone dynamic elements of systemplate to the jails. To restore optimal performance, make sure the files in [/opt/cool/systemplate/etc] >
Jan 28 14:13:13 work 5d7a0a7d71a1[862]: kit-31130-31130 2025-01-28 13:13:13.138330 +0000 [ kitbroker_0c1 ] ERR  Document [https%3A%2F%2Fabcd.efg.cz%3A443%2Findex.php%2Fapps%2Frichdocuments%2Fwopi%2Ffiles%2F5754936_oci6ys2z3zqh] is not loaded.| kit/Kit.cpp:2442
Jan 28 14:13:13 work 5d7a0a7d71a1[862]: kit-31130-31130 2025-01-28 13:13:13.140632 +0000 [ kitbroker_0c1 ] ERR  ToMaster-f33: Exception while handling [commandvalues command=.uno:SheetGeometryData]: Document https%3A%2F%2Fabcd.efg.cz%3A443%2Findex.php%2Fapps%2Frichdocuments%2Fwopi%2Ffiles%2F5754936_oci6ys2z3zqh is not loaded.| common/Session.cpp:321
Jan 28 14:13:13 work 5d7a0a7d71a1[862]: kit-31130-31130 2025-01-28 13:13:13.140657 +0000 [ kitbroker_0c1 ] ERR  Document [https%3A%2F%2Fabcd.efg.cz%3A443%2Findex.php%2Fapps%2Frichdocuments%2Fwopi%2Ffiles%2F5754936_oci6ys2z3zqh] is not loaded.| kit/Kit.cpp:2442
Jan 28 14:13:13 work 5d7a0a7d71a1[862]: kit-31130-31130 2025-01-28 13:13:13.140668 +0000 [ kitbroker_0c1 ] ERR  ToMaster-f33: Exception while handling [commandvalues command=.uno:ViewAnnotations]: Document https%3A%2F%2Fabcd.efg.cz%3A443%2Findex.php%2Fapps%2Frichdocuments%2Fwopi%2Ffiles%2F5754936_oci6ys2z3zqh is not loaded.| common/Session.cpp:321

The logs you posted seem to be just container logs, but not Nextcloud logs. Can you check your nextcloud.log during time periods where you have these events?

I have attached the memory usage of the nextcloud-aio-nextcloud container, the memory usage of the whole VM and the system from nextcloud.

I only see the VM + System. How much of the memory usage is just for the nextcloud-aio-nextcloud container? How does the memory of that contain compare to the other active containers (e.g. Collabora, etc).

What are the biggest memory consumers at a process level from the perspective of the underlying host (VM)?

Perhaps also post your Compose file for completeness.

Sorry, the logs are way too long for this. This is for whole yesterday: Work Faster CZ
Nextcloud froze around 1:00 yesterday.

Docker compose:

services:
  nextcloud-aio-mastercontainer:
    image: nextcloud/all-in-one:latest
    init: true
    restart: always
    container_name: nextcloud-aio-mastercontainer 
    volumes:
      - nextcloud_aio_mastercontainer:/mnt/docker-aio-config
      - /var/run/docker.sock:/var/run/docker.sock:ro
    ports:
      - 8080:8080
    environment: 
      APACHE_PORT: 11000
      APACHE_IP_BINDING: 0.0.0.0
      NEXTCLOUD_DATADIR: /mnt/ncdata
      NEXTCLOUD_MEMORY_LIMIT: 2048M

volumes: 
  nextcloud_aio_mastercontainer:
    external: true

I forgot to attach one screenshot, this now and chart of this month of nextcloud-aio-nextcloud.


And this is now, all containers.

The biggest memory consumer is elastic search.

So, i have done:

  • Public IP for server
  • Upgrade Debian 11 to 12
  • NPM to recommended Caddy

For now, it looks better and faster.
I’ll leave it like this and see how it looks in a week or two.