Why is my Nextcloud frontend so slow?

Hi,

i’m facing an issue with my Nextcloud instance. The loading time of my frontend page is way to high (5s up to 60s) mean value is somewhere between 10s and 20s. I am running Nextcloud in a docker on a Intel core i5 desktop PC with 4GB of RAM and a Samsung SSD. The OS of the docker host is ubuntu 18 LTS. The server is connected via 5GHz WiFi to my home router. The measured bandwith (measured from my laptop) is somewhere around 100MBit/s.
Of course i have read the sections regarding performance tweaking of Nextcloud in the manual (docs.nextcloud.com). I have memory caching and redis enabled. Furthermore i use a tweaked ini file that i mount into my php folder

	pm = dynamic
	pm.max_children = 120
	pm.start_servers = 12
	pm.min_spare_servers = 6
	pm.max_spare_servers = 18

mounted with

	.tweak.ini:/usr/local/etc/php/conf.d/tweak.ini:ro

I have read the performance thread here (NextCloud Performance Optimizations - Please Contribute!), but applying the DB performance settings in the my.cnf file seem not to help.
I am using a synology NAS as a reverse proxy (nginx) and my user authentication is based on LDAP.

I’m not sure if one of those components can cause the bottleneck. But maybe someone can give me a hint on how i can tackle that issue.

Thanks,

BR,

Michael

Steps to reproduce

Browse the frontend of my nextcloud instance.

Expected behaviour

Loading time of the frontend pages and apps is between 1s and 2s

Actual behaviour

the loading time of the frontend pages and apps is between 5s and 60s, with a mean value of 10s.

Server configuration detail

Operating system: official nextcloud docker image (nextcloud:18)

Webserver: Apache/2.4.38 (Debian) (apache2handler)

Database: mysql 10.3.22

PHP version:

7.3.15
Modules loaded: Core, date, libxml, openssl, pcre, sqlite3, zlib, ctype, curl, dom, fileinfo, filter, ftp, hash, iconv, json, mbstring, SPL, PDO, session, posix, Reflection, standard, SimpleXML, pdo_sqlite, Phar, tokenizer, xml, xmlreader, xmlwriter, mysqlnd, apache2handler, apcu, exif, gd, gmp, imagick, intl, ldap, memcached, pcntl, pdo_mysql, pdo_pgsql, redis, sodium, zip, Zend OPcache

Nextcloud version: 18.0.1 - 18.0.1.3

Updated from an older Nextcloud/ownCloud or fresh install: yes

Where did you install Nextcloud from: hub.docker.com

Signing status

Array
(
)

List of activated apps
Enabled:
 - accessibility: 1.4.0
 - activity: 2.11.0
 - admin_audit: 1.8.0
 - announcementcenter: 3.7.0
 - bookmarks: 2.3.4
 - calendar: 2.0.2
 - camerarawpreviews: 0.7.3
 - checksum: 0.4.4
 - cloud_federation_api: 1.1.0
 - comments: 1.8.0
 - contacts: 3.2.0
 - dav: 1.14.0
 - deck: 0.8.0
 - event_update_notification: 1.0.1
 - external: 3.5.0
 - extract: 1.2.3
 - federatedfilesharing: 1.8.0
 - federation: 1.8.0
 - files: 1.13.1
 - files_automatedtagging: 1.8.2
 - files_external: 1.9.0
 - files_mindmap: 0.0.20
 - files_pdfviewer: 1.7.0
 - files_rightclick: 0.15.2
 - files_sharing: 1.10.1
 - files_trashbin: 1.8.0
 - files_versions: 1.11.0
 - files_videoplayer: 1.7.0
 - firstrunwizard: 2.7.0
 - issuetemplate: 0.6.0
 - logreader: 2.3.0
 - lookup_server_connector: 1.6.0
 - maps: 0.1.6
 - metadata: 0.11.1
 - news: 14.1.3
 - nextcloud_announcements: 1.7.0
 - notifications: 2.6.0
 - oauth2: 1.6.0
 - ocr: 6.0.25
 - ocsms: 2.1.7
 - onlyoffice: 4.1.4
 - password_policy: 1.8.0
 - phonetrack: 0.6.2
 - photos: 1.0.0
 - polls: 1.3.0
 - previewgenerator: 2.2.0
 - privacy: 1.2.0
 - provisioning_api: 1.8.0
 - quota_warning: 1.7.0
 - rainloop: 6.1.0
 - recommendations: 0.6.0
 - scanner: 0.1.8
 - serverinfo: 1.8.0
 - settings: 1.0.0
 - sharebymail: 1.8.0
 - smb_test: 0.2.4
 - social: 0.2.101
 - spreed: 8.0.5
 - support: 1.1.0
 - survey_client: 1.6.0
 - systemtags: 1.8.0
 - talk_simple_poll: 1.1.0
 - tasks: 0.12.1
 - text: 2.0.0
 - theming: 1.9.0
 - twofactor_backupcodes: 1.7.0
 - twofactor_email: 1.0.1
 - twofactor_gateway: 0.15.1
 - twofactor_totp: 4.1.3
 - updatenotification: 1.8.0
 - user_ldap: 1.8.0
 - viewer: 1.2.0
 - workflow_pdf_converter: 1.3.1
 - workflow_script: 1.3.1
 - workflowengine: 2.0.0
Disabled:
 - encryption
 - files_frommail
 - notes

Configuration (config/config.php)
{
    "htaccess.RewriteBase": "\/",
    "memcache.local": "\\OC\\Memcache\\APCu",
    "apps_paths": [
        {
            "path": "\/var\/www\/html\/apps",
            "url": "\/apps",
            "writable": false
        },
        {
            "path": "\/var\/www\/html\/custom_apps",
            "url": "\/custom_apps",
            "writable": true
        }
    ],
    "instanceid": "***REMOVED SENSITIVE VALUE***",
    "passwordsalt": "***REMOVED SENSITIVE VALUE***",
    "secret": "***REMOVED SENSITIVE VALUE***",
    "trusted_domains": [
        "localhost",
        "192.168.1.4",
        "mydomain.de"
    ],
    "version": "18.0.1.3",
    "datadirectory": "***REMOVED SENSITIVE VALUE***",
    "dbtype": "mysql",
    "overwrite.cli.url": "https:\/\/mydomain.de",
    "overwriteprotocol": "https",
    "dbname": "***REMOVED SENSITIVE VALUE***",
    "dbhost": "***REMOVED SENSITIVE VALUE***",
    "dbport": "",
    "dbtableprefix": "oc_",
    "mysql.utf8mb4": true,
    "dbuser": "***REMOVED SENSITIVE VALUE***",
    "dbpassword": "***REMOVED SENSITIVE VALUE***",
    "installed": true,
    "ldapIgnoreNamingRules": false,
    "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory",
    "memcache.distributed": "\\OC\\Memcache\\Redis",
    "memcache.locking": "\\OC\\Memcache\\Redis",
    "redis": {
        "host": "***REMOVED SENSITIVE VALUE***",
        "port": 6379
    },
    "maintenance": false,
    "has_rebuilt_cache": true,
    "theme": "",
    "loglevel": 2,
    "mail_smtpmode": "smtp",
    "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpport": 465,
    "mail_smtpsecure": "ssl",
    "mail_smtpauth": true,
    "mail_smtpauthtype": "LOGIN",
    "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
    "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
    "mail_from_address": "***REMOVED SENSITIVE VALUE***",
    "mail_domain": "***REMOVED SENSITIVE VALUE***",
    "app_install_overwrite": [
        "social"
    ]
}

Are you using external storage, if yes which one: sftp

Are you using encryption:

Are you using an external user-backend, if yes which one: LDAP

LDAP configuration (delete this par if not used)
background_sync_interval: 43200background_sync_offset: 0background_sync_prefix: s01cleanUpJobOffset: 0enabled: yesinstalled_version: 1.8.0s01_lastChange: 1582022207s01has_memberof_filter_support: 1s01home_folder_naming_rule: s01last_jpegPhoto_lookup: 0s01ldap_agent_password: cmtpdGhKYW9ucWdWOWgxOW1XTXE=s01ldap_attributes_for_group_search: cns01ldap_attributes_for_user_search: displayNames01ldap_backup_host: s01ldap_backup_port: 636s01ldap_base: dc=mydomain,dc=des01ldap_base_groups: cn=groups,dc=mydomain,dc=des01ldap_base_users: cn=users,dc=mydomain,dc=des01ldap_cache_ttl: 600s01ldap_configuration_active: 1s01ldap_default_ppolicy_dn: s01ldap_display_name: cns01ldap_dn: uid=nextcloud,cn=users,dc=mydomain,dc=des01ldap_dynamic_group_member_url: s01ldap_email_attr: mails01ldap_experienced_admin: 0s01ldap_expert_username_attr: uids01ldap_expert_uuid_group_attr: s01ldap_expert_uuid_user_attr: s01ldap_gid_number: gidNumbers01ldap_group_display_name: cns01ldap_group_filter: (|(cn=mydomain))s01ldap_group_filter_mode: 1s01ldap_group_member_assoc_attribute: members01ldap_groupfilter_groups: mydomains01ldap_groupfilter_objectclass: s01ldap_host: ldaps://mydomain.des01ldap_login_filter: (&(|(objectClass=inetOrgPerson))(uid=%uid))s01ldap_login_filter_mode: 1s01ldap_loginfilter_attributes: s01ldap_loginfilter_email: 0s01ldap_loginfilter_username: 1s01ldap_nested_groups: 0s01ldap_override_main_server: s01ldap_paging_size: 500s01ldap_port: 636s01ldap_quota_attr: s01ldap_quota_def: s01ldap_tls: 0s01ldap_turn_off_cert_check: 0s01ldap_turn_on_pwd_change: 1s01ldap_user_display_name_2: s01ldap_user_filter_mode: 1s01ldap_userfilter_groups: mydomains01ldap_userfilter_objectclass: inetOrgPersons01ldap_userlist_filter: (&(|(objectclass=inetOrgPerson))(|(memberof=cn=mydomain,cn=groups,dc=mydomain,dc=de)))s01use_memberof_to_detect_membership: 1types: authentication

Client configuration

Browser: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0

Operating system: Windows 10

Logs

Web server error log

Nextcloud log
[core] Error: Can't get app storage, app files_external, user not logged in
[PHP] Error: Undefined index: log-detail at /var/www/html/custom_apps/issuetemplate/lib/Detail.php#59
POST /apps/issuetemplate/render
[remote] Error: TypeError: Argument 1 passed to OCA\WorkflowEngine\Helper\LogContext::setScopes() must be of the type array, object given, called in /var/www/html/apps/workflowengine/lib/Service/RuleMatcher.php on line 146 at <<closure>>

 0. /var/www/html/apps/workflowengine/lib/Service/RuleMatcher.php line 146
    OCA\WorkflowEngine\Helper\LogContext->setScopes(OCA\WorkflowEngine\Helper\ScopeContext {})
 1. /var/www/html/apps/workflowengine/lib/Service/RuleMatcher.php line 114
    OCA\WorkflowEngine\Service\RuleMatcher->getMatchingOperations("OCA\\WorkflowPDFConverter\\Operation", false)
 2. /var/www/html/custom_apps/workflow_pdf_converter/lib/Operation.php line 110
    OCA\WorkflowEngine\Service\RuleMatcher->getFlows(false)
 3. /var/www/html/apps/workflowengine/lib/AppInfo/Application.php line 114
    OCA\WorkflowPDFConverter\Operation->onEvent("*** sensitive parameter replaced ***", "*** sensitive parameter replaced ***", OCA\WorkflowEngine\Service\RuleMatcher {})
 4. /var/www/html/3rdparty/symfony/event-dispatcher/EventDispatcher.php line 247
    OCA\WorkflowEngine\AppInfo\Application->OCA\WorkflowEngine\AppInfo\{closure}("*** sensitive parameters replaced ***")
 5. /var/www/html/3rdparty/symfony/event-dispatcher/EventDispatcher.php line 73
    Symfony\Component\EventDispatcher\EventDispatcher->callListeners([Closure {}], "*** sensitive parameter replaced ***", "*** sensitive parameter replaced ***")
 6. /var/www/html/lib/private/EventDispatcher/EventDispatcher.php line 80
    Symfony\Component\EventDispatcher\EventDispatcher->dispatch("*** sensitive parameter replaced ***", "*** sensitive parameter replaced ***")
 7. /var/www/html/lib/private/EventDispatcher/SymfonyAdapter.php line 63
    OC\EventDispatcher\EventDispatcher->dispatch("*** sensitive parameter replaced ***", "*** sensitive parameter replaced ***")
 8. /var/www/html/lib/private/Files/Node/HookConnector.php line 107
    OC\EventDispatcher\SymfonyAdapter->dispatch("*** sensitive parameter replaced ***", "*** sensitive parameter replaced ***")
 9. /var/www/html/lib/private/legacy/hook.php line 111
    OC\Files\Node\HookConnector->postCreate({path: "/SharedC ... "})
10. /var/www/html/apps/dav/lib/Connector/Sabre/File.php line 389
    OC_Hook::emit("OC_Filesystem", "post_create", {path: "/SharedC ... "})
11. /var/www/html/apps/dav/lib/Connector/Sabre/File.php line 321
    OCA\DAV\Connector\Sabre\File->emitPostHooks(false)
12. /var/www/html/apps/dav/lib/Connector/Sabre/Directory.php line 156
    OCA\DAV\Connector\Sabre\File->put(null)
13. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 1096
    OCA\DAV\Connector\Sabre\Directory->createFile(".lock", null)
14. /var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php line 525
    Sabre\DAV\Server->createFile("SharedCloud1/Fi ... k", null, null)
15. <<closure>>
    Sabre\DAV\CorePlugin->httpPut(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
16. /var/www/html/3rdparty/sabre/event/lib/EventEmitterTrait.php line 105
    call_user_func_array([Sabre\DAV\CorePlugin {},"httpPut"], [Sabre\HTTP\Requ ... }])
17. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 479
    Sabre\Event\EventEmitter->emit("method:PUT", [Sabre\HTTP\Requ ... }])
18. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 254
    Sabre\DAV\Server->invokeMethod(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
19. /var/www/html/apps/dav/appinfo/v1/webdav.php line 82
    Sabre\DAV\Server->exec()
20. /var/www/html/remote.php line 165
    require_once("/var/www/html/a ... p")
PUT /remote.php/webdav/SharedCloud1/Finanzen/Ausgaben/AusgabenApp/aba01b54-70be-4517-bbcb-f31116cf5936/.lock
[PHP] Error: "continue" targeting switch is equivalent to "break". Did you mean to use "continue 2"? at /var/www/html/apps/workflowengine/lib/Helper/LogContext.php#55
Browser log

Did you check the RAM/CPU usage, i/o performance during slow uploads?

according to top the apache2 process (which i assume is the one from the docker image) uses around 15-25%. the memory usage is around 1-2% for the apache2 process. Still after reloading my frontend page in the files app (just an example) it takes around 10s.

Try to disable talk and some other lheavy apps, if not using them.

Will try that. What other apps can be considered as heavy apps?

Try to investigate through js debugger in Google Chrome.

Redis and opcache should be caching all of that locally. It should not take 2-3 minutes to load a single page. This sounds like a database issue to me. Are you seeing long queries in your mysql logs? Any errors in the nextcloud logs? How about the php logs?

Where can i check if i have long queries in mysql? Where can i check the php log?
Nextcloud throws periodically errors see also in my initial post, there is a collapsibale nextcloud log section.

The docker logs return the following

user@myhost: docker logs mynextcloudcontainer | grep error

AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.24.0.4. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.24.0.4. Set the 'ServerName' directive globally to suppress this message
[Sat Mar 28 09:07:44.735425 2020] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.38 (Debian) PHP/7.3.15 configured -- resuming normal operations
[Sat Mar 28 09:07:44.735478 2020] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'
[Sat Mar 28 09:17:07.115602 2020] [php7:warn] [pid 43] [client 192.168.1.2:5000] PHP Warning:  "continue" targeting switch is equivalent to "break". Did you mean to use "continue 2"? in /var/www/html/apps/workflowengine/lib/Helper/LogContext.php on line 55

One more question: Do you know how i can check if my redis and opcache are configured correctly?

What should i look for? Exceptions? I’m not a web developer (not yet :wink: )
I opened the console window in the debugger. When loading the files app i can see one single error (this error occurs every time i reload the files app):

Failed to load resource: the server responded with a status of 404 ()
https://mydomain.de/ocs/v2.php/apps/text/workspace?path=%2F

disabling the talk app seem to cut off at least one second. Switching from calendar to files takes about 11s with the talk app and without only 10s :slight_smile: .

Why are you using docker if you don’t know the basics? This is going to be hard to troubleshoot.

I would recommend installing from scratch on baremetal or a baremetal KVM so you learn how to configure this correctly while following the guide.

Why are you using docker if you don’t know the basics?

Because i can.

I would recommend installing from scratch on baremetal or a baremetal KVM so you learn how to configure this correctly while following the guide.

I was there already. One reason i switched to docker was the “dependency hell”. At that time i was running Nextcloud/ownCloud on a ARM based NAS and installing it there was quite hard. That was 10 years ago (with ownCloud i guess). The second reason for using docker is the portability. I can setup my docker container and then easily port it to another machine and being independent of the cpu arch and OS. I guess you should know that.
The third reason to run it on docker (or to develop a stable docker-compose file) was the goal to run it in a swarm to have redundancy - so i can take my baremetal offline for maintenance.

I’ll check the slow running option for my mariadb.

It’s not meant to be insulting. You can google “long mysql queries” and come up with an answer on the first result. You are coming across as someone who doesn’t know what they are doing.

The nextcloud guide will show you everything to configure and there are other guides on google about setting up a nextcloud installation. I would recommend using those.

You can google “long mysql queries” and come up with an answer on
did that and setup my db to log mysql queries. Now, i’m waiting a bit and see if something comes up in this log.

The nextcloud guide will show you everything to configure and there are other guides on google about setting up a nextcloud installation. I would recommend using those.

i read the docs on docs.nextcloud.com (prior posting here). My configuration file, posted initially, is setup according those docs (in regards of caching). Futhermore i read the docs on tuning php and implemented the suggested solutions.

You are coming across as someone who doesn’t know what they are doing.

yeah, my fault, i asked too many questions.

I apologize I’ve had a rough few days. Everyone is welcome to ask questions here.

Are there slow MySQL queries? Is the backend S3?

Hi,

after a few days i retreived the “mariadb-slow.log” from my db container and analyzed the content. The log contained the command, that was executed, the query time and a few more things. I created a LINQPad script and was able to extract the following table:

Command+Table Max Query Time Number Of Queries Time Span
SELECT FROM oc_activity 42,764183 96 3.22:59:59
SELECT FROM oc_filecache 3,859758 95 3.22:59:43
SELECT FROM oc_news_items 2,241038 37 3.18:00:00
DELETE FROM oc_activity 1,547358 1 00:00:00

As it seems, the activity app is the one, that does the “heaviest” queries on my instance. But still 42, seconds is not much. And 96 queries within 4 days is actually the result of the cron job running each hour.

It does not seem to me, that this might cause the issue of my instance beeing slow at the frontend side.

Perhaps you can stop the cron-job for a few days and look for differences.

Okay lets go back to the basics here. What kind of storage are you using… 5.2k rpm hard drive?

The way I got around my slow nextcloud (primary S3 is slow for small files) is to use nginx with it’s built in cache option enabled. I found both redis and opcache to not be effective.

It’s divided:
The server runs on an SSD. The data directory of nextcloud is on a CIFS-Volume mounted with docker-volume-netshare and the CIFS-Server is a Synology NAS with two Seagate IronWolf (7,2K RPM). The connection between the server and the NAS is done via Gigabit-LAN.
The Maria-DB runs on the server in a docker container, where the data volume is a named volume on the Server itself (meaning the DB runs on a SSD).

Let’s assume this isn’t a hardware or network problem.

  1. What is the current memory_limit set to in php?

  2. Is this php-fpm?