Nextcloud 27 - all Page-load times about 5 seconds, can it be made faster?

🏷️ General
1

This is a topic I have read quite a few posts about, but I haven’t found a cause or solution. I am about a month into working with Nextcloud, starting with 27.0.2 to the now current 27.1.2. I have Nextcloud set up on a test box as a normal web application and I have worked with the config to enable all recommended transactional locks and caching and I have eliminated all warnings. The Admin “Overview” checks report “All checks passed”. Nextcloud carries out all operations brilliantly (no warnings of errors in any of the server or Nextcloud log files).

The problem is page-load times are frustratingly slow. Clicking any of the apps in the menubar results in a consistent 5 second delay before the information is shown. That’s a pretty long wait, 1-1000, 2-1000, 3-1000, 4-1000, 5-1000… Photos is a little better at just over 4 seconds, but there are only 3 images total. Task is likewaise a little better at about 4 seconds, but there are only 2 test-tasks entered.

The box is an older Core2 Duo, so not snappy, but fine for test purposes and no reason page-load times should not be a reasonable 1 second or less. I run other similar web applications (with the exact same contacts and calendar entries also in a MariaDB database) and there is no problem with page-load times (all pages are generated and shown in sub-1-second times, including the same contacts and calendar entries). This is something unique to Nextcloud.

I have looked over a number of related links below and I have collected the information that was requested in the other posts to try and sort out where the issue is. The links are:

Related Links

The crux of the other posts is 4G of RAM on a lightly loaded x86_64 server with only a few users should be fine. I am the only user and I’m accessing the server over a gigabit LAN, so network I/O isn’t an issue. The server, while not exposed publicly, has a 300 MBs internet connection. The details of the install follow.

OS / Server Software

Nextcloud is installed as a normal webapp on the host web-server, not inside a container or any other virtualized install that would add overhead. The server software running on this small Dell test-box is:

  Nextcloud : 27.1.2
  Archlinux : 6.5.5-arch1-1
  Apache    : 2.4.57 (Unix) 
  OpenSSL   : 3.1.3 
  mod_fcgid : 2.3.9 
  PHP       : 8.1.23
  PHP-APCu  : 5.1.22
  Redis     : 7.2.1
  memcached : 1.6.21

Hardware

This is an old Dell computer I use as a test or prototype box before migrating apps to a production server. It’s a slow old Core2 Duo with 4G and a WD Carvair Blue internal SATA-III drive. While it is simply an old test box, it should be more than enough to get reasonable page-load times from Nextcloud. With similar web applications (eGroupware, etc…) all page loads are less than 1 second. This is using the same contacts (approximately 600 contacts) and calendar information (less than 5 current entries) in MariaDB databases. In fact the calendar and contacts in Nextcloud were imported from eGroupware.

Note, the page-load times did not change significantly between the time the Nextcloud database was empty and after the contacts and calendar entries were imported. Contacts was the most impacted which added another second or so for the 600 contacts.

The following is the information requested in the related links above that may impact page load-times:

CPU type        : Intel(R) Core(TM)2 Duo CPU     E7300  @ 2.66GHz
How much RAM    : 4G
Network speed   : LAN-Gigabit/GAT6, Internet-300 MBs
Hard disk       : Internal SATA-III, Western Digital Caviar Blue Serial ATA
How many users  : 1
Office suites   : None
Other services  : None
System Load     : (top output below)
  
  top - 02:21:12 up 23:57,  1 user,  load average: 0.08, 0.02, 0.01
  Tasks: 135 total,   1 running, 134 sleeping,   0 stopped,   0 zombie
  %Cpu0  :   0.0/0.0     0[                                                 ]
  %Cpu1  :   0.0/0.0     0[                                                 ]
  MiB Mem : 23.1/3844.1   [||||||||||||                                     ]
  MiB Swap:  0.0/1219.9   [                                                 ]

Per the MariaDB recommendation in Configuring Swappiness I have set vm.swappiness=1. There is no paging to swap taking place.

The File Access Restriction app is NOT installed – that was mentioned as a possible cause for slow page-loads in the Version 15-16 timeframe. I don’t even see that app available to install any more.

This test box is at idle waiting to serve Nextcloud requests. That is a primary reason the slow page-load times are of particular concern. There isn’t anything else running competing for CPU time. I could see the box being the culprit if the cores were saturated or the RAM exhausted, but that’s not the case. Nextcloud has complete access to all the CPU with well over half the RAM available for use.

Nextcloud Config

{
    "system": {
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "logfile": "\/var\/log\/nextcloud\/nextcloud.log",
        "default_locale": "en_US",
        "default_phone_region": "US",
        "knowledgebaseenabled": true,
        "apps_paths": [
            {
                "path": "\/usr\/share\/webapps\/nextcloud\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/lib\/nextcloud\/apps",
                "url": "\/wapps",
                "writable": true
            }
        ],
        "trusted_domains": [
            "localhost",
            "2pi.3111skyline.com",
            "192.168.6.111"
        ],
        "overwrite.cli.url": "https:\/\/2pi.3111skyline.com\/nextcloud",
        "htaccess.RewriteBase": "\/nextcloud",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "27.1.2.1",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_sendmailmode": "smtp",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "25",
        "maintenance": false,
        "app_install_overwrite": [
            "issuetemplate"
        ],
        "theme": "",
        "loglevel": 2,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "filelocking.enabled": true,
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "timeout": 0
        }
    }
}

Nextcloud Enabled Apps

The apps are the standard set of apps that come enable with the Archlinux install. The only app I have added is the “theming_customcss” – though currently there is no custom CSS configured. The apps and versions I have are:

Enabled:
  - activity: 2.19.0
  - calendar: 4.5.2
  - circles: 27.0.1
  - cloud_federation_api: 1.10.0
  - comments: 1.17.0
  - contacts: 5.4.2
  - dashboard: 7.7.0
  - dav: 1.27.0
  - federatedfilesharing: 1.17.0
  - federation: 1.17.0
  - files: 1.22.0
  - files_pdfviewer: 2.8.0
  - files_reminders: 1.0.0
  - files_rightclick: 1.6.0
  - files_sharing: 1.19.0
  - files_trashbin: 1.17.0
  - files_versions: 1.20.0
  - firstrunwizard: 2.16.0
  - groupfolders: 15.3.1
  - logreader: 2.12.0
  - lookup_server_connector: 1.15.0
  - nextcloud_announcements: 1.16.0
  - notes: 4.8.1
  - notifications: 2.15.0
  - oauth2: 1.15.1
  - password_policy: 1.17.0
  - photos: 2.3.0
  - privacy: 1.11.0
  - provisioning_api: 1.17.0
  - related_resources: 1.2.0
  - serverinfo: 1.17.0
  - settings: 1.9.0
  - sharebymail: 1.17.0
  - support: 1.10.0
  - survey_client: 1.15.0
  - systemtags: 1.17.0
  - tasks: 0.15.0
  - text: 3.8.0
  - theming: 2.2.0
  - theming_customcss: 1.14.0
  - twofactor_backupcodes: 1.16.0
  - updatenotification: 1.17.0
  - user_status: 1.7.0
  - viewer: 2.1.0
  - weather_status: 1.7.0
  - workflowengine: 2.9.0

Web Server Log / Errors

There are no web-server warnings or errors since the server reload. All Nextcloud menubar apps have been used since that time (great job!). The Nextcloud code is in really, really great shape.

[Thu Oct 05 21:41:31.602376 2023] [mpm_prefork:notice] [pid 421] AH00163: Apache/2.4.57 (Unix) OpenSSL/3.1.3 mod_fcgid/2.3.9 PHP/8.1.23 configured -- resuming normal operations
[Thu Oct 05 21:41:31.602423 2023] [core:notice] [pid 421] AH00094: Command line: '/usr/bin/httpd -D FOREGROUND'

That’s what makes the slow page-load times so puzzling. I’ve watched the server applications and their behavior in top during page loads and the only thing remarkable is that php-fpm seems to really hammer the CPU in several threads after the page request is made. It never saturates the CPU, but will consume 40-50% of CPU capacity for a brief second while Nextcloud is trying to serve the page.

Another area of high-demand by php-fpm is in contacts when simply scrolling the user-list down by a page or so (to the next 10-15 contacts) and then back. When that occurs all the initial bubbles disappear and are redrawn, apparently being regenerated which puts significant demand on the system for something that seems like a trivial act. I’m concerned the same type of time-consuming page-generation work is taking place for every page-load behind the scenes.

Is there anything else I can do to reduce page-load times on this test server? All caching is enabled, though with nothing paging to swap, there isn’t a whole lot of cache hits coming into play. Is there any way to find out if the Nextcloud framework is just spinning its wheels in places resulting in long page-load times? Is there some debug mode where we can add page-generation times to display as a nav-pane footer to keep track of the differences in application load times? Any ideas on how better to check on what is happening to see if things can’t be made a little snappier? Sorry for the flurry of questions, but I’ve been racking my brain on what to do or how to try and address this.

Don’t get me wrong, Nextcloud is well-written and working fine, it’s just rather painful to have to use the web interface (which is what I use a majority of the time). I’m also happy to get and post any additional information anyone thinks will be helpful. There is a solution somewhere.

All suggestions are welcomed.

2

I have never had a good experience running Nextcloud on this hardware when testing with 8gb ram.

The problem was immediately resolved by moving to an i7, which notably increased performance aross the board.

Testing was with one user on a local lan connected over rj45 + an ipad syncing over wifi.

3

A few thoughts off the top of my head:

  • HTTPS: The E7300 CPU lacks AES-NI support I believe. If you’re using HTTPS (TLS) it’s going to be slow:

This has been the traditional downside to the Raspberry Pi as well. One option is to offload HTTPS (TLS) elsewhere. Or use a VPN like Wireguard (instead of HTTPS - if an option) which has fast crypto for slower CPUs that lack AES-NI acceleration. Or accept it (as I did my first year or so) until you next upgrade your hardware.

  • Check your php-fpm logs. PHP-FPM defaults are typically too low even for tiny deployments.

  • Check your browser console under the Network tab. This will give you indications of precisely which transactions on the page are generating latency.

  • There are some profiler/performance monitoring options that can be turned on. They’re documented in the dev manual, but beware they also slow things down so use them for relative comparison not absolute benchmarking: Nextcloud developer documentation — Nextcloud latest Developer Manual latest documentation

4

That’s very good news. The Production box has an AMD Vishera with 8-cores and 32G of RAM, so hopefully that will resolve the issue. It was curious watching the threading on the dual-core. Many times more than 2 threads are created (up to 4 at times), and while the cores were never saturated, that does set up a scheduling issue.

Will move it to the production box in the next few days and see how things improve. I’ll definitely report the results back.

If it does improve markedly, it looks like the way the Nextcloud framework is setup is just a bit heavy for the old dual-core. What makes a good case for stronger recommendations for minimal hardware. The paradox is Nextcloud really isn’t putting that much information on the screen. For contacts, I can see getting the other 590 contacts prepped for display behind the scenes taking some additional horsepower, but for the Photos and Tasks where there are only 3 Photos and 2 test-tasks total, the long page-loads are just puzzling.

Anyway, will report back.

1 Like
5

Okay,

That is a great little tool. 10.24s for Contacts to load, and the primary culprits are core-common.js (1750 ms), core-contacts.js (1198 ms), notifications-vendors-node_modules_nextcloud_initial-state_dist_index_esm_js-node_modules_nextcloud_moment_-4efa53.js (2404 ms) and a whole slew of components in the 200-400 ms range.

I will check out the admin manual links. The reply by just seems to point to it just being the dual-core box not being up to running Nextcloud quickly. It that’s the case, things will improve with the move to the server.

The difference between page-loads for Nextcloud and say eGroupware was just something I was not expecting. I figured they would be within a 100% of each other. While I can generally pick through LAMP apps and understand where the work is being done, vue is a whole new framework to me – it will take a bit more time to digest (though I like what I see)

As mentioned, I’ll report back on how it behaves on the server hardware.

6

Another option is to test on somewhere like Linode on their free two month trial. Spin up a comparable box in terms of specs and see how it runs. I’ve had no trouble doing this and then deleting the instance. Here is a linux podcast trial code - linode.com/unplugged

1 Like
7

If you load the page again it should be much faster because it uses the cached version. Do you see such a behaviour that the long loading time are just for the first visit?

Some top versions also show the i/o-wait, that creates a huge load but in reality the processor are just waiting for data (then it can be caches and/or low memory, too slow hard drives, unoptimized code etc.).

1 Like
8

Could also be partly logging to the spinning rust. I’ve found switching the built in nextcloud logging from file to syslog, and lowering the loglevel, is quite helpful in removing some load that the DB is really sensitive too.

1 Like
9

That is a good thought. Also replacing the rust with SSD would be very, very helpful. (all in good time…, multi-terabyte SSDs are still a bit pricey compared to the near give-away giant spinning rust drives – and on a low demand server, rusting IronWolfs are still more than enough)

10

That core2duo with 4gb ram is tough. I had the same with 8gb and I couldn’t achieve acceptable performance for one user.

Curious on what you experiment with.

11

I have a production server 8-core Vishera and 32G which is the actual server. Before rolling any changes into that box, I have several old rust-pile boxes I do full test environment setups on to get apps ready to move to the production box. That is where my nextcloud testing on the tired old Dell Core2 Duo comes from.

I’ve never had slow performance of any web-app on the old Dell before, other than the expected relative decrease in speed, but all page generation and load times have always be within 1 second. This is for the same type MySQL/MariaDB/Postgres backend apps.

Nextcloud works perfectly on the test box, just way too slow to be usable on that box. I haven’t moved to the production box yet because this move will require a PHP version change and I want all apps working and configured before I roll the new PHP version and apps in.

So, just from an experience standpoint, I see 4-5 second page loads with Nextcloud on the tired old Dell. Everything works fine, but compared to other web-apps like eGroupware running on the same Dell – it’s just slow. To make progress I’ll have to just hurry the move.

I have one other app, not Nextcloud, that is the hold up. It’s gone to a docker install which is a GD nightmare as it doesn’t compose on Archlinux without significant modification. I’ll report back on the performance improvement on NC on the regular server when the move is done.

12

No need to store the data on SSD, just having the DB on one helps immensely. I have my Nextcloud data on a RUST ZFS pool and my DB data on an M.2