Nextcloud 27 - all Page-load times about 5 seconds, can it be made faster on a core2duo?

ℹ️ Support
,
1

This is a topic I have read quite a few posts about, but I haven’t found a cause or solution. I am about a month into working with Nextcloud, starting with 27.0.2 to the now current 27.1.2. I have Nextcloud set up on a test box as a normal web application and I have worked with the config to enable all recommended transactional locks and caching and I have eliminated all warnings. The Admin “Overview” checks report “All checks passed”. Nextcloud carries out all operations brilliantly (no warnings of errors in any of the server or Nextcloud log files).

The problem is page-load times are frustratingly slow. Clicking any of the apps in the menubar results in a consistent 5 second delay before the information is shown. That’s a pretty long wait, 1-1000, 2-1000, 3-1000, 4-1000, 5-1000… Photos is a little better at just over 4 seconds, but there are only 3 images total. Task is likewaise a little better at about 4 seconds, but there are only 2 test-tasks entered.

The box is an older Core2 Duo, so not snappy, but fine for test purposes and no reason page-load times should not be a reasonable 1 second or less. I run other similar web applications (with the exact same contacts and calendar entries also in a MariaDB database) and there is no problem with page-load times (all pages are generated and shown in sub-1-second times, including the same contacts and calendar entries). This is something unique to Nextcloud.

I have looked over a number of related links below and I have collected the information that was requested in the other posts to try and sort out where the issue is. The links are:

Related Links

The crux of the other posts is 4G of RAM on a lightly loaded x86_64 server with only a few users should be fine. I am the only user and I’m accessing the server over a gigabit LAN, so network I/O isn’t an issue. The server, while not exposed publicly, has a 300 MBs internet connection. The details of the install follow.

OS / Server Software

Nextcloud is installed as a normal webapp on the host web-server, not inside a container or any other virtualized install that would add overhead. The server software running on this small Dell test-box is:

  Nextcloud : 27.1.2
  Archlinux : 6.5.5-arch1-1
  Apache    : 2.4.57 (Unix) 
  OpenSSL   : 3.1.3 
  mod_fcgid : 2.3.9 
  PHP       : 8.1.23
  PHP-APCu  : 5.1.22
  Redis     : 7.2.1
  memcached : 1.6.21

Hardware

This is an old Dell computer I use as a test or prototype box before migrating apps to a production server. It’s a slow old Core2 Duo with 4G and a WD Carvair Blue internal SATA-III drive. While it is simply an old test box, it should be more than enough to get reasonable page-load times from Nextcloud. With similar web applications (eGroupware, etc…) all page loads are less than 1 second. This is using the same contacts (approximately 600 contacts) and calendar information (less than 5 current entries) in MariaDB databases. In fact the calendar and contacts in Nextcloud were imported from eGroupware.

Note, the page-load times did not change significantly between the time the Nextcloud database was empty and after the contacts and calendar entries were imported. Contacts was the most impacted which added another second or so for the 600 contacts.

The following is the information requested in the related links above that may impact page load-times:

CPU type        : Intel(R) Core(TM)2 Duo CPU     E7300  @ 2.66GHz
How much RAM    : 4G
Network speed   : LAN-Gigabit/GAT6, Internet-300 MBs
Hard disk       : Internal SATA-III, Western Digital Caviar Blue Serial ATA
How many users  : 1
Office suites   : None
Other services  : None
System Load     : (top output below)
  
  top - 02:21:12 up 23:57,  1 user,  load average: 0.08, 0.02, 0.01
  Tasks: 135 total,   1 running, 134 sleeping,   0 stopped,   0 zombie
  %Cpu0  :   0.0/0.0     0[                                                 ]
  %Cpu1  :   0.0/0.0     0[                                                 ]
  MiB Mem : 23.1/3844.1   [||||||||||||                                     ]
  MiB Swap:  0.0/1219.9   [                                                 ]

Per the MariaDB recommendation in Configuring Swappiness I have set vm.swappiness=1. There is no paging to swap taking place.

The File Access Restriction app is NOT installed – that was mentioned as a possible cause for slow page-loads in the Version 15-16 timeframe. I don’t even see that app available to install any more.

This test box is at idle waiting to serve Nextcloud requests. That is a primary reason the slow page-load times are of particular concern. There isn’t anything else running competing for CPU time. I could see the box being the culprit if the cores were saturated or the RAM exhausted, but that’s not the case. Nextcloud has complete access to all the CPU with well over half the RAM available for use.

Nextcloud Config

{
    "system": {
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "logfile": "\/var\/log\/nextcloud\/nextcloud.log",
        "default_locale": "en_US",
        "default_phone_region": "US",
        "knowledgebaseenabled": true,
        "apps_paths": [
            {
                "path": "\/usr\/share\/webapps\/nextcloud\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/lib\/nextcloud\/apps",
                "url": "\/wapps",
                "writable": true
            }
        ],
        "trusted_domains": [
            "localhost",
            "2pi.3111skyline.com",
            "192.168.6.111"
        ],
        "overwrite.cli.url": "https:\/\/2pi.3111skyline.com\/nextcloud",
        "htaccess.RewriteBase": "\/nextcloud",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "27.1.2.1",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_sendmailmode": "smtp",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "25",
        "maintenance": false,
        "app_install_overwrite": [
            "issuetemplate"
        ],
        "theme": "",
        "loglevel": 2,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "filelocking.enabled": true,
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "timeout": 0
        }
    }
}

Nextcloud Enabled Apps

The apps are the standard set of apps that come enable with the Archlinux install. The only app I have added is the “theming_customcss” – though currently there is no custom CSS configured. The apps and versions I have are:

Enabled:
  - activity: 2.19.0
  - calendar: 4.5.2
  - circles: 27.0.1
  - cloud_federation_api: 1.10.0
  - comments: 1.17.0
  - contacts: 5.4.2
  - dashboard: 7.7.0
  - dav: 1.27.0
  - federatedfilesharing: 1.17.0
  - federation: 1.17.0
  - files: 1.22.0
  - files_pdfviewer: 2.8.0
  - files_reminders: 1.0.0
  - files_rightclick: 1.6.0
  - files_sharing: 1.19.0
  - files_trashbin: 1.17.0
  - files_versions: 1.20.0
  - firstrunwizard: 2.16.0
  - groupfolders: 15.3.1
  - logreader: 2.12.0
  - lookup_server_connector: 1.15.0
  - nextcloud_announcements: 1.16.0
  - notes: 4.8.1
  - notifications: 2.15.0
  - oauth2: 1.15.1
  - password_policy: 1.17.0
  - photos: 2.3.0
  - privacy: 1.11.0
  - provisioning_api: 1.17.0
  - related_resources: 1.2.0
  - serverinfo: 1.17.0
  - settings: 1.9.0
  - sharebymail: 1.17.0
  - support: 1.10.0
  - survey_client: 1.15.0
  - systemtags: 1.17.0
  - tasks: 0.15.0
  - text: 3.8.0
  - theming: 2.2.0
  - theming_customcss: 1.14.0
  - twofactor_backupcodes: 1.16.0
  - updatenotification: 1.17.0
  - user_status: 1.7.0
  - viewer: 2.1.0
  - weather_status: 1.7.0
  - workflowengine: 2.9.0

Web Server Log / Errors

There are no web-server warnings or errors since the server reload. All Nextcloud menubar apps have been used since that time (great job!). The Nextcloud code is in really, really great shape.

[Thu Oct 05 21:41:31.602376 2023] [mpm_prefork:notice] [pid 421] AH00163: Apache/2.4.57 (Unix) OpenSSL/3.1.3 mod_fcgid/2.3.9 PHP/8.1.23 configured -- resuming normal operations
[Thu Oct 05 21:41:31.602423 2023] [core:notice] [pid 421] AH00094: Command line: '/usr/bin/httpd -D FOREGROUND'

That’s what makes the slow page-load times so puzzling. I’ve watched the server applications and their behavior in top during page loads and the only thing remarkable is that php-fpm seems to really hammer the CPU in several threads after the page request is made. It never saturates the CPU, but will consume 40-50% of CPU capacity for a brief second while Nextcloud is trying to serve the page.

Another area of high-demand by php-fpm is in contacts when simply scrolling the user-list down by a page or so (to the next 10-15 contacts) and then back. When that occurs all the initial bubbles disappear and are redrawn, apparently being regenerated which puts significant demand on the system for something that seems like a trivial act. I’m concerned the same type of time-consuming page-generation work is taking place for every page-load behind the scenes.

Is there anything else I can do to reduce page-load times on this test server? All caching is enabled, though with nothing paging to swap, there isn’t a whole lot of cache hits coming into play. Is there any way to find out if the Nextcloud framework is just spinning its wheels in places resulting in long page-load times? Is there some debug mode where we can add page-generation times to display as a nav-pane footer to keep track of the differences in application load times? Any ideas on how better to check on what is happening to see if things can’t be made a little snappier? Sorry for the flurry of questions, but I’ve been racking my brain on what to do or how to try and address this.

Don’t get me wrong, Nextcloud is well-written and working fine, it’s just rather painful to have to use the web interface (which is what I use a majority of the time). I’m also happy to get and post any additional information anyone thinks will be helpful. There is a solution somewhere.

All suggestions are welcomed.

2 Likes
2

I have never had a good experience running Nextcloud on this hardware when testing with 8gb ram.

The problem was immediately resolved by moving to an i7, which notably increased performance aross the board.

Testing was with one user on a local lan connected over rj45 + an ipad syncing over wifi.

1 Like
3

A few thoughts off the top of my head:

  • HTTPS: The E7300 CPU lacks AES-NI support I believe. If you’re using HTTPS (TLS) it’s going to be slow:

This has been the traditional downside to the Raspberry Pi as well. One option is to offload HTTPS (TLS) elsewhere. Or use a VPN like Wireguard (instead of HTTPS - if an option) which has fast crypto for slower CPUs that lack AES-NI acceleration. Or accept it (as I did my first year or so) until you next upgrade your hardware.

  • Check your php-fpm logs. PHP-FPM defaults are typically too low even for tiny deployments.

  • Check your browser console under the Network tab. This will give you indications of precisely which transactions on the page are generating latency.

  • There are some profiler/performance monitoring options that can be turned on. They’re documented in the dev manual, but beware they also slow things down so use them for relative comparison not absolute benchmarking: Nextcloud developer documentation — Nextcloud latest Developer Manual latest documentation

4

That’s very good news. The Production box has an AMD Vishera with 8-cores and 32G of RAM, so hopefully that will resolve the issue. It was curious watching the threading on the dual-core. Many times more than 2 threads are created (up to 4 at times), and while the cores were never saturated, that does set up a scheduling issue.

Will move it to the production box in the next few days and see how things improve. I’ll definitely report the results back.

If it does improve markedly, it looks like the way the Nextcloud framework is setup is just a bit heavy for the old dual-core. What makes a good case for stronger recommendations for minimal hardware. The paradox is Nextcloud really isn’t putting that much information on the screen. For contacts, I can see getting the other 590 contacts prepped for display behind the scenes taking some additional horsepower, but for the Photos and Tasks where there are only 3 Photos and 2 test-tasks total, the long page-loads are just puzzling.

Anyway, will report back.

1 Like
5

Okay,

That is a great little tool. 10.24s for Contacts to load, and the primary culprits are core-common.js (1750 ms), core-contacts.js (1198 ms), notifications-vendors-node_modules_nextcloud_initial-state_dist_index_esm_js-node_modules_nextcloud_moment_-4efa53.js (2404 ms) and a whole slew of components in the 200-400 ms range.

I will check out the admin manual links. The reply by just seems to point to it just being the dual-core box not being up to running Nextcloud quickly. It that’s the case, things will improve with the move to the server.

The difference between page-loads for Nextcloud and say eGroupware was just something I was not expecting. I figured they would be within a 100% of each other. While I can generally pick through LAMP apps and understand where the work is being done, vue is a whole new framework to me – it will take a bit more time to digest (though I like what I see)

As mentioned, I’ll report back on how it behaves on the server hardware.

6

Another option is to test on somewhere like Linode on their free two month trial. Spin up a comparable box in terms of specs and see how it runs. I’ve had no trouble doing this and then deleting the instance. Here is a linux podcast trial code - linode.com/unplugged

2 Likes
7

If you load the page again it should be much faster because it uses the cached version. Do you see such a behaviour that the long loading time are just for the first visit?

Some top versions also show the i/o-wait, that creates a huge load but in reality the processor are just waiting for data (then it can be caches and/or low memory, too slow hard drives, unoptimized code etc.).

1 Like
8

Could also be partly logging to the spinning rust. I’ve found switching the built in nextcloud logging from file to syslog, and lowering the loglevel, is quite helpful in removing some load that the DB is really sensitive too.

1 Like
9

That is a good thought. Also replacing the rust with SSD would be very, very helpful. (all in good time…, multi-terabyte SSDs are still a bit pricey compared to the near give-away giant spinning rust drives – and on a low demand server, rusting IronWolfs are still more than enough)

10

That core2duo with 4gb ram is tough. I had the same with 8gb and I couldn’t achieve acceptable performance for one user.

Curious on what you experiment with.

11

I have a production server 8-core Vishera and 32G which is the actual server. Before rolling any changes into that box, I have several old rust-pile boxes I do full test environment setups on to get apps ready to move to the production box. That is where my nextcloud testing on the tired old Dell Core2 Duo comes from.

I’ve never had slow performance of any web-app on the old Dell before, other than the expected relative decrease in speed, but all page generation and load times have always be within 1 second. This is for the same type MySQL/MariaDB/Postgres backend apps.

Nextcloud works perfectly on the test box, just way too slow to be usable on that box. I haven’t moved to the production box yet because this move will require a PHP version change and I want all apps working and configured before I roll the new PHP version and apps in.

So, just from an experience standpoint, I see 4-5 second page loads with Nextcloud on the tired old Dell. Everything works fine, but compared to other web-apps like eGroupware running on the same Dell – it’s just slow. To make progress I’ll have to just hurry the move.

I have one other app, not Nextcloud, that is the hold up. It’s gone to a docker install which is a GD nightmare as it doesn’t compose on Archlinux without significant modification. I’ll report back on the performance improvement on NC on the regular server when the move is done.

12

No need to store the data on SSD, just having the DB on one helps immensely. I have my Nextcloud data on a RUST ZFS pool and my DB data on an M.2

13

Updated test box to SSD and page load times may have improved by about 10-15% even though drive throughput improved by over 10X. There is definite improvement, just not that much with Nextcloud. And for fun, changed the 2.7GHz Core2-duo to a 3.3GHz Core2-duo (E8600). Box zips along nicely, all databases and web apps (including Nextcloud) are on the same SSD, but the Nextcloud UI response time is still, well, slow. This is a hard puzzle to figure out. eGroupware running on the same box has sub 1-sec page loads, so there is a definite design difference. Both use PHP and both use TS and both use MariaDB. Will keep testing as time permits and report back further findings.

14

Try using a faster system. Sorry for the broken record response, but use something notably faster than a core2duo. The i7 performs very well in my experience. Happy New Years!

1 Like
15

I will. I’m under no false-impressions that the current test-box is “enough”, but in the same light it poses a great opportunity to determine where the bottlenecks are that are simply masked-over by throwing more cores at it.

I’ll move to an 8-core after I get the update to 28 done, but until then I want to see if I can determine why changing from say “calendar” to “contacts” or to “notes” is like starting Nextcloud over instead of just loading a new view.

That seems to be the whole crux of the issue. While other LAMP apps have a framework and apps are simply different views of the same data within the main app API, Nextcloud seems to be more a loose collection of individual applications that require a full-load when changing between them. That’s about the only thing that would explain the difference.

1 Like
16

Absolutely. This stack requires the best possible hardware. It is designed for enterprise usage, so all we can ask for is a handshake deal. Using a 2 core system is exactly what it sounds like. Sorry if you were hoping for otherwise.

I highly, highly recommend syncthing if your goal is to make your current system scream. I was not impressed with the performance on my core2duo back in 2009 with even 1 user. Can certainly imagine how frustrating it must be in 2023.

edit: the #1 improvement you can make is in swapping your hard drive with an SSD.
2nd improvement is to upgrade your networking connection, if possible, to 1gb wired.
3rd improvment is to upgrade your RAM.
Besides SSD, your best bet is to buy a better machine. Used i5 and i7 and even Ryzen machines are readily available. Last I looked, a Dell Wyze was less than $30 on ebay.

1 Like
17

If you don’t use all the RAM, that is perhaps already a point where you could improve your configuration. If the database can use the cache, the disk limitations are less important. Did you try to optimize anything, or are all the services running with their default config?

18

Good news is Nextcloud 28 does seem quite a bit more responsive than 27, great job. I do need to finish some upgrade tweaks, but this is very encouraging.

19

Here’s my two-cents, for whatever they’re worth:

A number of good suggestions have been made. Faster CPU with more modern features, SSD instead of spinning rust, PHP tuning, and so on. I could suggest many of the same things.

Honestly, though, the best piece of advice I can offer here is…

  • Manage your expectations.

I think we all wish this could be a reality. Many of us (including yours truly) have spent literal days of our lives trying to eek a bit more snappiness out of the web interface to no avail. I get about 2-3s app load times on a local connection with assets cached client-side.

Basically, the sky is blue, grass is green, water is wet, and Nextcloud is slow. :disappointed:

To be clear, I don’t mean for this to come across offensively. I could be completely wrong, and if someone can produce a configuration that gives consistent, sub-1-second load times with a full suite of apps, I’d love to hear about it.

1 Like
20

Just to put things into perspective:

@drankinatty uses a E7300 CPU, which is a 2 core / 2 threads mid-range 65 watt desktop CPU from 2008!

For comparison, a current 6W! low-end CPU (with only 4 single threaded efficient cores) produces about 4x the raw compute speed (2x single core speed). Intel Core2 Duo E7300 @ 2.66GHz vs Intel N100 [cpubenchmark.net] by PassMark Software. In addition to that, modern CPUs come with features like HW offloading for encryption etc… and the platform offers a much faster memory interface, the boards have M.2 slots for fast NVMe SSDs etc. etc.

And again, we are talking about a low-end Celeron/Atom class CPU. Now imagine using a mid-range office PC with a current 65-watt CPU… That would probably offer better performance than an average 2008 enterprise server :wink:

…well, that might be a bit ambitious, but 1-2 seconds is definitely possible on a newer system.

The following is running in a VM with 2 cores and 4GB RAM on an i7 7700, which is also a 65Watt mid-range desktop CPU, but from 2016…

nc

3 Likes