Help.nextcloud.com is very slow

Whenever I try to got to these forums, the page loading time is extremely slow. Sometimes I am waiting up to a good 2-5 minutes. Sometimes the webpage just won’t load all together and I just get a blank white page. I have tested on Mac and Windows and on Firefox, Chrome, and Safari and get the same results. This can be quite annoying at times because I often come to this site to regularly view forum posts and get help. All other sites I visit load in a second or less. My internet download speed is roughly 50-60mb/s and I am located in Australia.

Any ideas on why this is?

1 Like

It can be that the server is saturated, or it is a bad network connection between the providers. Nextcloud uses Hetzner, you can use a looking glass service to see which way the packages pass and where the bottleneck is: Point in Germany, Falkenstein, data center Hetzner from SpaceCore

Usually they have a pretty open peering policy and present at multiple points in Europe, for overseas they rely on larger carriers (see the transit providers https://www.hetzner.com/unternehmen/rechenzentrum/). Sometimes, end-user providers have bad peering connections to these larger carriers.

We can perhaps get an idea, if you can post the mtr output to your home connection (you can anonymize your own ip, the interesting part is when it reaches your ISP’s network).

I’m also in Australia and experiencing this issue. It’s a consistent problem and not something that comes and goes, so I don’t think it’s a server load issue. I find it takes minutes to load any pages on the forum as Artichoke1 said.

I can provide mtr output, but could you explain what you’re looking for? My home network to help.nextcloud.com or from Looking House to my home network?

1 Like

even me being not literally on the other side of the world do experience certain load-problems often causing a 500 error to me. I already filed an issue on GH for it.

you’re welcome to leave your :+1: there.

If you do the traceroute, there is a first hop at hetzner, like:
core0.fra.hetzner.com (213.239.252.37)
And then just the last hope before hetzner, or if you manage to do the looking glass to your ip, the first hop outside hetzner network.

You can also test and download here: http://speed.hetzner.de/

That is more for the server itself and not the network…

I know. That’s why I opened this issue there. And we don’t really know where the slowness comes from. Could be as well from gettingt timeouts due to serverproblems.

Download speed for the 100mb file peaked at around 6mb/s and was at least a couple of mb/s for the whole download. Didn’t have any issues with slow starting or anything, so all seems normal.

Here’s the result of mtr to my local network using the looking.house link you posted:

Start: 2022-03-08T09:25:25+0300
HOST: 22085.vm.hosted-by-spacecore.pro              Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- static.106.123.201.138.clients.your-server.de  0.0%     1    0.3   0.3   0.3   0.3   0.0
  2.|-- static.209.172.9.5.clients.your-server.de      0.0%     1    1.8   1.8   1.8   1.8   0.0
  3.|-- core23.fsn1.hetzner.com                        0.0%     1   21.1  21.1  21.1  21.1   0.0
  4.|-- core4.fra.hetzner.com                          0.0%     1    5.2   5.2   5.2   5.2   0.0
  5.|-- ???                                           100.0     1    0.0   0.0   0.0   0.0   0.0
  6.|-- ???                                           100.0     1    0.0   0.0   0.0   0.0   0.0
  7.|-- ???                                           100.0     1    0.0   0.0   0.0   0.0   0.0
  8.|-- if-ae-49-2.tcore1.pvu-paris.as6453.net         0.0%     1  163.1 163.1 163.1 163.1   0.0
  9.|-- ???                                           100.0     1    0.0   0.0   0.0   0.0   0.0
 10.|-- ???                                           100.0     1    0.0   0.0   0.0   0.0   0.0
 11.|-- ???                                           100.0     1    0.0   0.0   0.0   0.0   0.0
 12.|-- ???                                           100.0     1    0.0   0.0   0.0   0.0   0.0
 13.|-- if-ae-14-2.tcore3.nto-newyork.as6453.net       0.0%     1  173.9 173.9 173.9 173.9   0.0
 14.|-- if-ae-22-2.tcore1.nto-newyork.as6453.net       0.0%     1  170.5 170.5 170.5 170.5   0.0
 15.|-- if-ae-36-2.tcore1.sqn-sanjose.as6453.net       0.0%     1  168.3 168.3 168.3 168.3   0.0
 16.|-- ???                                           100.0     1    0.0   0.0   0.0   0.0   0.0
 17.|-- if-ae-0-2.tcore1.sv1-santaclara.as6453.net     0.0%     1  160.3 160.3 160.3 160.3   0.0
 18.|-- ???                                           100.0     1    0.0   0.0   0.0   0.0   0.0
 19.|-- xxxxxxxxxxxxxxxxxxxxxxx                       0.0%     1  336.3 336.3 336.3 336.3   0.0

AS6453 seems to be TATA with a quite good connection: https://www.hetzner.com/unternehmen/rechenzentrum

And you can download from the speed server quite ok. I’ve just seen that the loading of some resources either doesn’t work or timeout (especially avatars, either timeout or some take very long like 9s). Not sure if that adds up in a bad way with the increased delay that you obtain much more timeouts. Problem of these nice interactive pages lazy-loading (depending on how much you scroll) works very well for short delays but if the delays are increasing and you have problems, it can be very painful to use…

I’m not getting timeouts, everything does eventually load. The problem I’m having is it takes 1.7 minutes from initially starting to load the page to see anything on screen. I had to wait that long to open this up to write the reply (I timed it). Once everything’s loaded it’s more usable, if a bit slow. For example I don’t get an exceptionally long delay posting replies and I can navigate around the forums without too much delay.

I looked in the browser’s network inspector to see if there was anything specific causing a problem, but it’s basically every resource taking a very long time to load. For example, this resource:

https://help.nextcloud.com/theme-javascripts/fde9bc7c258930129c4085bb1fbd4886a90e5d33.js?__ws=help.nextcloud.com

Has the following timings:

Screen Shot 2022-03-09 at 8.19.22 am

You can see it’s an exceptionally long wait and download, especially since the actual file is tiny.

I tried loading that same asset with curl and the delay was still slow but much, much shorter. Here’s the command (curl-format.txt borrowed from SO):

curl -w "@curl-format.txt" -o /dev/null -s https://help.nextcloud.com/theme-javascripts/fde9bc7c258930129c4085bb1fbd4886a90e5d33.js\?__ws\=help.nextcloud.com

And timings I got

     time_namelookup:  0.005279s
        time_connect:  0.359582s
     time_appconnect:  2.010141s
    time_pretransfer:  2.010278s
       time_redirect:  0.000000s
  time_starttransfer:  2.368495s
                     ----------
          time_total:  2.371560s

This seems to be consistent, in that in curl it always takes around 2 seconds and in my browser (I’ve tried several different machines and browsers) it always takes just under 2 minutes. Maybe there’s some kind of cumulative effect happening?

By comparison, the link you posted takes around 5 or so seconds to fully load, which is a little slow, but it’s still usable.

These curl commands, I didn’t know, really cool :sunglasses:

For the javascript, for me it is <0.5s from Europe.

However, the strange letter avatars:

curl -w "@curl-format.txt" -o /dev/null -s https://help.nextcloud.com/letter_avatar_proxy/v4/letter/e/d78d45/90.png
     time_namelookup:  0.004868s
        time_connect:  0.037185s
     time_appconnect:  0.143471s
    time_pretransfer:  0.143888s
       time_redirect:  0.000000s
  time_starttransfer:  60.242762s
                     ----------
          time_total:  60.243041s
1 Like

See also: Help.nextcloud.com slow page load times

I’m also seeing consistently slow (first view/cold cache) page load times for help.nextcloud.com. Over a minute to be able to see and do anything. Similar results in both Firefox and Chromium.

I’m really confused why I can’t reproduce the problem using www.webpagetest.org. Timings from that site are consistently and reasonably fast enough, say, 5 seconds or less. So I’m trying to isolate what might be different between www.webpagetest.org and my home network. One difference is DNS.

I thought perhaps when stats.nextcloud.com is blocked it slows down the help.nextcloud.com user experience. At home I am usually blocking stats.nextcloud.com using a Pi-hole. This uses a DNS trick so stats.nextcloud.com resolves to 0.0.0.0. With the DNS block in place, it takes minutes for the first/cold page load of https://help.nextcloud.com. But when I allow proper DNS resolution for the stats.nextcloud.com domain it seems to take just as long.

It works quickly on my mobile phone web browser when I am not on my home network (and not behind my Pi-hole), though.

I think we really need help working through a browser-side performance profile showing each URL, which ones take a long time, and especially looking for anything that blocks. Anyone have experience doing this? I can definitely see some scripts taking a long time when I’m using Firefox Developer Tools but I’m having trouble sharing the result.

I also tried the curl timing examples others mentioned (very cool, thank you!) but the results are intermittent. One time an avatar placeholder image will take 2 minutes to fetch, then another time it’ll take 2 seconds.

Hello,

I am from India. Here too the initial Help.NextCloud website webpage loading time is really really slow. Takes ages for the page to load but once loaded, it works relatively fine.

Here are my ping / trace route results.

C:\Windows\system32>ping help.nextcloud.com

Pinging help.nextcloud.com [95.217.53.146] with 32 bytes of data:
Reply from 95.217.53.146: bytes=32 time=197ms TTL=48
Reply from 95.217.53.146: bytes=32 time=198ms TTL=48
Reply from 95.217.53.146: bytes=32 time=198ms TTL=48
Reply from 95.217.53.146: bytes=32 time=197ms TTL=48

Ping statistics for 95.217.53.146:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 197ms, Maximum = 198ms, Average = 197ms

C:\Windows\system32>tracert help.nextcloud.com

Tracing route to help.nextcloud.com [95.217.53.146]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.168.1.1
  2     6 ms    16 ms    14 ms  xx.xxx.xxx.xxx
  3     *        *        *     Request timed out.
  4     3 ms     1 ms     1 ms  10.240.246.1
  5     3 ms     2 ms     3 ms  10.200.22.1
  6    34 ms    32 ms    32 ms  125.23.238.225
  7     *        *        *     Request timed out.
  8   188 ms   186 ms   188 ms  decix2-gw.hetzner.com [80.81.193.164]
  9   177 ms   181 ms   177 ms  core9.fra.hetzner.com [213.239.224.178]
 10   198 ms   198 ms   209 ms  core32.hel1.hetzner.com [213.239.224.169] 
 11   206 ms   207 ms   226 ms  ex9k2.dc2.hel.hetzner.com [213.239.224.146]
 12   202 ms   200 ms   201 ms  s5.nextcloud.com [95.216.247.163]
 13   197 ms   197 ms   197 ms  help.nextcloud.com [95.217.53.146]

Trace complete.

C:\Windows\system32>

:roll_eyes:

Thanks.

There are still problems for everybody, so it doesn’t seem to be a peering problem (which can be the case in specific networks). Regarding the network, I found this: https://docs.hetzner.com/robot/dedicated-server/troubleshooting/network-diagnosis-and-report-to-hetzner/

And for the last hop outside your local network, it looks good from a host at hetzer (not the same datacenter, but probably not that different):

HOST: host.example.com            Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- static.ip1.clie            0.0%  1000    0.2   0.1   0.1   1.0   0.1
  2.|-- static.ip2.clients.        0.0%  1000    0.6   0.6   0.4  13.8   0.9
  3.|-- core24.fsn1.hetzner.com   10.9%  1000    7.0   5.7   0.4  93.8   9.1
  4.|-- core0.fra.hetzner.com      0.0%  1000    9.7   8.0   5.0  54.3   6.1
  5.|-- dx1.in.airtel.com          0.0%  1000   20.2  23.9  20.1 181.9  12.2
  6.|-- 125.23.238.225             0.0%  1000  187.6 186.9 181.6 275.5   5.1

I was still seeing a few 500 errors this week, it was perhaps a bit worse two weeks ago.

The curl-test in my post above is still exactly the same. It’s just a bit above 60s, so I’m wondering if there is a timeout that is triggered after 60s.

I tried to ping the Nextcloud team on github (experiencing a lot of 500 errors with help.nextcloud.com... · Issue #1613 · nextcloud/nextcloud.com · GitHub), if you have problems as well, give the original post a thumbs up to increase the visibility.

For the 60s: it seems to be a default proxy_pass setting in nginx:
http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout

And in older post, there were already problems regarding the avatars:
https://meta.discourse.org/t/users-profile-image-not-showing-letter-avatars/109224
https://review.discourse.org/t/refactor-proxy-letter-avatars-in-rails-instead-of-nginx/5814

See also: This forum is EXTREMELY slow

2 Likes

Check out mtr (my traceroute). That tool shows me 85% packet loss from 4.68.38.173 (the 4th hop), and 0% packet loss from all others. I like to imagine that router is somewhere deep in the Atlantic ocean. :shark: