Forum performance: still a problem?

sorry it is: https://speed.hetzner.de/

Oooops missed that! Thanks again :slight_smile:

I think the most of the traceroute shows that there is more a problem with the access to Hetzner through the different backbones (ASN) than problems with the server.

To minimize the problem maybe the data of the forum can be minimized.
But i think that is not really possible because of the forum software https://www.discourse.org .
The result is not really bad:

Test of this thread website:
Website Speed Test | Pingdom Tools
help.nextcloud.com...Virginia USA - EC2 - WebPageTest Result

2 Likes

I am going to discuss your suggestion with our sysadmin. Hopefully we can do something about it. Thank you :blue_heart:

1 Like

Thanks. I tried downloading the 100MB.bin test file. Took 24.164201s, so about 4.14 megabytes/sec or 33.1 megabits/sec. That’s plenty fast… about 400x faster for me than downloading the discourse Javascript source.

I saw similar download speeds for the 1GB.bin and 10GB.bin test files, although I stopped those test downloads before they completed.

Oh. Then I take back my opinion from above. Traceroute seems to be not very meaningful after all. :wink: Or maybe there are people who have problems with traceroute and the forum.

My traceroute. Maybe fast because of connection over DE-CIX.

  6    15 ms    14 ms    14 ms  decix-gw.hetzner.com [80.81.192.164]
  7    17 ms    16 ms    17 ms  core9.fra.hetzner.com [213.239.252.18]
  8    37 ms    37 ms    56 ms  core31.hel1.hetzner.com [213.239.224.165]
  9    37 ms    36 ms    36 ms  ex9k2.dc2.hel1.hetzner.com [213.239.224.142]
 10    37 ms    36 ms    36 ms  s5.nextcloud.com [95.216.247.163]
 11    39 ms    37 ms    37 ms  help.nextcloud.com [95.217.53.146]

@meonkeys
Your traceroute seems to use:

63.231.10.75
AS209 CenturyLink Communications, LLC details - IPinfo.io
CenturyLink Communications, LLC

4.69.161.98
AS3356 Level 3 Parent, LLC details - IPinfo.io
Level 3 Parent, LLC

213.239.224.38
AS24940 Hetzner Online GmbH details - IPinfo.io
Hetzner Online GmbH

Longer pings seems to begin at Level 3 Parent, LLC.

1 Like

Ok, that’s a bit strange. It’s not impossible, that hetzner routes the traffic differently for their test-downloads. Because if it were the Nextcloud server, it should have an impact on everybody not just for those from overseas. However, most of the regular users probably have a cached version of the static files…

It’s not useless but it has limitations. There is a documentation how you can analyse network issues:
https://docs.hetzner.com/robot/dedicated-server/troubleshooting/network-diagnosis-and-report-to-hetzner/
However, here it doesn’t seem so bad that we actually have packet losses. So there is the routing information, so if we find that users via a certain peering provider have problems, that can be a starting point. Within a network, the providers manage their traffic, meaning they can prioritize certain traffic and/or filter traffic as well.

If you manage to bypass problems by using a VPN (or TOR), it shows that it is rather network related rather than the server.

2 Likes

Great idea. I tried the same curl timings while using a VPN and also while using my phone as a mobile Wi-Fi hotspot. In both cases the download speed was about 10x-15x faster (at least 100 kibibytes/sec). So yeah, that is definitely pointing at something strange while I’m trying to get to the forum from inside my home network. Very odd though… I haven’t noticed this kind of selective slowdown, even, for example, from other Discourse forums hosted overseas from me.

Using either the VPN or mobile hotspot WAN connections the Nextcloud forum loads for me (starting from cold cache) in let’s say a much more reasonable time… <1min. This was easily reproducible.

Here’s tracepath help.nextcloud.com while I’m using a VPN (hosted somewhere east of the pond):

 1?: [LOCALHOST]                      pmtu 1500
 1:  _gateway                                            203.953ms 
 1:  _gateway                                            204.712ms 
 2:  redacted                                            246.456ms 
 3:  redacted                                            204.738ms 
 4:  ae1-6.RT.RAD.HKI.FI.retn.net                        254.188ms asymm  9 
 5:  GW-Hetzner.retn.net                                 213.104ms asymm  9 
 6:  core31.hel1.hetzner.com                             306.527ms asymm  9 
 7:  ex9k2.dc2.hel.hetzner.com                           305.978ms asymm  9 
 8:  s5.nextcloud.com                                    306.854ms 
 9:  help.nextcloud.com                                  306.561ms reached
     Resume: pmtu 1500 hops 9 back 9

and here’s mtr help.nextcloud.com while I’m on a VPN:

                                          Packets               Pings
 Host                                   Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. _gateway                             0.0%    31  163.4 177.5 162.1 283.7  30.4
 2. redacted                             0.0%    31  171.6 180.4 163.5 235.9  17.5
 3. redacted                             0.0%    31  176.3 177.4 161.2 247.1  21.9
 4. ae2-5.RT.RAD.HKI.FI.retn.net         0.0%    31  226.9 213.6 192.8 301.5  30.3
 5. GW-Hetzner.retn.net                  0.0%    30  193.5 205.5 191.8 292.3  24.6
 6. core31.hel1.hetzner.com              0.0%    30  219.4 209.0 193.3 289.3  22.3
 7. ex9k2.dc2.hel1.hetzner.com           0.0%    30  208.9 217.3 193.3 284.1  30.5
 8. s5.nextcloud.com                     3.3%    30  194.4 209.5 192.6 369.5  35.6
 9. help.nextcloud.com                   3.3%    30  194.5 222.8 193.4 359.0  42.0

Side note: I see IPv6 addresses when I use a mobile hotspot. I am excluding those traces because this post is getting pretty long.

Looks like compression of Javascript is enabled: I see Content-encoding: br in the response headers. Brotli compression, apparently. And I also see smaller numbers for “Transferred” vs. “Size” in the Firefox developer tools “Network” tab.

re: caching - I am definitely seeing benefits from cached Javascript assets vs. when I specifically test with a cold browser cache or using curl. It seems to be once a day or so that these Javascript assets are re-downloaded by my browser though (even when I have a login/cookies from the day before). That seems wrong… the Discourse app probably doesn’t change that often so I shouldn’t make new/cache-busting Javascript requests. The cache-related HTTP response headers seem reasonable (fresh for a year, not modified since July, etc.).

I’m not sure if either of these would be expected to make much of a difference, but I’m just curious:

  1. Is a CDN being used?
  2. What about combining Javascript assets?

this forum is consistently the slowest site i use… loading can take 10-30 seconds… no other sites/forums take so long… it feels like the whole database/forum is self hosted on a raspeberry pi!

anyway an mtr/tracert doesnt show it that bad… but it took a good 2 minutes to load tonight…
really you need to change hosting provider whatever the issue! is almost like loading porn back in the 56k modem days trying to load this forum sometimes

*usually there is a noticeable 15-20sec delay that ive just learnt to live with)
image

and what’s your geographical location and are you using vpn?

Hey sorry for the delay, UK and no VPN used to visit here

1 Like