Forum performance: still a problem?

sorry it is: https://speed.hetzner.de/

Oooops missed that! Thanks again :slight_smile:

I think the most of the traceroute shows that there is more a problem with the access to Hetzner through the different backbones (ASN) than problems with the server.

To minimize the problem maybe the data of the forum can be minimized.
But i think that is not really possible because of the forum software https://www.discourse.org .
The result is not really bad:

Test of this thread website:
Website Speed Test | Pingdom Tools
help.nextcloud.com...Virginia USA - EC2 - WebPageTest Result

2 Likes

I am going to discuss your suggestion with our sysadmin. Hopefully we can do something about it. Thank you :blue_heart:

1 Like

Thanks. I tried downloading the 100MB.bin test file. Took 24.164201s, so about 4.14 megabytes/sec or 33.1 megabits/sec. Thatā€™s plenty fastā€¦ about 400x faster for me than downloading the discourse Javascript source.

I saw similar download speeds for the 1GB.bin and 10GB.bin test files, although I stopped those test downloads before they completed.

Oh. Then I take back my opinion from above. Traceroute seems to be not very meaningful after all. :wink: Or maybe there are people who have problems with traceroute and the forum.

My traceroute. Maybe fast because of connection over DE-CIX.

  6    15 ms    14 ms    14 ms  decix-gw.hetzner.com [80.81.192.164]
  7    17 ms    16 ms    17 ms  core9.fra.hetzner.com [213.239.252.18]
  8    37 ms    37 ms    56 ms  core31.hel1.hetzner.com [213.239.224.165]
  9    37 ms    36 ms    36 ms  ex9k2.dc2.hel1.hetzner.com [213.239.224.142]
 10    37 ms    36 ms    36 ms  s5.nextcloud.com [95.216.247.163]
 11    39 ms    37 ms    37 ms  help.nextcloud.com [95.217.53.146]

@meonkeys
Your traceroute seems to use:

63.231.10.75
https://ipinfo.io/AS209
CenturyLink Communications, LLC

4.69.161.98
AS3356 Level 3 Parent, LLC details - IPinfo.io
Level 3 Parent, LLC

213.239.224.38
AS24940 Hetzner Online GmbH details - IPinfo.io
Hetzner Online GmbH

Longer pings seems to begin at Level 3 Parent, LLC.

1 Like

Ok, thatā€™s a bit strange. Itā€™s not impossible, that hetzner routes the traffic differently for their test-downloads. Because if it were the Nextcloud server, it should have an impact on everybody not just for those from overseas. However, most of the regular users probably have a cached version of the static filesā€¦

Itā€™s not useless but it has limitations. There is a documentation how you can analyse network issues:
https://docs.hetzner.com/robot/dedicated-server/troubleshooting/network-diagnosis-and-report-to-hetzner/
However, here it doesnā€™t seem so bad that we actually have packet losses. So there is the routing information, so if we find that users via a certain peering provider have problems, that can be a starting point. Within a network, the providers manage their traffic, meaning they can prioritize certain traffic and/or filter traffic as well.

If you manage to bypass problems by using a VPN (or TOR), it shows that it is rather network related rather than the server.

2 Likes

Great idea. I tried the same curl timings while using a VPN and also while using my phone as a mobile Wi-Fi hotspot. In both cases the download speed was about 10x-15x faster (at least 100 kibibytes/sec). So yeah, that is definitely pointing at something strange while Iā€™m trying to get to the forum from inside my home network. Very odd thoughā€¦ I havenā€™t noticed this kind of selective slowdown, even, for example, from other Discourse forums hosted overseas from me.

Using either the VPN or mobile hotspot WAN connections the Nextcloud forum loads for me (starting from cold cache) in letā€™s say a much more reasonable timeā€¦ <1min. This was easily reproducible.

Hereā€™s tracepath help.nextcloud.com while Iā€™m using a VPN (hosted somewhere east of the pond):

 1?: [LOCALHOST]                      pmtu 1500
 1:  _gateway                                            203.953ms 
 1:  _gateway                                            204.712ms 
 2:  redacted                                            246.456ms 
 3:  redacted                                            204.738ms 
 4:  ae1-6.RT.RAD.HKI.FI.retn.net                        254.188ms asymm  9 
 5:  GW-Hetzner.retn.net                                 213.104ms asymm  9 
 6:  core31.hel1.hetzner.com                             306.527ms asymm  9 
 7:  ex9k2.dc2.hel.hetzner.com                           305.978ms asymm  9 
 8:  s5.nextcloud.com                                    306.854ms 
 9:  help.nextcloud.com                                  306.561ms reached
     Resume: pmtu 1500 hops 9 back 9

and hereā€™s mtr help.nextcloud.com while Iā€™m on a VPN:

                                          Packets               Pings
 Host                                   Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. _gateway                             0.0%    31  163.4 177.5 162.1 283.7  30.4
 2. redacted                             0.0%    31  171.6 180.4 163.5 235.9  17.5
 3. redacted                             0.0%    31  176.3 177.4 161.2 247.1  21.9
 4. ae2-5.RT.RAD.HKI.FI.retn.net         0.0%    31  226.9 213.6 192.8 301.5  30.3
 5. GW-Hetzner.retn.net                  0.0%    30  193.5 205.5 191.8 292.3  24.6
 6. core31.hel1.hetzner.com              0.0%    30  219.4 209.0 193.3 289.3  22.3
 7. ex9k2.dc2.hel1.hetzner.com           0.0%    30  208.9 217.3 193.3 284.1  30.5
 8. s5.nextcloud.com                     3.3%    30  194.4 209.5 192.6 369.5  35.6
 9. help.nextcloud.com                   3.3%    30  194.5 222.8 193.4 359.0  42.0

Side note: I see IPv6 addresses when I use a mobile hotspot. I am excluding those traces because this post is getting pretty long.

Looks like compression of Javascript is enabled: I see Content-encoding: br in the response headers. Brotli compression, apparently. And I also see smaller numbers for ā€œTransferredā€ vs. ā€œSizeā€ in the Firefox developer tools ā€œNetworkā€ tab.

re: caching - I am definitely seeing benefits from cached Javascript assets vs. when I specifically test with a cold browser cache or using curl. It seems to be once a day or so that these Javascript assets are re-downloaded by my browser though (even when I have a login/cookies from the day before). That seems wrongā€¦ the Discourse app probably doesnā€™t change that often so I shouldnā€™t make new/cache-busting Javascript requests. The cache-related HTTP response headers seem reasonable (fresh for a year, not modified since July, etc.).

Iā€™m not sure if either of these would be expected to make much of a difference, but Iā€™m just curious:

  1. Is a CDN being used?
  2. What about combining Javascript assets?

this forum is consistently the slowest site i useā€¦ loading can take 10-30 secondsā€¦ no other sites/forums take so longā€¦ it feels like the whole database/forum is self hosted on a raspeberry pi!

anyway an mtr/tracert doesnt show it that badā€¦ but it took a good 2 minutes to load tonightā€¦
really you need to change hosting provider whatever the issue! is almost like loading porn back in the 56k modem days trying to load this forum sometimes

*usually there is a noticeable 15-20sec delay that ive just learnt to live with)
image

and whatā€™s your geographical location and are you using vpn?

Hey sorry for the delay, UK and no VPN used to visit here

1 Like

Heads up with these curl tests: be sure to include the --compressed flag. Donā€™t do what I did and leave it out! It makes a huge difference in timings. The main Discourse Javascript resource can be 3MB uncompressed and 540KB compressed. For me thatā€™s 5min vs 1min to download at 10KB/s. And yeah, 10KB/s is still mysteriously slow, but at least compressing traffic will give us more accurate single-file download speed tests.

I just texted ~100 nerds in couple chats asking them to try this curl test from different networks:

curl --compressed -o /dev/null https://help.nextcloud.com/assets/discourse-12a807403b8f23c14a92099c4e078ddbbac432d1409edb731c4150ac17a7d55d.js

Most of them got fast downloads, say, less than 2sec to get the Discourse Javascript file. A couple of them, (also on Lumen/Centurylink fiber!) reported very slow download speeds: ~12KB/s, for a total of around a minute to download that ~.5MB file. But another person (also on lumen/centurylink fiber) reported normal/fast download speed (sub-second), so I donā€™t have a 100% repro.

Others also reported that one hop in the traceroute was also dropping 90% or more packets. The IP address is 4.68.38.173 and apparently it is owned by Level 3 (which is owned by Lumen/CenturyLink). No idea if that is related.

One of the chats is called ā€œSeattle Matrixā€, in case anyone is curious and wants to join.

I have had a problem for quite some time.
If a havenā€™t connected to the forum URL for a week or so when I try I have the 5 color dots for at least 45 seconds.

Then once that has happened its ok opening more tabs.
Thanks

I think there is a problem with first load of the website (no cache, no cookie). In my case i must load e.g. 28 Javascript files with 25 MB of data. I think that is not only a problem of Discourse because other Discourse e.g. OpenStreetMap Community are much smaller with better performance. Maybe someone like to comare both Discourse installations (files and filesize). Oh yes. And everything that is not Discourse should be thrown out.

1 Like

I just ran this on a small post.

https://www.webpagetest.org/result/230223_AiDc2F_5KY/

A NC server login page is about 14 mb?

At some free advise. I have used this site on and off for 15 years. https://www.webpagetest.org/result/230223_AiDc2F_5KY/1/experiments/#Usable

1 Like

Im currently crawling this forum with Yacy and its slow. I have set yacy not to load images plus a bunch of other file extensions. eg ISO,gz.etc.

The text is in my search engine I have just seen its raw html code.

I have added a Pic because the forum translates it,

.jpeg 2x' data-dominant-color='D3EBF8'></a></div><p></p> <p><a href='https://help.nextcloud.com/t/sync-tool-osx-and-performance-owncube/152029/3'>Read full topic</a></p>
#### [Sync tool OSX and performance [owncube]](https://help.nextcloud.com/t/sync-tool-osx-and-performance-owncube/152029/3)

[![bookmark|11x11](http://localhost:8090/env/grafics/empty.gif "bookmark")](http://localhost:8090/yacysearch.html?query=performance&maximumRecords=100&resource=local&verify=ifexist&prefermaskfilter=&cat=href&constraint=&contentdom=text&strictContentDom=false&meanCount=5&former=performance&auth&startRecord=0&bookmarkurl=b%7CaHR0cHM6Ly9oZWxwLm5leHRjbG91ZC5jb20vdC9zeW5jLXRvb2wtb3N4LWFuZC1wZXJmb3JtYW5jZS1vd25jdWJlLzE1MjAyOS8z&bookmarkref=kiWSNexFffU5)[![recommend|11x11](http://localhost:8090/env/grafics/empty.gif "recommend")](http://localhost:8090/yacysearch.html?query=performance&maximumRecords=100&resource=local&verify=ifexist&prefermaskfilter=&cat=href&constraint=&contentdom=text&strictContentDom=false&meanCount=5&former=performance&auth&startRecord=0&recommendref=kiWSNexFffU5)[![delete|11x11](http://localhost:8090/env/grafics/empty.gif "delete")](http://localhost:8090/yacysearch.html?query=performance&maximumRecords=100&resource=local&verify=ifexist&prefermaskfilter=&cat=href&constraint=&contentdom=text&strictContentDom=false&meanCount=5&former=performance&auth&startRecord=0&deleteref=kiWSNexFffU5)[![blacklist host|11x11](http://localhost:8090/env/grafics/empty.gif "blacklist host")](http://localhost:8090/yacysearch.html?query=performance&maximumRecords=100&resource=local&verify=ifexist&prefermaskfilter=&cat=href&constraint=&contentdom=text&strictContentDom=false&meanCount=5&former=performance&auth&startRecord=0&blacklisturl=b%7CaHR0cHM6Ly9oZWxwLm5leHRjbG91ZC5jb20vdC9zeW5jLXRvb2wtb3N4LWFuZC1wZXJmb3JtYW5jZS1vd25jdWJlLzE1MjAyOS8z)

https://help.nextcloud.com/t/sync-tool-osx-and-performance-owncube/152029/3

.jpeg 2x' data-dominant-color='D3EBF8'></a></div><p></p> <p><a href='https://help.nextcloud.com/t/sync-tool-osx-and-**performance**-owncube/152029/3'>Read full topic</a></p>

Mon, 19 Dec 2022 | [Citations](http://localhost:8090/api/citation.html?hash=kiWSNexFffU5&filter=true) | [![Browse index|15x8](http://localhost:8090/env/grafics/minitree.png "Browse index")](http://localhost:8090/IndexBrowser_p.html?path=https://help.nextcloud.com/t/sync-tool-osx-and-performance-owncube/152029/3)

I am not sure whatā€™s going on.
Todays posts are fine. The other discourse forums have the same problem.
I will post on the yacy forum or raise at github issue.

Images load time on the forum at ~10 kb/s, 5.1 seconds for png. optimized. 55kb

100%[==================================>] 346.59K 10.0KB/s in 33s
The result of the pic above.

Sorry only 5 MB for https://help.nextcloud.com (mostly Javascript).
And here 40 errors in the website.
One error 404: Wordpress Fonts from Nextcloud in Discourse? I do not see the need.

Looks like your site has iframe sort of errors. I am not really sure too sure.

But if you open the page in chrome or firefox and hit ctrl + shift + j you will see the errors on that page.

That was how I found a way to set my local web server as a back end/front end???
I had similar problems with my piholes to put in a page in an iframe on another server.

It could be that we are talking past each other. Itā€™s not about a web server or Nextcloud of mine, but about the forum here.

With ctrl + shift + j i get much less errors. But the errors does not really help.