Performance tuning AIO

paulororke · September 29, 2023, 6:19pm

Hi all,

I recently set up a NextCloud AIO instance at work and am having performance issues. I was excited to see a reference to a NextCloud Performance Tuning on Raspberry Pi.

I thought that might be a good place to start. If people are running this successfully on a Pi, then my Dell server with 24 cores and 128GB RAM and SSD RAID10 should be up to the task. I even put it on bare metal because my preparatory reading said a VM (as I typically would have done) will suffer a performance hit.

What I am seeing is similar to what was reported here: AIO Solution, slow performance - but his solution was a re-install, something I would prefer avoiding.

I am being a tad “subversive” here and setting up a collaboration server for our devs. The use case is simply securely sharing resources (large files) to disparate locations, but my “hidden” goal is to present to them the collaboration capabilities in the hope of encouraging them to more fully support webDAV in our PDF application. Our PDF application is very popular in Germany, and I am of the mind that we should be leveraging the power of NextCloud collaboration to provide an underlying mechanism for collaboration on PDF files within our software.

Having NextCloud available and exposing them to it’s capabilities would, I believe help me get the team on board with my idea. Unfortunately in it’s current form it is barely usable.

I would love to get this performing properly and present my case to the team. Can anyone pint me in the right direction for how to figure out what is happening?

On first attempting to access the site it can just sit there with the cursor spinning. Sometimes it is minutes, sometime literally days. Yup, I have left a login attempt running that long to see.

It seems that once a session is established it sometimes is snappy, then will revert to being very very slow. System resources are barely used. It is not a lack of hardware.

Nextcloud AIO v7.2.1

I am really keen to see this showcase all the great things NextCloud offer and get my team supportive of building on it. Any pointers would be greatly appreciated.

Paul

szaimen · September 29, 2023, 6:22pm

Hi, can you follow How to debug performance issues with Nextcloud? · nextcloud/all-in-one · Discussion #2475 · GitHub?

just · September 29, 2023, 6:24pm

Hi @paulororke,
You are missing the required support template. Please fill this form out and edit into your post. You will need to enter your docker container to confirm some of this information: How To Use docker exec to Run Commands in a Docker Container | DigitalOcean

This will give us the technical info and logs needed to help you! Thanks.

paulororke · September 29, 2023, 7:18pm

I had high hopes for that brute force piece, unfortunately disabling it didn’t change the behaviour.

I really appreciate the speed with which you guys responded. This has helped a lot I am seeing my IP is being throttled.

However the docs it suggests reading are for an instance behind a reverse proxy. I am not using one, It is directly behind a port forward in the DMZ. I need any external IP to be able to connect without being throttled.

I can add this one to trusted_proxies but it seems to me that is barking up the wrong tree.

I wear many hats and it may be a little while before I can come back with additional details, but thanks to this I now have a reading list.

I do apologize for not including the template details, I wasn’t prompted and forgot…

I’ll be back after I do some more reading and tests. Thank you for the incredibly speedy reply.

paulororke · October 4, 2023, 9:06pm

That was a good tack to take, it didn’t turn out being the cause, but was definitely good to know.

I have it nicely responsive now. It seems that the issue wasn’t the Brute Force detection, but some misconfiguration on the back end with some of the collaboration features and timeouts connecting to services not properly configured.

Since the primary use case is secure sharing of large files I just disabled any apps that looked like that may be related and the server is snappy.

I am still getting lots of timeout errors in the logs but that is a different issue I will come back to as time permits.

Thanks for the help.

szaimen · October 4, 2023, 9:22pm

Can you clarify this? Usually AIO should take care of the correct configuration so that wasnt the case for you apparently?

paulororke · October 4, 2023, 10:12pm

Hi again szaimen

to be honest I cannot remember whether these were “Out of the Box” or whether I was mucking about with apps. I am a “generalist” and I wear too many hats. Sorry - I didn’t take decent notes while doing this.

I saw:

Failed to fetch the Collabora capabilities endpoint: cURL error 28: Connection timed out after 45000 milliseconds

And started disabling apps.

"2023-10-04T19:44:12+00:00","remoteAddr":"a.b.c.d","user":"admin","app":"richdocuments","method":"POST","url":"/login/confirm","message":"Failed to fetch the Collabora capabilities endpoint: cURL error 28: Connection timed out after 45000 milliseconds (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://*** sensitive parameters replaced ***/hosting/capabilities","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36","version":"27.1.0.7","exception":{"Exception":"GuzzleHttp\\Exception\\ConnectException","Message":"cURL error 28: Connection timed out after 45000 milliseconds (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://*** sensitive parameters replaced ***/hosting/capabilities","Code":0,"Trace":[{"file":"/var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php","line":158,"function":"createRejection","class":"GuzzleHttp\\Handler\\CurlFactory","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php","line":110,"function":"finishError","class":"GuzzleHttp\\Handler\\CurlFactory","type":"::","args":[["GuzzleHttp\\Handler\\CurlHandler"],"*** sensitive parameters replaced ***",["GuzzleHttp\\Handler\\CurlFactory"]]},{"file":"/var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlHandler.php","line":47,"function":"finish","class":"GuzzleHttp\\Handler\\CurlFactory","type":"::","args":[["GuzzleHttp\\Handler\\CurlHandler"],"*** sensitive parameters replaced ***",["GuzzleHttp\\Handler\\CurlFactory"]]},{"file":"/var/www/html/3rdparty/guzzlehttp/guzzle/src/Middleware.php","line":137,"function":"__invoke","class":"GuzzleHttp\\Handler\\CurlHandler","type":"->","args":["*** sensitive parameters replaced ***","*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/Http/Client/DnsPinMiddleware.php","line":114,"function":"GuzzleHttp\\{closure}","class":"GuzzleHttp\\Middleware","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php","line":35,"function":"OC\\Http\\Client\\{closure}","class":"OC\\Http\\Client\\DnsPinMiddleware","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/guzzlehttp/guzzle/src/Middleware.php","line":31,"function":"__invoke","class":"GuzzleHttp\\PrepareBodyMiddleware","type":"->","args":["*** sensitive parameters replaced ***","*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/guzzlehttp/guzzle/src/RedirectMiddleware.php","line":71,"function":"GuzzleHttp\\{closure}","class":"GuzzleHttp\\Middleware","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/guzzlehttp/guzzle/src/Middleware.php","line":63,"function":"__invoke","class":"GuzzleHttp\\RedirectMiddleware","type":"->","args":["*** sensitive parameters replaced ***","*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/guzzlehttp/guzzle/src/HandlerStack.php","line":75,"function":"GuzzleHttp\\{closure}","class":"GuzzleHttp\\Middleware","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/guzzlehttp/guzzle/src/Client.php","line":331,"function":"__invoke","class":"GuzzleHttp\\HandlerStack","type":"->","args":["*** sensitive parameters replaced ***","*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/guzzlehttp/guzzle/src/Client.php","line":168,"function":"transfer","class":"GuzzleHttp\\Client","type":"->","args":["*** sensitive parameters replaced ***","*** sensitive parameters replaced ***"]},{"file":"/var/www/html/3rdparty/guzzlehttp/guzzle/src/Client.php","line":187,"function":"requestAsync","class":"GuzzleHttp\\Client","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/Http/Client/Client.php","line":230,"function":"request","class":"GuzzleHttp\\Client","type":"->","args":["*** sensitive parameters replaced ***","*** sensitive parameters replaced ***/hosting/capabilities",["/mnt/ncdata/files_external/rootcerts.crt",45,[true],["Nextcloud Server Crawler","gzip"],true]]},{"file":"/var/www/html/custom_apps/richdocuments/lib/Service/CapabilitiesService.php","line":135,"function":"get","class":"OC\\Http\\Client\\Client","type":"->","args":["https://*** sensitive parameters replaced ***/hosting/capabilities",[45,[true]]]},{"file":"/var/www/html/custom_apps/richdocuments/lib/Service/CapabilitiesService.php","line":73,"function":"refetch","class":"OCA\\Richdocuments\\Service\\CapabilitiesService","type":"->","args":[]},{"file":"/var/www/html/custom_apps/richdocuments/lib/AppInfo/Application.php","line":90,"function":"getCapabilities","class":"OCA\\Richdocuments\\Service\\CapabilitiesService","type":"->","args":[]},{"file":"/var/www/html/lib/private/AppFramework/Bootstrap/FunctionInjector.php","line":66,"function":"OCA\\Richdocuments\\AppInfo\\{closure}","class":"OCA\\Richdocuments\\AppInfo\\Application","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/lib/private/AppFramework/Bootstrap/BootContext.php","line":50,"function":"injectFn","class":"OC\\AppFramework\\Bootstrap\\FunctionInjector","type":"->","args":[["Closure"]]},{"file":"/var/www/html/custom_apps/richdocuments/lib/AppInfo/Application.php","line":145,"function":"injectFn","class":"OC\\AppFramework\\Bootstrap\\BootContext","type":"->","args":[["Closure"]]},{"file":"/var/www/html/lib/private/AppFramework/Bootstrap/Coordinator.php","line":200,"function":"boot","class":"OCA\\Richdocuments\\AppInfo\\Application","type":"->","args":[["OC\\AppFramework\\Bootstrap\\BootContext"]]},{"file":"/var/www/html/lib/private/App/AppManager.php","line":437,"function":"bootApp","class":"OC\\AppFramework"nextcloud.log.1" 1456L, 10487937C

I disabled:

Collabora Online - Built-in CODE Server
Full text search
Full text search - Elasticsearch Platform
Full text search - Files
Nextcloud Office

and figured I would run a process of elimination enabling them one at a time.

One thing I noticed, I re-enabled brute force detection using OCC and the command appears to have run successfully, see the output below, yet “Brute-force settings” is still in the apps list in the UI as needing to be enabled:

@nc-hub:~$ sudo docker exec --user www-data -it nextcloud-aio-nextcloud php occ config:system:set uth.bruteforce.protection.enabled --type bool --value=true
System config value uth.bruteforce.protection.enabled set to boolean true
@nc-hub:~$ sudo docker exec --user www-data -it nextcloud-aio-nextcloud php occ config:system:set ratelimit.protection.enabled --type bool --value=true
System config value ratelimit.protection.enabled set to boolean true

Are they not the same thing?

cheers

szaimen · October 4, 2023, 10:40pm

I wonder why did you have this enabled? This is nothing AIo installs so probably you did? This could explain the problem that you ran into.

No they arent.

paulororke · October 5, 2023, 7:20pm

Yes, I confess to “playing” around with different apps before I really knew what I was doing. I am not sure I do yet regards real time collaboration, and have more reading and planning to do there.

I have this instance delivering files nicely now, this issue of the performance is resolved, I will post some other questions I have as new topics.

Thanks for getting me though this.

paulororke · October 31, 2023, 8:25pm

I am not sure if etiquette requires a new topic here or not, but I am again seeing “intermittent” timeouts.

Only reply here if:

You have additional details

The solution doesn’t work for you

I am going with the second because it is better than it was but still timing out.

As time permits I will attempt a second instance and see if AIO does indeed handle all the collaboration set up as suggested. Before I can even go there, I need to know that this simple cloud server install (simple as in no collaboration required, just file serving and sharing) will fit the bill.

I am very sure it is exactly what we need, and those who have been using it thus far have had very positive responses saying it significantly improved their efficiency. So…

The logs are filled with this “GuzzleHttp\Exception\ConnectException”

[richdocuments] Error: GuzzleHttp\Exception\ConnectException: cURL error 28: Connection timed out after 45002 milliseconds (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://files.redacted.redacted.support/hosting/capabilities at <<closure>>

 0. /var/www/html/custom_apps/fulltextsearch_elasticsearch/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php line 158
    GuzzleHttp\Handler\CurlFactory::createRejection("*** sensitive parameters replaced ***")
 1. /var/www/html/custom_apps/fulltextsearch_elasticsearch/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php line 110
    GuzzleHttp\Handler\CurlFactory::finishError(["GuzzleHttp\\Handler\\CurlHandler"], "*** sensitive parameters replaced ***", ["GuzzleHttp\\Handler\\CurlFactory"])
 2. /var/www/html/custom_apps/fulltextsearch_elasticsearch/vendor/guzzlehttp/guzzle/src/Handler/CurlHandler.php line 47
    GuzzleHttp\Handler\CurlFactory::finish(["GuzzleHttp\\Handler\\CurlHandler"], "*** sensitive parameters replaced ***", ["GuzzleHttp\\Handler\\CurlFactory"])
 3. /var/www/html/custom_apps/fulltextsearch_elasticsearch/vendor/guzzlehttp/guzzle/src/Middleware.php line 142
    GuzzleHttp\Handler\CurlHandler->__invoke("*** sensitive parameters replaced ***", "*** sensitive parameters replaced ***")
 4. /var/www/html/lib/private/Http/Client/DnsPinMiddleware.php line 114
    GuzzleHttp\Middleware::GuzzleHttp\{closure}("*** sensitive parameters replaced ***")
 5. /var/www/html/custom_apps/fulltextsearch_elasticsearch/vendor/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php line 35
    OC\Http\Client\DnsPinMiddleware->OC\Http\Client\{closure}("*** sensitive parameters replaced ***")
 6. /var/www/html/custom_apps/fulltextsearch_elasticsearch/vendor/guzzlehttp/guzzle/src/Middleware.php line 31
    GuzzleHttp\PrepareBodyMiddleware->__invoke("*** sensitive parameters replaced ***", "*** sensitive parameters replaced ***")
 7. /var/www/html/custom_apps/fulltextsearch_elasticsearch/vendor/guzzlehttp/guzzle/src/RedirectMiddleware.php line 71
    GuzzleHttp\Middleware::GuzzleHttp\{closure}("*** sensitive parameters replaced ***")
 8. /var/www/html/custom_apps/fulltextsearch_elasticsearch/vendor/guzzlehttp/guzzle/src/Middleware.php line 66
    GuzzleHttp\RedirectMiddleware->__invoke("*** sensitive parameters replaced ***", "*** sensitive parameters replaced ***")
 9. /var/www/html/custom_apps/fulltextsearch_elasticsearch/vendor/guzzlehttp/guzzle/src/HandlerStack.php line 75
    GuzzleHttp\Middleware::GuzzleHttp\{closure}("*** sensitive parameters replaced ***")
10. /var/www/html/custom_apps/fulltextsearch_elasticsearch/vendor/guzzlehttp/guzzle/src/Client.php line 333
    GuzzleHttp\HandlerStack->__invoke("*** sensitive parameters replaced ***", "*** sensitive parameters replaced ***")
11. /var/www/html/custom_apps/fulltextsearch_elasticsearch/vendor/guzzlehttp/guzzle/src/Client.php line 169
    GuzzleHttp\Client->transfer("*** sensitive parameters replaced ***", "*** sensitive parameters replaced ***")
12. /var/www/html/custom_apps/fulltextsearch_elasticsearch/vendor/guzzlehttp/guzzle/src/Client.php line 189
    GuzzleHttp\Client->requestAsync("*** sensitive parameters replaced ***")
13. /var/www/html/lib/private/Http/Client/Client.php line 230
    GuzzleHttp\Client->request("*** sensitive parameters replaced ***", "https://files.t ... s", ["/mnt/ncdata/fi ... e])
14. /var/www/html/custom_apps/richdocuments/lib/Service/CapabilitiesService.php line 135
    OC\Http\Client\Client->get("https://files.t ... s", [45,[true]])
15. /var/www/html/custom_apps/richdocuments/lib/Service/CapabilitiesService.php line 73
    OCA\Richdocuments\Service\CapabilitiesService->refetch()
16. /var/www/html/custom_apps/richdocuments/lib/AppInfo/Application.php line 90
    OCA\Richdocuments\Service\CapabilitiesService->getCapabilities()
17. /var/www/html/lib/private/AppFramework/Bootstrap/FunctionInjector.php line 66
    OCA\Richdocuments\AppInfo\Application->OCA\Richdocuments\AppInfo\{closure}("*** sensitive parameters replaced ***")
18. /var/www/html/lib/private/AppFramework/Bootstrap/BootContext.php line 50
    OC\AppFramework\Bootstrap\FunctionInjector->injectFn(["Closure"])
19. /var/www/html/custom_apps/richdocuments/lib/AppInfo/Application.php line 145
    OC\AppFramework\Bootstrap\BootContext->injectFn(["Closure"])
20. /var/www/html/lib/private/AppFramework/Bootstrap/Coordinator.php line 200
    OCA\Richdocuments\AppInfo\Application->boot(["OC\\AppFramewo ... "])
21. /var/www/html/lib/private/App/AppManager.php line 437
    OC\AppFramework\Bootstrap\Coordinator->bootApp("richdocuments")
22. /var/www/html/lib/private/App/AppManager.php line 216
    OC\App\AppManager->loadApp("richdocuments")
23. /var/www/html/lib/private/legacy/OC_App.php line 126
    OC\App\AppManager->loadApps([])
24. /var/www/html/lib/base.php line 1048
    OC_App::loadApps()
25. /var/www/html/index.php line 36
    OC::handleRequest()

GET /index.php/apps/files/preview-service-worker.js
from 154.5.66.51 by admin at 2023-10-31T20:16:56+00:00

My old school approach of disabling things until the error goes away is not exactly efficient or the way I want to handle this, so can anyone tell me what it is trying to do and what the right approach to fixing this might be?

szaimen · October 31, 2023, 8:32pm

Hi, see How to debug problems with Collabora and/or Talk · nextcloud/all-in-one · Discussion #1358 · GitHub

paulororke · October 31, 2023, 9:42pm

Damn you are good!

Timeouts have stopped and I am thrilled how fast this is!

paulororke · November 6, 2023, 3:26pm

It would seem I am not properly removing the features that are causing the time outs because they keep being “restored”.

I will confess to having only first “disabled” them in the Admin UI. They would re-enable the next day, so next time I disabled and “removed” them. They still come back.

Do I need to change the Docker command that was used to set this up initially to tell it not to use “Nextcloud Office”? Does AIO have a dependency on “Nextcloud Office” that it is trying to fulfill?

I am so close to having this do what I want, but the timeouts returning is causing user resistance. At this juncture I do not need the Office components and I need to understand how to stop the timeouts. Is there a better way to stop them returning than “Disabling” and “Removing” them?

I did expect “removing” them to prevent their being re-enabled. What am I missing? Is it because it is a “Featured” app?

szaimen · November 6, 2023, 3:41pm

You need to disable Collabora from the AIO interface.

paulororke · November 6, 2023, 5:34pm

That looks to have done it.

Once again I am in your debt.