[Nextcloudpi] Performance enhancements

ā„¹ļø Support šŸ“¦ Appliances (Docker, Snappy, VM, NCP, AIO)
1

Iā€™d been posting this on github, but nachoparker pointed out that it would be much better to have this discussion on the forums, so I copy & paste my comment here:

Hi,

Iā€™ve been using owncloud later nextcloud for a couple of years now. In the early days I crashed my sqlite DB every now an then, but for the last 2 to 3 years I was able to establish a good backup routine. When I saw that you guys created this faboulous nextcloudpi project I wanted to get rid of all the configuration affort I have under arch linux and migrated my database first to mariadb and afterwards over to a new nextcloudpi image.

Since the late Nextcloud 11 versions Iā€™ve been struggling with the performance. The patch here initially helped me a a lot, but my NC install is still very very slow (compared to where it was half a year ago on exactly the same hardware).
nextcloud/server#2272

I was wondering if you have any recommendations on how to identify bottlenecks. I currently donā€™t even know if the database is the problem, if the nextcloud code is the problem or if I configured stuff incorrectly.

Since this project focuses on limited hardware resources, I thought it might be helpful to have a conversation here. Discussions about performance in other places on github usually includes people with high powered VMā€™s and endless RAM - not really what we have here.

2

Good idea, letā€™s see how far we can go with some tuning. What we need are some test scenarios to measure the performance, e.g. time to login, upload of two mp3 albums, upload of Nextcloud source code (via NC-client, directly via webdav, ā€¦ and the best configuration for fast upload of small files and the use of calendar and contacts with as many clients as possible will probably be different.

Where did you see the slower performance, is it just the web-interface, the sync performance or everything? And which apps are used?

3

Thanks for the quick reply!

pi@nextcloudpi:~ $ sudo -u www-data php /var/www/nextcloud/occ app:list
Enabled:

  • activity: 2.5.2
  • admin_audit: 1.2.0
  • admin_notifications: 1.0.0
  • audioplayer: 2.1.0
  • calendar: 1.5.6
  • comments: 1.2.0
  • contacts: 2.0.1
  • dav: 1.3.0
  • deck: 0.2.4
  • federatedfilesharing: 1.2.0
  • federation: 1.2.0
  • files: 1.7.2
  • files_clipboard: 0.6.4
  • files_external: 1.3.0
  • files_pdfviewer: 1.1.1
  • files_sharing: 1.4.0
  • files_texteditor: 2.4.1
  • files_trashbin: 1.2.0
  • files_versions: 1.5.0
  • firstrunwizard: 2.1
  • gallery: 17.0.0
  • groupfolders: 1.1.0
  • logreader: 2.0.0
  • lookup_server_connector: 1.0.0
  • mail: 0.7.4
  • news: 11.0.5
  • nextcloud_announcements: 1.1
  • notes: 2.3.1
  • notifications: 2.0.0
  • oauth2: 1.0.5
  • password_policy: 1.2.2
  • phonetrack: 0.0.5
  • provisioning_api: 1.2.0
  • serverinfo: 1.2.0
  • sharebymail: 1.2.0
  • survey_client: 1.0.0
  • systemtags: 1.2.0
  • tasks: 0.9.5
  • theming: 1.3.0
  • twofactor_backupcodes: 1.1.1
  • updatenotification: 1.2.0
  • workflowengine: 1.2.0
    Disabled:
  • encryption
  • files_videoplayer
  • user_external
  • user_ldap

I am using the nextcloud dev app, davdroid, owncloud news, ulogger on android. The nextcloud client, evolution for calendar, contacts and tasks under linux.

All clients and the website show extreme slow behavior (iā€™d experienced way better performance on the same hardware with the same clients before).

4

More testing! this is great!

It seems that we still need more and more of this. I was surprised that I didnā€™t hear about the sync problems from the sync client before.

In order for this to be really helpful for everyone, I suggest that instead of migrating your old database, you export your contacts, task/calendar, opml and so on, and copy your files somewhere else, and then start with a fresh copy of NCP (no database modifications, clean), re-import everything and we are in a good spot to start testing. You would only have to re-create the users by hand.

Then we can try to come up with some testing methodology. I will try to create a test plan based on this and the other thread about the sync client, but I need help for this because my hands are full right now

I know I can trust a fellow Arch user to be techy savy for this :wink:

5

Sounds like a plan! I am currently on the road, so it might take some days until I can create a new image, but I think I will keep the current image in order to get back to it in case we want some testing material.

As to me experience, I havent touched php code in 15 years, so I wonā€™t be able to help write any tests, but I am good in punching lines of commands into a terminal if that helps.

6

Great! Sounds good

I donā€™t think we need to edit any php code, but come up with a methodology and tools for performance testing, specially if they can be scripted and automated, in bash or other tool.

Anything that is tested through a browser can be written in selenium, capybara, rspecā€¦

For instance, NCP uses python-selenium for its tests, we can measure loading, login times, as well as any operation that can be done in a web, and itā€™s python, so itā€™s relatively easy

If we are using some external tools, for instance a command line capable webdav client (even davfs?), that could also go into the mix.

Then we need to see what are or KPIs

  • CPU
  • I/O
  • waiting time (locks)
  • time to upload/throughput

And compare.

We donā€™t need to go that far, anything we can get done will be really useful

7

Ok, here the plan (if my wife lets me do this on a Sunday nightā€¦):

  • I will use a fresh nextcloudpi image on a fresh SD card.
  • Run the test
  • log the results
  • Add contacts to mariadb
  • Run the test
  • log the results
    Add calendars to mariadb
  • Run the test
  • log the results
    Add files to mariadb
  • Run the test
  • log the results
  • Rinse and repeat for other apps like News/PhoneTrack/Gallery
8

cool,

what do you mean by ā€˜run the testā€™?

what performance indicators are we looking at?