NextCloud Performance Optimizations - Please Contribute!

The single page we do have on optimizing NextCloud performance, in itā€™s entirety, only refers to configuration a depreciated feature (static file caching, i.e. assets folder). I have spent so much time searching on this topic and I would like to get input on anything others know about improving NextCloud performance. I will compile it all into a single post and share the link until it can be posted to the wiki.

I have ran many ownCloud & NextCloud servers on some pretty high-end hardware. Needless to say it doesnā€™t run near as fast as many other webapps, even other dynamic php-based platforms. Obviously itā€™s not going to run like a static HTML site, but there has got to be many improvements that can be made.

Assuming one has installed NextCloud per default instructions (Apache, MySQL, PHP 5.6), and nothing else has been done, what can be done to increase speed and optimize performance of NextCloud and all subsystems?

It would be useful to hear any experience running alternate versions of PHP (PHP-FPM, HHVM, etc), alternate Databases, various database optimizations (anything done beyond a default install of MySQL), static file caching through reverse proxies (Nginx, etc), alternate webservers (Nginx, Litespeed, LigHTTPD, etcā€¦) and, redis optimizations, as well as alternate memcache implementations, and anything else that can optimize/increase performance and response time.


UPDATED 11/25/16


Unverified / Further Testing Needed / Unknown:

.
.

  1. Install non-redis memcache?? - Compatibility and performance unknown.
  2. Alternate DB Server (MariaDB, Postgres, etc)ā€¦ - Compatibility and performance unknown.
  3. Alternate webserver (Nginx, Lighttpd, Litespeed, etc) - Compatibility and performance impact unknown, Further testing needed.
  4. Nginx Caching in front of Apache - Performance impact unknown. Further testing needed
  5. PHP-FPM - Compatibility unknownā€¦
  6. HHVM - Compatibility unknownā€¦
  7. MySQL Performance Tuning - Significant performance improvements possible. More testing and specifics needed.
    . . Tuning Primer ā€“ Performance metrics and specifics TBD.
    . . MySQLTuner ā€“ Performance metrics and specifics TBD.

ā€¦
ā€¦
ā€¦

Verified Performance Optimizations:

.
.

  1. MySQL buffer tweak to drastically increase performance syncing large # of small files on RaspberryPi.
  2. Install Redis for memcaching - Significant performance increase - Metrics TBD.
  3. Upgrade to PHP 7 - Improves performance approx x2. Exact metrics TBD.

ā€¦
ā€¦
ā€¦
ā€¦
ā€¦
ā€¦
.

Thanks!
Mike

6 Likes

Nothing? No one has been able to optimize performance at all?

php 7.0 will improve the performance a bit (up to a factor 2). If you already touched redis, I would have a look into mysql-caches and other settings:
Tuning Primer
https://launchpad.net/mysql-tuning-primer

MySQLTuner
http://mysqltuner.com/

I could considerably increase the upload speed:

I have done since 2 days my first installation of nextcloud. This is on a small Raspberry Pi with nginx and php-fpm.For now, i hadnā€™t much time to test and tweak the performance of this installation. Not all parts of my installation and primary Requirements are work for now, so i must make them work before. But after that, i could made some tweaks.

Specially i have one thing which makes me headaches for now: The Pi has limited resources and i would like to use additionally Z-Push to deliver Calendar, Mail and Contacts to my Smartphones. But if i try to use this, it comes to timeouts with nginx, because, nginx cant handle this much connections. Raising the timeout to 300 or more does not really help. Maybe someone has another idea.

@tflidd Thank you so much for your suggestions. Iā€™m coming up with a baseline method to do performance testing metrics. I will test php7 first to see how much performance is gained. Also, thank you for the links on tuning MySQL, I will do the same with those.

@Dist1958 Thank you as well for the info. I am also interested in using Z-Push. Is it possible to integrate Z-Push on the NextCloud server to deliver cal/contacts and external mail (say via IMAP server)? That would solve many problems for myself and a couple colleagues. I can definitely see how every bit of performance you can squeeze out of a RPi would be helpful. Hopefully when itā€™s all said and done we can compile enough info here to do just that.

I will add this info to the OP and update when performance metrics available. In the mean time, maybe some more folks can chime in. Thanks again.

Ohhh Z-Push works on the PI with every CalDAV, CardDAV and IMAP Server. It doesnā€™t matter, if the Server is inside or outside the network. Only, all Accounts for one user should share the same Credentials for Username and Password. Thatā€™s the only limitation. Since Z-Push uses a TCP Connection to every Service, thatā€™s the bottleneck in my opinion. I had run Z-Push with CalDAV, CardDAV and a external IMAP on the same PI before. CalDAV and CardDAV was a Baikal Server instead of NextCloud. Sure, i hade the above Problems with timeouts, but i hope that we can solve them. Unfortunately, my SD Card crashed with this installation, so i have lost my configuration i have made for this day and i start from Scratch.

IMAP works, CalDAV comes back with a http error 500. I think there is some php lib missing. CardDAV is untested. If i get CalDAV work today, i try as next CardDAV and then the Combined Backend.If it works basically, i can report my installation so far. then wen can optimize it.

PS: Found the missing lib. It was libawl-php, now CalDAV and CardDAV works both. I try Combined Backend today in the evening. But since all the standalone works, the Combination should work also. There are only minor configuration changes what i must make, such as mapping of IMAP Folders and such things. Butā€¦ it works!

That is a game changer for me, and a few of my customers that are running NextCloud. I had never even looked into Z-Push for activesync + NextCloud as I assumed external IMAP servers would be a problem.

Do you have any specific documentation you have used to setup this configuration? I will get it set up on my personal NextCloud server. That will help with getting additional performance metrics and optimization information as well.

On another note, which SD card are you using in your RPi? I did A LOT of testing of MicroSD cards in my RP3, and read just about everything there is to read, and I found the Samsung Evo Plus 32GB card to vastly outperform all other cards, specifically on random reads and writes, which is probably the most vital metric using NextCloud + RPi. There was a massive difference in performance when I switched to this card. I canā€™t tell you enough good things about it. Every single use case Iā€™ve used it in has ran circles around other cards, especially anything using a MySQL database, or heavy random reads/writes. There are a few other cards out there that will smoke it on sequential reads but they fall far behind on general use and randoms, which is more important than sustained transfers in just about every case except for simple backups or storage of movie/music or other static files.

This is the exact card I use.

Thanks again for joining the thread and providing the info!!

1 Like

Some hints from my OC@PI3:

  • mysql: increase the innodb buffer to have all data cached
  • mount /tmp and /var/tmp to temfs
  • use the current nginx from stretch repository (default apt is outdated; not http2 capable)
  • i mounted oc-data, webroot and sql-files onto a USB3 stick to take load off the sd-card

For some more detail tweaks of PHP check

Harhar, my SD cardā€¦ its a so cheap 4GB NoName Cardā€¦ I think its only Class 4ā€¦ but i use this only for booting. All other files are on a 1TB HDD. Speed via Firefox and NextCloud GUI is OK for my feeling, but at this time, im the only person which uses the PI. There are only some small access times, where other in my family syncs their small KeePass Databases via WebDAV. In this Case, NextCloud isnā€™t involved, as i use WebDAV for this via Nginx and additional headers patch. I think this WebDAV is a lot faster than the WebDAV from NextCloud. I have not done some real testing for now, but i try to install a small soapUI suite, to test a little bit, when is up and running.

I would like to use the standard Raspbian Applications, so that every user could install it with ease. Maybe only if it is really needed, i could compile some Applications with special setups. In this case, the result could be a Image for a SD-Card, ready for downloading. But in this case, we should have someone, how can maintain his own repository for the custom software builds.

About MySQL buffer tweak:

I followed your links till the end and found this settings for the my.cnf:

  • innodb_buffer_pool_size = 512M
  • innodb_buffer_pool_instance = 1
  • innodb_flush_log_at_trx_commit = 2
  • innodb_log_buffer_size = 32M
  • innodb_max_dirty_pages_pct = 90

innodb_buffer_pool_size defaults to 128M: source
Increasing it should of course depend on your system and there should be enough memory left for the remaining to run safely. Also it doesnā€™t make sense to increase the buffer, if the database is too small to fill.

innodb_buffer_pool_instance already defaults to 1, so there is no need to change: source

Reading on it seems, that at least the change of innodb_flush_log_at_trx_commit can lead to data loss when the system looses power. So this should only be set, if there is a battery backup or something. Also the performance improvement was not confirmed by all, but most of the users.

innodb_log_buffer_size defaults to 8 (source), so yeah, at least it seems consistent to adjust it by the same factor then the buffer pool size.

innodb_max_dirty_pages_pct is the percentage of dirty pages that is tried to no exceed by flushing the buffer regularly. I donā€™t have enough knowledge about it, but it has benefits and disadvantages to adjust this value and there will be some reason, why it defaults to 75 (%): source

I am always interested in increasing the performance of my nextcloud server, but as I already thought: Just changing the default values of database, webserver, php, whatever, to a set, given by some random user (of course I appreciate the hints, and it will work for his specific system/setup) is not an always good advice and should be taken with care. There are good reasons why the standard values are set as they set. Of course they are not set to maximum performance but indeed to a good compromise of stability, compatibility and performance.

Upgrading to PHP 7 and using file caching/locking should be some good way to increase performance along with other hints in the admin manual.

@Rello: /var/tmp seems to be used for temporary files, that are meant to survive reboot, as far as I could find out. I donā€™t know how important this is and if it could lead to problems, but this should be kept in mind. /tmp on the other hand seems to be for files, that do not ā€œneedā€ to survive reboot. Of course its always a question how this folder are actually used by the system. At least for the /tmp folder there is already a setting in debian jessie: /etc/defaults/tmpfs
RAMTMP=yes
It defaults to no, set to yes should add it to tmpfs after reboot.

By the way: While having a look into the nextcloud admin manual I found some default php mysql configuration for /etc/php5/conf.d/mysql.ini. But actually with PHP7 I donā€™t have this file, nor could I find some mysql.ini anywhere. mysqli.ini and mysqlnd.ini instead. Does anyone know which file I have to adjust according to the manual with PHP7?

Thanks for the info.

Default settings should always be taken with a grain of salt. As with most packages, they are NOT optimized for performance or security out of the box. They are set to be universally compatible in hopes the default configuration will ā€œjust workā€ with whatever software, hardware, and os is thrown at it. I donā€™t believe the defaultā€™s should be used for many cases. From the operating system down, it can be dangerous in many cases to use default settings. It can also be extremely beneficial to performance to dig in and adjust appropriately. NextCloud (and ownCloud) do not perform well, out of the box, with defaults. Iā€™ve never been on to accept ā€œgood enoughā€ when we can do better. In the end I think substantial gains can be attained with minimal effort without added risk.

Thank you for the details WRT to the MySQL buffer tweak. I will add this info to the OP. As these tweaks (and that thread) is focused on the RaspberryPi and other severely-limited-performance use cases, I might break that out from the rest of the info in the OP and create itā€™s own sub-topic.

Thanks for all of the input so far!

1 Like

Ya of course I agree with you, that there should be done several changes and additions to the standard i.e. Raspbian system. Some of them are more less universally recommendable for all kinds of debian servers. But the more you go into detail, the more you have to figure out what is the right solution for your specific system.
My nextcloud database for example is about 48 MB, so it should have absolutely has no impact on performance, if I increase its buffer size. On the other hand it would limit the memory for the remaining system. If I had some other stuff running on that server, would use the pi as media center with hd movies an increased the video memory for that, I could run into problems :wink: . But of course in other situation, huge mysql databases, it could be a great improvement.

There are already some well proved and back checked advices in the admin manual with explanations and different alternatives. Optimizations that are not mentioned there might be difficult to advice in general. But of course thatā€™s a reason to make this a topic here. It would be great to enhance the performance related aspects in the admin manual :slight_smile: .

I agree. I couldnā€™t find much of anything in the admin manual except for references to the depreciated assets folder.

I just installed and testet MySQLTuner:

  • installed via # apt-get install mysqltuner on raspbian jessie.
  • startet via # mysqltuner
  • output:

>> MySQLTuner 1.3.0 - Major Hayden major@mhtx.net
>> Bug reports, feature requests, and downloads at http://mysqltuner.com/
>> Run with ā€˜ā€“helpā€™ for additional options and output filtering
[OK] Logged in using credentials from debian maintenance account.
[OK] Currently running supported MySQL version 5.5.52-0+deb8u1
[OK] Operating on 32-bit architecture with less than 2GB RAM

-------- Storage Engine Statistics -------------------------------------------
[ā€“] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MRG_MYISAM
[ā€“] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
[ā€“] Data in InnoDB tables: 21M (Tables: 53)
[!!] Total fragmented tables: 53

-------- Security Recommendations -------------------------------------------
[OK] All database users have passwords assigned

-------- Performance Metrics -------------------------------------------------
[ā€“] Up for: 21h 18m 43s (55K q [0.726 qps], 1K conn, TX: 359M, RX: 7M)
[ā€“] Reads / Writes: 53% / 47%
[ā€“] Total buffers: 328.0M global + 2.7M per thread (151 max threads)
[!!] Maximum possible memory usage: 733.8M (99% of installed RAM)
[OK] Slow queries: 0% (0/55K)
[OK] Highest usage of available connections: 3% (6/151)
[OK] Key buffer size / total MyISAM indexes: 16.0M/103.0K
[OK] Query cache efficiency: 85.2% (34K cached / 40K selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 1K sorts)
[!!] Joins performed without indexes: 954
[OK] Temporary tables created on disk: 15% (1K on disk / 6K total)
[OK] Thread cache hit rate: 99% (6 created / 1K connections)
[OK] Table cache hit rate: 34% (94 open / 270 opened)
[OK] Open file limit used: 4% (48/1K)
[OK] Table locks acquired immediately: 100% (7K immediate / 7K locks)
[OK] InnoDB buffer pool / data size: 256.0M/21.8M
[OK] InnoDB log waits: 0
-------- Recommendations -----------------------------------------------------
General recommendations:
=> Run OPTIMIZE TABLE to defragment tables for better performance
=> MySQL started within last 24 hours - recommendations may be inaccurate
=> Reduce your overall MySQL memory footprint for system stability
=> Enable the slow query log to troubleshoot bad queries
=> Adjust your join queries to always utilize indexes
Variables to adjust:
=> *** MySQLā€™s maximum memory usage is dangerously high ***
=> *** Add RAM before increasing MySQL buffer variables ***
=> join_buffer_size (> 128.0K, or always use indexes with joins)

  • First of all I could see that the FEDERATED engine is not used. I could deactivate it by adding skip-federated to the [mysqld] section of the my.cnf
    ā‚¬: Ah, no the tool sais, that the engine is not active, because the federated engine is deactivated by default. So no change necessary here ;).

  • It gives the advice to optimize my tables to reduce defragmentation. I ran # mysqlcheck -uroot -p -o --all-databases but as InnoDB does not support optimization (source) there was no change in fragmentation.

  • I have no idea how to limid the MySQLs maximum memory usage beside adjusting all buffer variables separately. For now I never had a memory usage that was even close to full, so I will ignore that warning for now. I doubled my innodb pool buffer, which I would revert now, after realising that my databases are less than 50 MB. Maybe that already resolves the warning.

  • I donā€™t know much about this join queries thing and if I should really add i.e. join_buffer_size = 256.0K. Maybe someone could give me some explanation about that. Also itā€™s a good idea to let the database run longer to collect more statistics before making some more changes.

So I just wanted to give you some short insight into this tool. It seems to be some good starting point for MySQL analyses and optimization. The tests made are listet here on GitHub, but further explanations and guides how to resolve the warnings/results are missing so far and need to be found out by oneself ;).

Ciao,
before touch DB I would suggest to find out IF the bottleneck is somewhere elseā€¦
I use
iotop: to find out which process stress more the INPUT/OUTPUT
atop: memorize an history of the machine load and can be used to investigate in the past
dstat -v 5 : to monitor the overall performance
mpstat 5, tiptop, slabtop, saidar, statgrab, glances, lslocks, powertop, ā€¦

Also I suggest to try to transfer a big file from the same server where is installed NC but without pass through NC
(only webserver directly) to find out IF there is any difference.
If there is no difference, poor speed is not NC faults.

Fast storage is mandatory, and even better is to physically separate O.S. from Data dir, and DB dir.

My 2 cent: Iā€™m running the current NC release (12.0.3) on an Odroid XU4 with 2GB ram for 3 users and some 200GB of data.
Mysql-default was: innodb_buffer_pool_size = 128M, resulting in like 90% buffer usage and 50% key efficency accoring to MySQL-Workbench Monitoring.
Just changed innodb_buffer_pool_size = 256M, which leads to 43% buffer usage and 87% key efficency.

i run nc 12.0.3 on debian 9.2, too (with apache, php7, mariadb). i confiugured opcache, but i ā€œfeltā€ (really did) the greatest performance increase after adjusting the (maria-)db-settings.
also, what really speeds up the gallery is preview-generation (which takes time, though, around 150-180minutes/night via cron).
i posted everything on these forums, but i just found that the tiltes are misleading:
https://help.nextcloud.com/t/slow-performance-generally-with-nc-12/21390/4
https://help.nextcloud.com/t/debian-9-1-admin-warnings-despite-opcache-running-clarification-on-settings/20881
GOOD LUCK!

Jep, I had ~the same. innodb_buffer_pool_size just needs to be a as large as the database and good ;).

Btw. there is a newer version of MySQLTuner available than offered in common repos, showing some more information and recommendations: https://github.com/major/MySQLTuner-perl

At last is useful to get a general usage overview.

I had a very slow windows client, with important ā€œwaitingā€ delays between consecutive downloads. I have digging in several recent posts during 3 days without results. Then I read that old post from Michalng, and added the conf. into my.cnf i.e

  • innodb_buffer_pool_size = 512M
  • innodb_buffer_pool_instance = 1
  • innodb_flush_log_at_trx_commit = 2
  • innodb_log_buffer_size = 32M
  • innodb_max_dirty_pages_pct = 90

Then suddenly, my client boosted up.

Thank you

1 Like
  • If all elements of stack are co-hosted on same hardware or VM, utilize UNIX sockets for redis and database connections to/from app server (can even be done in Docker)
  • Utilize glibc rather than musl based OSes/containers (i.e. not Alpine) - Nextcloud performance boost by using Redis via UNIX sockets instead of TCP Ā· GitHub (also has glibc vs musl benchmark data)
  • Pay attention to which WebDAV client you utilize - performance varies dramatically even among the well known ones and for different use cases - WebDAV Client Small Files Performance Testing Ā· GitHub
  • Understand the impact of using TLS/HTTPS on non-AES-NI accelerated CPUs (like Raspberry Pis unfortunately). Consider using HTTP+WireGuard (which uses ChaCha20-Poly1305 which is fast even on lower end CPUs) or making sure your servers+clients support (and have enabled and prefer) ChaCha20-Poly1305.