tflidd
November 15, 2016, 10:40am
7
The difference for a lot of small files is that your database needs to create a lot of entries. You can optimize your mysql settings with help of scripts like tuning-primer.sh. I did some test in the past where I found some parameters that can increase the speed considerably, part is the innodb_flush-stuff that @victorbw mentioned (however there is a risk that some queries will be lost in case your server crashes):
opened 12:22AM - 06 Dec 15 UTC
closed 04:54PM - 06 Apr 17 UTC
enhancement
discussion
performance
Upload of many small files is very slow. I tested on a raspbery pi 2 which is kn… own to have a bad i/o-performance. During uploads, mysql-operation create an important io-wait on the system. It was worse with journaling of ext4, so I disabled it (it's a test system).
Uploading a folder with 1180 files (18 MB) via the oc-client takes about 20-30 minutes (pictures, a lot of text files, take any php code of a project), but the number of sql-queries is extremely high (>50 queries per uploaded file!):
| command | number |
| --- | --- |
| select | 49,681 |
| update | 11,523 |
| insert select | 2,487 |
| set option | 1,526 |
| delete | 511 |
| insert | 423 |
| begin | 17 |
| commit | 17 |
For upload, a smaller number of queries can be obtained when files are uploaded in an external folder via ftp (takes a few seconds) then this folder is included into owncloud, the scanning process takes a few minutes (it's hard to tell, 5 minutes?) and the number of queries (some queries are due to handling the web-interface):
| command | number |
| --- | --- |
| select | 11,054 |
| update | 3,040 |
| insert select | 3,004 |
| set option | 425 |
| delete | 138 |
I remember you worked on the deletion process for OC 8.0, this scales much better: 3000 files take 3500 update and 1400 select queries. Perhaps there is a similar improvement possible for upload operations ;-)
System: RPi2, Debian Jessie
Webserver: nginx/php-fpm
Database: mariadb
Owncloud 8.2.1
Apps enabled:
- activity: 2.1.3
- calendar: 0.8.1
- contacts: 0.5.0.0
- files: 1.2.0
- files_external: 0.3.0
- files_pdfviewer: 0.7
- files_sharing: 0.7.0
- files_texteditor: 2.0
- files_trashbin: 0.7.0
- files_versions: 1.1.0
- files_videoviewer: 0.1.3
- firstrunwizard: 1.1
- galleryplus: 14.3.0
- provisioning_api: 0.3.0
- templateeditor: 0.1
I haven't tested with a pure webdav-client yet.
I should run this tests again with the current version of NC.