Files:scan performance

ℹ️ Support
1

Nextcloud version : 16.0.5
Operating system and version : CentOS 7
Apache or nginx version : Apache 2.4.6
PHP version : PHP 7.2.24

Intel Xeon E3 1225v2
4 cores / 4 threads
3.2 GHz
16Go DDR3
2x 2 To SATA RAID 1

config.php :

  'dbtype' => 'mysql',
  'version' => '16.0.5.1',
  'mysql.utf8mb4' => true,
  'loglevel' => 2,
'memcache.local' => '\OC\Memcache\Redis',
'memcache.distributed' => '\OC\Memcache\Redis',
'redis' => [
     'host'     => '/var/run/redis/redis.sock',
     'port'     => 0,
     'dbindex'  => 0,
     'password' => 'xxx',
     'timeout'  => 1.5,
],

When i do a files:scan, it seems to me it is very long and i’m wondering if this is coming from a lack of performance from nextcloud or if i can optimize my software configuration.

Example :
Scanning a user folder (all files were put there by ftp so they were all unknown from nextcloud)

sudo -u apache php /home/www/nextcloud/occ files:scan --path=“dca”
±--------±-------±-------------+
| Folders | Files | Elapsed time |
±--------±-------±-------------+
| 41892 | 454701 | 16:33:22 |
±--------±-------±-------------+

For instance, it scan one file every 131ms or 7.6 files per second.

What took the time ? What nextcloud do during scan ? Analysing content of each file ? or just putting their path in database ?
Should i disable lock file system (if this is what took time) ?

2

There are lots of database requests when scanning files. Check both redis and mysql cpu and disk usage. Mysql could be a bottleneck especially on spinning disks compared to SSD. What filesystem are you using?

3

Filesystem is ext4, not SSD
Only Mysql seems very active : Mariadb is actually using 4.3% of RAM and about 20% of CPU

4

SSD means Solid State Disk. It’s a flash memory compared to a normal HDD which has a rotating disk with moving heads. SSD’s are ten or hundreds of times faster than HDD’s for database use.

EXT4 is the filesystem on the disk which stores your files. There are many guides online that explains how to tune your performance. One simple tip is to mount your disk with the noatime,nodirtime options.

I think if you look at iotop -o you’ll see that your disk is fully saturated. Look at the IO % column.

20191106_160941
20191106_160941.jpg1080×315 143 KB

Perhaps you should look to see if you can tweak your MySQL/InnoDB settings to improve your performance.

5

Sorry, i was not clear enough.

Filesystem is ext4, and this is not a SSD, it is a HDD

Thank you, i’ll check optimisation of disk but i doubt it is the real reason of slowness. A full copy (with cp) of these files only last a few minutes, not 16 hours.

Does anybody knows what the scan do with each file ? Generating etag and inserting file path should not take so long. There must be something else.