At what point do too many files cause a slowdown and how to battle it?

ℹ️ Support
,
1

Nextcloud version (eg, 29.0.5): 30.0.0
Operating system and version (eg, Ubuntu 24.04): Debian 24.11
Apache or nginx version (eg, Apache 2.4.25): 2.4.62
PHP version (eg, 8.3): 8.3
MariaDB : 10.11

The issue you are facing:

Slow loadtimes, when browsing. The initial page takes 14s to load. Subsequently entering folders is similarly slow.

There are 1.2m files at a total size of 2.5TB, and the nextclouddb from MariaDB is 1.3GB.

The OS, Nextcloud without the data and MariaDB are installed on an SSD.
The CPU load is 10-15% while loading the website and RAM usage is stable at 1GB, leaving 3GB unused.

Redis and APCu are enabled.

Some noteworthy PHP settings:

Version: *8.3.12*

Memory limit: *1 GB*

Max execution time: *3600*

Upload max size: *2 MB*
 
OPcache Revalidate Frequency: *2*

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => 'myInstanceID',
  'passwordsalt' => 'mySaltedPassword',
  'trusted_domains' =>
  array (
    0 => '192.168.2.126',
    1 => 'myDomain.ddns.net',
    2 => 'localhost',
  ),
  'datadirectory' => '/mnt/NextCloudDrive/nextcloud/data',
  'dbtype' => 'mysql',
  'version' => '30.0.0.14',
  'overwrite.cli.url' => 'https://myDomain.ddns.net/nextcloud',
  'dbname' => 'nextclouddb',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'nextclouduser2341',
  'dbpassword' => 'MyPasswort',
  'installed' => true,
  'maintenance' => false,
  'app_install_overwrite' =>
  array (
    0 => 'apporder',
    1 => 'files_mindmap',
    2 => 'maps',
    3 => 'passwords',
    4 => 'tasks',
    5 => 'files_rightclick',
    6 => 'files_3dmodelviewer',
    7 => 'camerarawpreviews',
    8 => 'transfer',
  ),
  'enable_previews' => true,
  'enabledPreviewProviders' =>
  array (
    0 => 'OC\\Preview\\PNG',
    1 => 'OC\\Preview\\JPEG',
    2 => 'OC\\Preview\\GIF',
    3 => 'OC\\Preview\\HEIC',
    4 => 'OC\\Preview\\BMP',
    5 => 'OC\\Preview\\XBitmap',
    6 => 'OC\\Preview\\MP3',
    16 => 'OC\\Preview\\DNG',
    17 => 'OC\\Preview\\CR2',
  ),
  'preview_imaginary_url' => 'http://127.0.0.1:9191',
  'theme' => '',
  'loglevel' => 2,
  'simpleSignUpLink.shown' => false,
  'defaultapp' => 'files',
  'default_phone_region' => 'DE',
  'memories.exiftool' => '/var/www/nextcloud/apps/memories/bin-ext/exiftool-aarch64-glibc',
  'memories.vod.path' => '/var/www/nextcloud/apps/memories/bin-ext/go-vod-aarch64',
  'memories.vod.ffmpeg' => '/usr/bin/ffmpeg',
  'memories.vod.ffprobe' => '/usr/bin/ffprobe',
  'memories.vod.disable' => false,
  'preview_max_x' => 2048,
  'preview_max_y' => 2048,
  'preview_max_memory' => 248,
  'memories.vod.external' => true,
  'updater.release.channel' => 'stable',
  'maintenance_window_start' => 1,
  'mysql.utf8mb4' => true,
  'memories.db.triggers.fcu' => true,
  'allow_local_remote_servers' => true,
  'secret' => 'uY16ON0r9XiIiaYvPI2sQix1a3MFwFCIfo6jiSEx9njWjEuJ',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
    'host' => '/run/redis/redis-server.sock',
    'port' => 0,
  ),

The output of your Apache2 log in /var/log/apache2/error.log:

[Tue Oct 01 00:00:07.104666 2024] [mpm_prefork:notice] [pid 1077:tid 1077] AH00163: Apache/2.4.62 (Debian) OpenSSL/3.0.14 configured -- resuming normal operations
[Tue Oct 01 00:00:07.104958 2024] [core:notice] [pid 1077:tid 1077] AH00094: Command line: '/usr/sbin/apache2'
[Tue Oct 01 05:20:12.392593 2024] [core:error] [pid 33427:tid 33427] [client 154.213.184.15:47846] AH10244: invalid URI path (/cgi-bin/.%%%%32%%65/.%%%%32%%65/.%%%%32%%65/.%%%%3>[Tue Oct 01 05:22:33.436831 2024] [proxy_fcgi:error] [pid 24594:tid 24594] [client 159.203.80.171:56098] AH01071: Got error 'Primary script unknown'
[Tue Oct 01 05:22:34.325067 2024] [proxy_fcgi:error] [pid 24593:tid 24593] [client 159.203.80.171:56132] AH01071: Got error 'Primary script unknown'
[Tue Oct 01 05:22:34.751955 2024] [proxy_fcgi:error] [pid 25247:tid 25247] [client 159.203.80.171:56144] AH01071: Got error 'Primary script unknown'
[Tue Oct 01 05:48:48.463448 2024] [mpm_prefork:notice] [pid 1077:tid 1077] AH00170: caught SIGWINCH, shutting down gracefully
[Tue Oct 01 05:52:56.214525 2024] [mpm_prefork:notice] [pid 1079:tid 1079] AH00163: Apache/2.4.62 (Debian) OpenSSL/3.0.14 configured -- resuming normal operations
[Tue Oct 01 05:52:56.214983 2024] [core:notice] [pid 1079:tid 1079] AH00094: Command line: '/usr/sbin/apache2'
[Tue Oct 01 07:17:10.794518 2024] [core:error] [pid 6003:tid 6003] [client 91.10.159.193:50684] AH00124: Request exceeded the limit of 10 internal redirects due to probable conf>

Output errors in nextcloud.log in /var/www/ or as admin user in top right menu, filtering for errors. Use a pastebin service if necessary.

	
ValueError hash_hkdf(): Argument #2 ($key) cannot be empty
Error while running background job OCA\DAV\BackgroundJob\EventReminderJob (id: 14, arguments: null) 

 ConnectException cURL error 7: Failed to connect to 192.168.178.178 port 80 after 7 ms: Couldn't connect to server (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://192.168.178.178/ocm-provider/
error while discovering ocm provider

Ideally, I’d get a more snappy user interface, when browsing Nextcloud. Is the number of files an issue? Should I use different caching method? At this point, very few files if ever get changed. I’d reckon, it’s 25 files a week, plus whatever photos i’m taking. So I don’t understand why it can’t get cached to serve the file overview faster when browsing.

2

Hi! FIrst of all, I am sorry you are having issues.

First thing I noticed was this:

Version: 8.3.12

Memory limit: 1 GB

Max execution time: 3600

Upload max size: 2 MB

OPcache Revalidate Frequency: 2

This will limit the amount of memory your PHP app can use, to only 1 GB. With 4GB of ram on the machine, you should consider to raise it to 2GB (you do need some memory to run the host and OS itself, aswell as the Database if on the same host).

Further them 2 MB of maximum allowed size of files that you uploads, are extremely low. Why is that?

With 1GB of memory available for the PHP app, Previews enabled, database on same host and Redis on same aswell, you do challenge your system unreasonably.

When that is said, then because of the Calendar event bug that was introduced with NC 30.0.0, I aswell has seen some longer load times. As the bug is triggered with cron job, which runs every 5 minutes and uses alot of resources as it generates errors, I would say that it might fix itself when 30.0.1 is released. I still believe you should raise that memory limit to at least 2GB though.

1 Like