High IO wait because mysql read IOPs

Support intro

Sorry to hear you’re facing problems :slightly_frowning_face:

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can :heart:

Nextcloud version (eg, 29.0.5): 27.1.4
Operating system and version (eg, Ubuntu 29.04): Ubuntu 22.04
Apache or nginx version (eg, Apache 2.4.25): 2.4.52
PHP version (eg, 8.3): 8.1

The issue you are facing:

Is this the first time you’ve seen this error? (Y/N): y

I have an NC instance for calendar only and Mariadb is eating all the disk IOs,
most of the iops are read ones using 200-300MB/s all the time causing big IO wait.

Looking at Mariadb i see a few process running that looks like they are looping example:

| 4935608 | ncuser | localhost | nextdb | Query   |    0 | Sending data | SELECT `uri`, `calendardata`, `lastmodified`, `etag`, `size` FROM `oc_schedulingobjects` WHERE `principaluri` = 'principals/users/USER1'
| 4935621 | ncuser | localhost | nextdb | Query   |    0 | Sending data | SELECT `uri`, `calendardata`, `lastmodified`, `etag`, `size` FROM `oc_schedulingobjects` WHERE `principaluri` = 'principals/users/USER2'
| 4935750 | ncuser | localhost | nextdb | Query   |    0 | Sending data | SELECT `uri`, `calendardata`, `lastmodified`, `etag`, `size` FROM `oc_schedulingobjects` WHERE `principaluri` = 'principals/users/USER3'
| 4936034 | ncuser | localhost | nextdb | Query   |    1 | Sending data | SELECT `uri`, `calendardata`, `lastmodified`, `etag`, `size` FROM `oc_schedulingobjects` WHERE `principaluri` = 'principals/users/USER4'
| 4936179 | ncuser | localhost | nextdb | Query   |    0 | Sending data | SELECT `uri`, `calendardata`, `lastmodified`, `etag`, `size` FROM `oc_schedulingobjects` WHERE `principaluri` = 'principals/users/USER5'
| 4936457 | ncuser | localhost | nextdb | Query   |    0 | Sending data | SELECT `uri`, `calendardata`, `lastmodified`, `etag`, `size` FROM `oc_schedulingobjects` WHERE `principaluri` = 'principals/users/USER6'
| 4936575 | ncuser | localhost | nextdb | Query   |    0 | Sending data | SELECT `uri`, `calendardata`, `lastmodified`, `etag`, `size` FROM `oc_schedulingobjects` WHERE `principaluri` = 'principals/users/USER7'

They are users using MacOs as client for the calendar(Caldav).
I don’t see this queries with user using thunderbird for example…

if i re-run all the time ‘show full processlist;’ in mysql always i see this users
and those queries.

Not sure but if something related with MacOs and caldav inbox(not sure exactly
what is for) i add apache logs.

LDAP backend but the clients are using app tokens for auth.

Don’t know what else can i try o how to continue debugging this.
Any help or ideas to discover what is going on is appreciated.

The output of your Nextcloud log in Admin > Logging:

Nothing related about this.

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => 'SECRET',
  'skeletondirectory' => '',
  'defaultapp' => 'calendar',
  'passwordsalt' => 'SECRET',
  'secret' => 'SECRET',  
  'trusted_domains' =>
  array (
    0 => 'REMOVED.TLD',
  ),
  'datadirectory' => '/var/www/nextcloud/data',
  'dbtype' => 'mysql',
  'version' => '27.1.4.1',
  'overwrite.cli.url' => 'https://REMOVED.TLD',
  'dbname' => 'nextdb',
  'dbhost' => 'localhost',
  'dbport' => '',  
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'ncuser',
  'dbpassword' => 'SECRET',
  'installed' => true,
  'ldapProviderFactory' => 'OCA\\User_LDAP\\LDAPProviderFactory',
  'mail_from_address' => 'noreply',
  'mail_smtpmode' => 'smtp',
  'mail_sendmailmode' => 'smtp',
  'mail_domain' => 'REMOVED:TLD',
  'mail_smtphost' => 'localhost',
  'mail_smtpport' => '25',
  'maintenance' => false, 
  'session_lifetime' => 14400,
  'remember_login_cookie_lifetime' => 172800,
  'token_auth_enforced' => true,
  'loglevel' => 2,
);

The output of your Apache/nginx/system log in /var/log/____:

RELATED lines i see often are similar like this one maybe with more bytes returning:

172.19.204.98 - USER5 [02/Jul/2024:19:10:36 +0000] "PROPFIND /remote.php/dav/calendars/USER5/inbox/ HTTP/1.1" 207 112937 "-" "macOS/14.4.1 (23E224) dataaccessd/1.0"
172.19.204.98 - USER5 [02/Jul/2024:19:10:39 +0000] "REPORT /remote.php/dav/calendars/USER5/inbox/ HTTP/1.1" 207 546884 "-" "macOS/14.4.1 (23E224) dataaccessd/1.0"

Output errors in nextcloud.log in /var/www/ or as admin user in top right menu, filtering for errors. Use a pastebin service if necessary.

NOTHING RELATED

Before anyone troubleshoots this much you’ll likely save everyone (including yourself) some time and hassle by, at a minimum, upgrading to the final maintenance release of v27 (27.1.11) before it went end of support. Your v27 is very very out-of-date: Nextcloud server changelog

Ideally you bump up to a still supported major release as well (e.g. v28), but that’s at your discretion since v27.1.11 just hit end-of-support.

Hi Jtr, im waiting some fixes, because there is a regression with the calendar email notifications in 27.1.5 and newers. This one: [stable27] fix(scheduling): don't send iMIP emails to rooms / resources by backportbot-nextcloud[bot] · Pull Request #41315 · nextcloud/server · GitHub and others… too it will take some time test the new version and search for regression like this one.

I attach a screenshot as example of iotop showing mariadb using lots of reads.