My Database is 110GB with just one user. Can this be right?

Nextcloud version : 29.0.3
Operating system and version : Ubuntu 22.04 with Docker
Apache or nginx version: Apache 2.4
PHP version: 8.2.20

The issue you are facing: My database is 110 GB big. The VM is regularly running out of space and I have to resize it.

Some info, maybe it’s useful:

  • I’m using external SFTP storage
  • The setup is a couple of years old
  • The size of the database is constantly growing. I have to enlarge the VM every couple of month by like 20 GB or so
  • Some apps i’m using are: Deck, Mail, Notes, Onlyoffice
  • I already ran occ files:cleanup and occ files:scan --all, but it made no difference
  • i already emptied the trash bin
  • There’s just one user on my instance
  • In the admin overview I often get the warning that some indices are missing. I then run occ db:add-missing-indices, but after couple of weeks I get the message again
  • A couple of times I ran completely out of storage. Maybe that corrupted my database?

Why is the database so big? Can I clean it up or maybe deactivate a feature that produces so much data?

Steps to replicate it:

  1. Just use it for a long time

The output of your Nextcloud log in Admin > Logging:

nothing

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

$CONFIG = array (
  'htaccess.RewriteBase' => '/',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'apps_paths' =>
  array (
    0 =>
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 =>
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'instanceid' => 'xxx',
  'passwordsalt' => 'xxx',
  'secret' => 'xxx',
  'trusted_domains' =>
  array (
    0 => '192.168.11.3:8080',
    1 => 'cloud.domain.com',
    2 => 'app',
  ),
  'datadirectory' => '/var/www/html/data',
  'dbtype' => 'mysql',
  'version' => '29.0.3',
  'overwrite.cli.url' => 'https://cloud.domain.com',
  'dbname' => 'nextcloud',
  'dbhost' => 'db',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud',
  'dbpassword' => 'xxx',
  'installed' => true,
  'overwriteprotocol' => 'https',
  'maintenance' => false,
  'app_install_overwrite' =>
  array (
    0 => 'files_external_gdrive',
    1 => 'documentserver_community',
  ),
  'loglevel' => 0,
  'app.mail.verify-tls-peer' => false,
  'theme' => '',
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
    'host' => 'redis',
    'password' => 'xxx',
    'port' => 6379,
  ),
  'default_phone_region' => 'DE',
  'allow_local_remote_servers' => true,
  'mail_from_address' => 'no-reply',
  'mail_smtpmode' => 'smtp',
  'mail_sendmailmode' => 'smtp',
  'mail_domain' => 'domain.com',
  'mail_smtphost' => 'smtp',
  'maintenance_window_start' => 1,
);

Are you really talking about your SQL database or do you mean the data folder on the disk?

What about the folder file_versions in the data folder?

1 Like

my DB size is 2.6GB according to /settings/admin/serverinfo and the size of my DB directory is 2.8GB with NC listing ~2mio files in serverinfo output (nc29 with postgres 16). so your DB seems to be far bigger. This doesn’t mean something is broken - if you frequently update many files and keep excessive versions history this might be OK.

In the past I remember transaction logs of MariaDB filled up my disk and I had to adjust them (you’ll find many mariadb.bin.000xx files) - google for “mariadb binary log” - but better first check what is using the space first.

Can you also check, if there is one specific table that is unususally larger?

On vanilla setups, it’s mostly the oc_filecache table that is the biggest. Some apps might add other tables that might be larger, and in case there is a bug in an apps or something, that could explain it. Or we had issues with people where a lot of sessions were created, and the sessions table increased by a lot…

1 Like

Yes, actually the SQL Database.

file_versions has 89GB

I never configured versioning. Where can I see my settings / change them?

Yes, oc_filecache.ibm is by far the biggest file with 122G

each file version is basically another file stored in oc_filecache (as well).

https://docs.nextcloud.com/server/stable/admin_manual/configuration_server/config_sample_php_parameters.html#file-versions

but before you are going to adjust versions settings I would first review why do you have so many versions. this could happen if you sync some application creating many new or additional files often. maybe a look into Activity app helps already

1 Like

You can significantly shrink your database size when you create a dump and restore your database back from that dump.

You should exactly know what you do:

→ how to create a database dump ←

→ how to restore a database dump ←

The effect is, that you create your database with all data but defragmented and cleaned from data trash.
A nice side effect is a slight better performance.


Much and good luck,
ernolf

2 Likes

This only works if the database does not really contain 110 GB of data. So first check what’s in there

1 Like

Yes! FullAck!

@maeries:

Could you look the result of this SQL Queries:

List tables sorted by size:

SELECT table_schema AS `Database`, 
       table_name AS `Table`, 
       ROUND((data_length + index_length) / 1024 / 1024, 2) AS `Size (MB)` 
FROM information_schema.tables 
ORDER BY `Size (MB)` DESC;

With this query, you can see if your innoDB tables are fragmented. Tables with a large “data_free_MB” Value can be optimized:

SELECT table_schema, 
       table_name, 
       engine, 
       round(data_length/1024/1024) as data_length_MB, 
       round(index_length/1024/1024) as index_length_MB, 
       round(data_free/1024/1024) as data_free_MB 
FROM information_schema.tables 
WHERE data_free > 0 
ORDER BY data_free_MB DESC;

you can use the following command to defragment such a table (only for InnoDB):
(“your_table” is only a placeholder and must be replaced by the real table name)

OPTIMIZE TABLE your_table;

I hope this helps


Much and good luck,
ernolf

Finally found some time to work on this again. There seems to be a bit of free space on the database:

MariaDB [(none)]> SELECT table_schema, 
    ->        table_name, 
    ->        engine, 
    ->        round(data_length/1024/1024) as data_length_MB, 
    ->        round(index_length/1024/1024) as index_length_MB, 
    ->        round(data_free/1024/1024) as data_free_MB 
    -> FROM information_schema.tables 
    -> WHERE data_free > 0 
    -> ORDER BY data_free_MB DESC;
+--------------+-------------------------+--------+----------------+-----------------+--------------+
| table_schema | table_name              | engine | data_length_MB | index_length_MB | data_free_MB |
+--------------+-------------------------+--------+----------------+-----------------+--------------+
| nextcloud    | oc_filecache            | InnoDB |          62722 |           50066 |         8690 |
| nextcloud    | oc_files_metadata       | InnoDB |            135 |               4 |           22 |
| nextcloud    | oc_activity             | InnoDB |             15 |              10 |           15 |
| nextcloud    | oc_files_metadata_index | InnoDB |              6 |              16 |            5 |
| nextcloud    | oc_jobs                 | InnoDB |              0 |               0 |            5 |
| nextcloud    | oc_file_metadata        | InnoDB |              2 |               0 |            4 |
| nextcloud    | oc_files_versions       | InnoDB |              6 |               4 |            4 |
| nextcloud    | oc_files_trash          | InnoDB |              1 |               0 |            3 |
| nextcloud    | oc_authtoken            | InnoDB |              1 |               0 |            3 |
| nextcloud    | oc_mail_messages        | InnoDB |              3 |               2 |            3 |
| nextcloud    | oc_calendarobjects      | InnoDB |              1 |               0 |            3 |
| nextcloud    | oc_mail_classifiers     | InnoDB |              0 |               0 |            3 |
| nextcloud    | oc_mail_recipients      | InnoDB |              2 |               2 |            2 |
| nextcloud    | oc_cards                | InnoDB |              2 |               0 |            2 |
| nextcloud    | oc_filecache_extended   | InnoDB |              2 |               2 |            2 |
+--------------+-------------------------+--------+----------------+-----------------+--------------+
15 rows in set (0.236 sec)

But optimizing it

MariaDB [(none)]> USE nextcloud;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [nextcloud]> OPTIMIZE oc_filecache;

does not change that.

I also had a loot into the activities, but there was nothing suspicious. I have some notes that get updated often, but they are very small.

the size of the files doesn’t matter for the size of the database… each file takes the same amount of space in there… it only metadata like dates, access rights etc.

now once we know the size is effectively used by the database (62gb for the table and 50GB for index). Next step would be to check if you have some extra data in this table. I would start with " select count(*) from oc_filecache" checking if the number of the items roughly matches the number of files stored in your data directory. If yes nothing is wrong if you find big difference you have to analyze the contents of the table further… there I would look at storage column first - I remember somebody reported issues with external storage when items remain in the DB if one disconnect/remount other storages e.g. like here:

UPDATE: maybe little harder than my first idea oc_filecache stores previews as well which live outside of data folder

here is a small reverse engineering of the table and versions mechanics.

Another reference, maybe it provides some useful insights as well: