[nextcloud docker] no space left: multiple large ROW files

fuchsi3010 · May 30, 2022, 5:39am

Nextcloud version (eg, 20.0.5): 24.0.1
Operating system and version (eg, Ubuntu 20.04): Ubuntu 20.04.4 LTS for Host System
Docker Image Version:

[18:08:07] ~ $ docker image ls
REPOSITORY              TAG       IMAGE ID       CREATED         SIZE
redis                   alpine    c3ea2db12504   2 days ago      28.4MB
mysql                   5.7       7ccb2b64de3b   4 days ago      462MB
mariadb                 10        784190069700   4 days ago      389MB
mariadb                 latest    784190069700   4 days ago      389MB
nextcloud               latest    f42d12dbb30f   7 days ago      880MB
mariadb                 10.5      9a0e59bdf6f0   6 months ago    408MB

The issue you are facing:
i have 2 users with about 30GB files all-in-all. normally, about 60GB of my 100GB quota on my virtual server are in use.
last week i got a ‘no space left on device’ error. i tracked the problem down to the ‘db’ directory in my nextcloud installation, which was full of ‘ROW.00****’ files with increasing file size. i moved all but the 3 ‘most recent’ ROW files out of the directory and started the container, and everything worked great - nothing was missing (files, calendar, …)
1 week later the same problem has cropped up.

the db folder looks like this:

ls db/
total 37G
-rwxrwxrwx 1 systemd-coredump www-data         624K May 25 21:04 aria_log.00000001
-rwxrwxrwx 1 systemd-coredump www-data           52 May 25 21:04 aria_log_control
drwxrwxrwx 5 systemd-coredump www-data          280 Mar 14  2021 _data
-rw-rw---- 1 systemd-coredump systemd-coredump  16K May 19 12:33 ddl_recovery-backup.log
-rw-rw---- 1 systemd-coredump systemd-coredump    9 May 24 04:01 ddl_recovery.log
-rw-rw---- 1 systemd-coredump systemd-coredump  16K May 24 04:01 ib_buffer_pool
-rwxrwxrwx 1 systemd-coredump www-data          76M May 25 22:56 ibdata1
-rw-rw---- 1 systemd-coredump systemd-coredump  96M May 25 23:05 ib_logfile0
-rw-rw---- 1 systemd-coredump systemd-coredump  12M May 24 05:35 ibtmp1
-rwxrwxrwx 1 systemd-coredump www-data            0 Oct 18  2020 multi-master.info
drwxrwxrwx 2 systemd-coredump www-data         4.0K Jul 14  2021 mysql
-rw-r--r-- 1 systemd-coredump root               15 Jul 14  2021 mysql_upgrade_info
drwxrwxrwx 2 systemd-coredump www-data          16K May 19 12:56 nextcloud
drwx------ 2 systemd-coredump systemd-coredump   20 Jul 14  2021 performance_schema
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 21 03:15 ROW.000432
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 21 06:57 ROW.000433
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 21 10:12 ROW.000434
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 21 13:26 ROW.000435
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 21 17:09 ROW.000436
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 21 20:53 ROW.000437
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 21 23:55 ROW.000438
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 22 03:00 ROW.000439
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 22 06:58 ROW.000440
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 22 10:13 ROW.000441
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 22 13:22 ROW.000442
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 22 16:20 ROW.000443
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 22 19:18 ROW.000444
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 22 22:35 ROW.000445
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 23 01:48 ROW.000446
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 23 05:02 ROW.000447
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 23 08:12 ROW.000448
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 23 11:29 ROW.000449
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 23 14:40 ROW.000450
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 23 17:27 ROW.000451
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 23 21:05 ROW.000452
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 24 03:36 ROW.000454
-rw-rw---- 1 systemd-coredump systemd-coredump 109M May 24 04:01 ROW.000455
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 24 07:20 ROW.000456
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 24 10:25 ROW.000457
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 24 13:24 ROW.000458
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 24 16:42 ROW.000459
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 24 19:15 ROW.000460
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 24 22:20 ROW.000461
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 25 01:28 ROW.000462
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 25 04:45 ROW.000463
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 25 07:57 ROW.000464
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 25 11:04 ROW.000465
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 25 14:37 ROW.000466
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 25 17:32 ROW.000467
-rw-rw---- 1 systemd-coredump systemd-coredump 1.1G May 25 20:37 ROW.000468
-rw-rw---- 1 systemd-coredump systemd-coredump 810M May 25 23:20 ROW.000469
-rwxrwxrwx 1 systemd-coredump www-data         1.3K May 25 20:37 ROW.index
drwx------ 2 systemd-coredump systemd-coredump 8.0K Jul 14  2021 sys

Is this the first time you’ve seen this error? (Y/N): no
this also happened a few months back, but has been stable since then.

Steps to replicate it: i did not find any other reports of this problem online and i’m not sure what causes it.

docker-compose.yml :

[18:14:17] ~/apps/nextcloud $ cat docker-compose.yml
---
version: '3.3'

services:
  nextcloud-db:
    image: mariadb
    container_name: nextcloud-db
    command: --transaction-isolation=READ-COMMITTED --log-bin=ROW --skip-innodb-read-only-compressed
    restart: unless-stopped
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - /home/##REMOVED##/apps/nextcloud/db:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=##REMOVED##
      - MYSQL_PASSWORD=##REMOVED##
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
      - MYSQL_INITDB_SKIP_TZINFO=1 # Behebt die bekannten Startprobleme der Datenbank

  nextcloud-redis:
    image: redis:alpine
    container_name: nextcloud-redis
    hostname: nextcloud-redis
    networks:
        - default
    restart: unless-stopped
    command: redis-server --requirepass ##REMOVED## # Redis Passwort wählen

  nextcloud-app:
    image: nextcloud
    container_name: nextcloud-app
    restart: always
    depends_on:
      - nextcloud-db
      - nextcloud-redis
    environment:
        REDIS_HOST: nextcloud-redis
        REDIS_HOST_PASSWORD: ##REMOVED## # Redis Passwort von oben
    volumes:
      - /home/##REMOVED##/apps/nextcloud/app:/var/www/html
      - /home/##REMOVED##/apps/nextcloud/data:/var/www/html/data
    links:
      - nextcloud-db
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.nextcloud.entrypoints=web-secure"
      - "traefik.http.routers.nextcloud.rule=Host(`##REMOVED##`)" #Domain anpassen
      - "traefik.http.routers.nextcloud.tls.certresolver=default"
      - "traefik.http.services.nextcloud.loadbalancer.server.port=80"
      - "traefik.http.middlewares.nextcloud-dav.replacepathregex.regex=^/.well-known/ca(l|rd)dav"
      - "traefik.http.middlewares.nextcloud-dav.replacepathregex.replacement=/remote.php/dav/"
    networks:
      - traefik_proxy0
      - default

  nextcloud-cron:
    image: nextcloud
    restart: always
    volumes:
      - /home/##REMOVED##/apps/nextcloud/app:/var/www/html
      - /home/##REMOVED##/apps/nextcloud/data:/var/www/html/data
    entrypoint: /cron.sh
    depends_on:
      - nextcloud-db
      - nextcloud-redis

networks:
  traefik_proxy0:
    external:
      name: traefik_proxy0
  default:
    driver: bridge

The output of your Nextcloud log in Admin > Logging:

 bunch of RSS feed 404s
Error	Could not find resource js/maintenance.js to load 
2022-05-26T17:00:04+0200
Error	Could not find resource js/maintenance.js to load	
2022-05-26T16:59:59+0200
Error	Could not find resource js/maintenance.js to load	
2022-05-26T16:59:59+0200
Error	Could not find resource js/maintenance.js to load	
2022-05-26T16:59:54+0200
Error	Could not find resource js/maintenance.js to load	
2022-05-26T16:59:54+0200

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

➜  config cat config.php
<?php
$CONFIG = array (
  'htaccess.RewriteBase' => '/',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'apps_paths' =>
  array (
    0 =>
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 =>
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
    'host' => 'nextcloud-redis',
    'password' => '##REMOVED##',
    'port' => 6379,
  ),
  'instanceid' =>##REMOVED##',
  'passwordsalt' => ##REMOVED##',
  'secret' => ##REMOVED##',
  'trusted_domains' =>
  array (
    0 => ##REMOVED##',
  ),
  'trusted_proxies' =>
  array (
    0 => '172.20.0.3',
  ),
  'datadirectory' => '/var/www/html/data',
  'dbtype' => 'mysql',
  'version' => '24.0.1.1',
  'overwrite.cli.url' => '##REMOVED##',
  'dbname' => 'nextcloud',
  'dbhost' => 'nextcloud-db',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud',
  'dbpassword' => '##REMOVED##',
  'installed' => true,
  'overwriteprotocol' => 'https',
  'maintenance' => false,
  'app_install_overwrite' =>
  array (
    0 => 'cookbook',
    1 => 'tasks',
    2 => 'apporder',
  ),
  'loglevel' => 2,
);

The output of your Apache/nginx/system log in /var/log/____:

/var/log/apache2/error.log is symlinked to /dev/stderr

tomcatcw1980 · July 5, 2022, 9:24pm

Hi there,

I have the exact same issues. Did you ever find a solution?

Greetings.

fuchsi3010 · July 6, 2022, 7:19am

hi,
i used this docker-compose file:

github.com

cbirkenbeul/docker-homelab/blob/master/compose-files-traefik-predefined/nextcloud/docker-compose.yaml

---
version: '3.3'

services:
  nextcloud-db:
    image: mariadb:10.8
    container_name: nextcloud-db
    command: --transaction-isolation=READ-COMMITTED --log-bin=ROW
    restart: always
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - /var/docker/nextcloud/database:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD}
      - MYSQL_DATABASE=${MYSQL_DATABASE}
      - MYSQL_USER=${MYSQL_USER}
      - MYSQL_INITDB_SKIP_TZINFO=1  # Behebt die bekannten Startprobleme der Datenbank

This file has been truncated. show original

which has the command: --transaction-isolation=READ-COMMITTED --log-bin=ROW set.

i removed that whole line and that stopped the issue.
i dont know why the command is in that docker-compose file, and it kind of worries me that i just deleted it. i assume it was there for good reason?

until now i haven’t had any data issues (calendar entries, tasks, …)

i plan on moving to this compose file once i get time: https://www.reddit.com/r/selfhosted/comments/vf6jeg/i_used_unix_sockets_to_improve_the_performance_of/ but the backup app seems to not work on the latest version of nextcloud, i wanted to use that to move the data

i hope this helps somewhat.
i also dug into the mariadb config files, it this post doesnt resolve your issues i can go back and have a look at what i did!

tomcatcw1980 · July 6, 2022, 7:36pm

Hi Daniel,

I have found a solution for myself, thanks to your hint. I added this flag to the command (see MySQL :: MySQL 8.0 Reference Manual :: 17.1.6.4 Binary Logging Options and Variables ). The whole command for me is now

command: --transaction-isolation=READ-COMMITTED --log-bin=ROW --innodb-file-per-table=1 --skip-innodb-read-only-compressed --binlog-expire-logs-seconds=432000

Now the duration of these logs is limited to 5 days (5 x 24 x 60 x 60), instead of 30 days by default before.

Then restarted via docker-compose up -d --force-recreate and lo and behold I just freed up 140 GB.

Greetings.

fuchsi3010 · July 30, 2022, 8:47am

hi,

i have not yet found any errors without the --log-bin=ROW command.
i do prefer 0 bytes of ROW files over a (semi) fixed amount…

is there any reason why you want to keep the logging in there?
greetings, daniel

Jodl_Dodl · September 27, 2022, 5:50am

That would interest me too! Whats the point of this command? It generates useless 1GB ROW files until there is no space left?!