NC thinks the drive is full, but it's not

Support intro

Sorry to hear you’re facing problems :slightly_frowning_face:

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can :heart:

Nextcloud version: 27.0.1
Operating system and version: Docker 24.0.5 on Ubuntu 22.04.3

The issue you are facing:

I can’t upload any file or create a new one on the webinterface or via the any app. It looks like the harddrive is full, but df -h shows 13G of free space.

Is this the first time you’ve seen this error?: Yes

Steps to replicate it:

  1. Nothing in particular. Just use it for a couple of years, I guess

  2. Upload a file or browse pictures that don’t have a preview yet or create a note or anything that writes to disk

The output of your Nextcloud log in Admin > Logging:
Depending on if the app or the webinterface triggers the issue, the logs look different:

No space left on device at /var/www/html/lib/private/Files/Storage/Local.php#313

OCP\Files\NotPermittedException: Could not create folder "/appdata_octv2v68jkae/preview/4/5/7/d/7/1/8/18324792"

OCP\Files\NotPermittedException: Could not create path "/user/files/Neue Textdatei.md"

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'htaccess.RewriteBase' => '/',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'apps_paths' => 
  array (
    0 => 
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 => 
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'instanceid' => 'qwerty',
  'passwordsalt' => 'abc123',
  'secret' => 'xyz',
  'trusted_domains' => 
  array (
    0 => '192.168.11.3:8080',
    1 => 'cloud.domain.com',
    2 => 'app',
  ),
  'datadirectory' => '/var/www/html/data',
  'dbtype' => 'mysql',
  'version' => '27.0.1.2',
  'overwrite.cli.url' => 'cloud.domain.com',
  'dbname' => 'nextcloud',
  'dbhost' => 'db',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud',
  'dbpassword' => '12345',
  'installed' => true,
  'overwriteprotocol' => 'https',
  'maintenance' => false,
  'app_install_overwrite' => 
  array (
    0 => 'files_external_gdrive',
    1 => 'documentserver_community',
  ),
  'loglevel' => 0,
  'app.mail.verify-tls-peer' => false,
  'theme' => '',
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' => 
  array (
    'host' => 'redis',
    'password' => false,
    'port' => 6379,
  ),
  'default_phone_region' => 'DE',
  'allow_local_remote_servers' => true,
  'mail_from_address' => 'no-reply',
  'mail_smtpmode' => 'smtp',
  'mail_sendmailmode' => 'smtp',
  'mail_domain' => 'domain.com',
  'mail_smtphost' => 'smtp',
);

The output of your Apache/nginx/system log in /var/log/____: nano /var/log/apache2/error.log shows

[ "error.log" is a device file ]

Output errors in nextcloud.log in /var/www/ or as admin user in top right menu, filtering for errors. Use a pastebin service if necessary.

There is no /var/www/nextcloud.log

Additional Information
Excerpt from my docker-compose.yml:

  app:
    image: nextcloud
    links:
      - db
      - redis
    volumes:
      - nextcloud:/var/www/html
      - /mnt/speicher/cloud:/var/www/html/data

/mnt/speicher is a mergerfs drive.

$ df -h
Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              1.3G  4.6M  1.3G   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv  180G  158G   13G  93% /
tmpfs                              6.2G  8.0K  6.2G   1% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
mergerfs                            33T   24T  7.7T  76% /mnt/speicher
/dev/sda2                          2.0G  253M  1.6G  14% /boot
/dev/sdd                           3.6T  3.4T   68G  99% /mnt/harddrive3
/dev/sde                           7.3T  6.9T   53G 100% /mnt/harddrive4
/dev/sdf                           3.6T  3.4T   81G  98% /mnt/harddrive5
/dev/sdc                            11T  2.9T  7.4T  29% /mnt/harddrive2
/dev/sdb                           7.3T  6.9T   50G 100% /mnt/harddrive1
tmpfs                              1.3G  4.0K  1.3G   1% /run/user/1000

Each of those calls will fail if unable to write. It’s up to the filesystem to decide whether that’s possible. Maybe it has something to do with mergerfs’s minfreespace or moveonenospc parameters?

1 Like

That was also my first idea, but /var/www/html/lib/private/Files/Storage/Local.php#313 is not on the mergerfs filesystem. But I also have no idea what that file has to do with the problem at all

That file isn’t the problem, that’s merely referencing the line of code that is triggering one of the error messages you’re seeing. Specifically that line calls file_put_contents()which is just a standard PHP function that writes data to a file.

In your NC configuration, the target of that function call would be a file located in your datadirectory.

You were right, it was a mergerfs issue. The defaults option includes

Category Policy
action epall
create epmfs
search ff

“ep” stands for “existing path” meaning it wants to put the NC files where the other NC files already are which is the one full (i.e. less than 50G free) drive

Read more about it in this issue and the Readme

1 Like