[SOLVED] Not enough free space ?!

ℹ️ Support 📦 Appliances (Docker, Snappy, VM, NCP, AIO)
,
1

I desperately need help figuring out how to debug this strange phenomenon.

The issue:
Suddenly, out of nowhere nextcloud complains “No free space” when trying to put whatever tiny kb of file into the nextcloud file folders. The disk has plenty of GBs left, and it is possible to create new md file and directory from within, and move files, but uploading is a no no.

The user testing has administrator rights, but no other user can save anything either.

Even rsync’ing /var/www/html/data to a brand new 240GB ssd disk (happily formatted to BTRFS) doesn’t change a thing.

Manipulation users quota from unlimited to any limit or vise versa doesn’t change a thing.

I believe it might have started when adding external disks (beyond the pre existing CIFS/SMB mount) using sftp as well as webdav. I have now deleted all those mounts, but still… no free disk space…

As I have no clue why it suddenly believes the disk is full I have no idea how to replicate it either.

and worse of all, no matter the debug level set there are no log rows mentioning any particular problem.

Nextcloud version: 19.0.4 (nextcloud/stable)
docker container: Debian 10
host OS: Debian 9
Apache version within the container: Apache/2.4.38
PHP version within the container 7.4.11
reverse proxy: separate docker container running nginx

2

If you have updated the Nextcloud perhaps you can delete something in …/nextcloud/data/updater-****/backups

Sorry i do not use docker. Perhaps it is not possible.

3

Thank you for your thoughts.
Your suggestion seems to assume the disk is full, and it has 33gb+ left…

Anyways, there is neither a directory nor any file starting with the name updater-* where suggested. The only ones find are a few png files and I presume they will not do much harm.

4

I think it does not help. But search big files.

find /path/to/your/nextcloud -size +100M

5

Well, I do have a few large files, but as far as I know nextcloud doesn’t have any limit on how big the files are to be, right?

this is from within the docker container:

‘’‘root@368b2b4b886f:/var/www/html# df -hl
Filesystem Size Used Avail Use% Mounted on
overlay 110G 71G 33G 69% /
tmpfs 64M 0 64M 0% /dev
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
shm 64M 0 64M 0% /dev/shm
/dev/sda1 110G 71G 33G 69% /etc/hosts
tmpfs 3.9G 0 3.9G 0% /proc/asound
tmpfs 3.9G 0 3.9G 0% /proc/acpi
tmpfs 3.9G 0 3.9G 0% /sys/firmware ‘’’

I deleted another 4.9GB of big files, but still “Not enough free space”

The question is, where/what is calculating the free space coming to the conclusion there is none?

6

Am I assuming correctly this is a pure calculation error, which is the very reason it doesn’t turn up in any logs?

Could it be that adding remote storages containing terrabytes of storage has lead to nextcloud store this in the DB and now when it calculates something overflows?

7

Now this is getting bizarre…

So i created a second nextcloud service having its own storage and database (using the same separate MariaDB server and nextcloud/latest(20.0.1)). After fresh install, trying to upload a file… “Not enough free space”.

So i take another computer, install docker, do a fresh docker install (using nextcloud/latest (v20.0.1)) with new database (using the same MariaDB server)… “Not enough free space”.

So I ditch this docker container and erase all data, do a fresh docker nextcloud choosing sqlite db… “Not enough free space”

How on earth can a new install on a different computer still give the same problem?

8

Yesterday I saw the same message… But only with one user, another user had no problems uploading files. No quotas enabled, plenty of space left.

I’m also using the official Docker images.

9

Finally I have succeeded to find an error message that give some hint. I don’t know what though…

{“reqId”:“9LF0M1bWf9RfMrDYzMwo”,“level”:3,“time”:“2020-11-05T21:29:47+00:00”,“remoteAddr”:“192.168.101.200”,“user”:“theo”,“app”:“PHP”,“method”:“GET”,“url”:"/index.php/apps/files/",“message”:{“Exception”:“Error”,“Message”:“Undefined offset: 3 at /var/www/html/lib/private/legacy/OC_Helper.php#548”,“Code”:0,“Trace”:[{“file”:"/var/www/html/lib/private/legacy/OC_Helper.php",“line”:548,“function”:“onError”,“class”:“OC\Log\ErrorHandler”,“type”:"::",“args”:[8,“Undefined offset: 3”,"/var/www/html/lib/private/legacy/OC_Helper.php",548,{“path”:"/",“rootInfo”:{“class”:“OC\Files\FileInfo”
“includeExtStorage”:false,“used”:16608273,“quota”:10737418240,“mount”:{“class”:“OC\Files\Mount\MountPoint”
“storage”:{“cache”:null,“scanner”:{“class”:“OC\Files\Cache\Scanner”
“watcher”:null,“propagator”:null,“updater”:null,“class”:“OCA\Files_Trashbin\Storage”
“sourceStorage”:{“cache”:null,“scanner”:{“class”:“OC\Files\Cache\Scanner”
“watcher”:null,“propagator”:null,“updater”:null,“class”:“OCA\Files_Trashbin\Storage”
“free”:10720809967,“total”:10737418240,“relative”:0.15,“ownerId”:“theo”,“ownerDisplayName”:“theo”,“owner”:{“class”:“OC\User\User”}}]
{“file”:"/var/www/html/apps/files/lib/Controller/ViewController.php",“line”:139,“function”:“getStorageInfo”,“class”:“OC_Helper”,“type”:"::",“args”:["/",{“class”:“OC\Files\FileInfo”}]
{“file”:"/var/www/html/apps/files/lib/Controller/ViewController.php",“line”:187,“function”:“getStorageInfo”,“class”:“OCA\Files\Controller\ViewController”,“type”:"->",“args”:[]
{“file”:"/var/www/html/lib/private/AppFramework/Http/Dispatcher.php",“line”:169,“function”:“index”,“class”:“OCA\Files\Controller\ViewController”,“type”:"->",“args”:["","",null,false]
{“file”:"/var/www/html/lib/private/AppFramework/Http/Dispatcher.php",“line”:100,“function”:“executeController”,“class”:“OC\AppFramework\Http\Dispatcher”,“type”:"->",“args”:[{“class”:“OCA\Files\Controller\ViewController”
“index”]
{“file”:"/var/www/html/lib/private/AppFramework/App.php",“line”:152,“function”:“dispatch”,“class”:“OC\AppFramework\Http\Dispatcher”,“type”:"->",“args”:[{“class”:“OCA\Files\Controller\ViewController”
“index”]
{“file”:"/var/www/html/lib/private/Route/Router.php",“line”:308,“function”:“main”,“class”:“OC\AppFramework\App”,“type”:"::",“args”:[“OCA\Files\Controller\ViewController”,“index”,{“class”:“OC\AppFramework\DependencyInjection\DIContainer”
{“action”:null,"_route":“files.view.index”}]
{“file”:"/var/www/html/lib/base.php",“line”:1009,“function”:“match”,“class”:“OC\Route\Router”,“type”:"->",“args”:["/apps/files/"]
{“file”:"/var/www/html/index.php",“line”:37,“function”:“handleRequest”,“class”:“OC”,“type”:"::",“args”:[]}],“File”:"/var/www/html/lib/private/Log/ErrorHandler.php",“Line”:91,“CustomMessage”:"–"
“userAgent”:“Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0”,“version”:“20.0.1.1”}

10

Ok, this error will be fixed in 20.0.2:

11

could that be the culprit to it all?

12

Nope… still the dreaded “Not enough free space”, and no new error messages. :frowning:

13

Check your actual volume you are using in docker.

On the host server please run df -h /mnt/docker/volume/data/ and please copypasta the results here for the volume location you have mounted into your Docker container.

14

This is how df -hl looks on the docker host (i removed the tmpfs and udev stuff):

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/bmw–vg-root 28G 14G 13G 51% /
/dev/mapper/bmw–vg-home 75G 3.5G 67G 5% /home
/dev/sdb2 1.9T 209G 1.7T 12% /media/theo/Untitled 3

at the moment they are mounted into docker as such:

volumes:
  - ./html:/var/www/html
  - /home/nextcloud_data:/var/www/html/data

becuse the /home partition is a bit bigger. As this one is the last fresh install, I can change it to whatever you see fit.

This is my newest attempt trying to get it to work. On the original install they are docker containers on the same partition.

15

Here is a guide I wrote for setting up Docker on NextcloudPi. Maybe it will help (or not). Basically principles have worked fine for me.

16

The thing is, even if I am just doing a simple:

docker run -d -p 8080:80 nextcloud

I have the same issue. Even changing computer to one that didn’t even have docker installed before doing a fresh install of it all. No matter where I put the data, on the same disks, on different disks. No matter if I try the 19.0.4 or try the 20.0.1 I get the same results. I have them all upp and running and they all give me “Not enough free space”

The only common nominator left is more or less the nginx proxy server, the mac client and the browser, everything else I have changed only to get the same result.

17

I’d use an actual volume, which a literal directory on the server mapped to docker. This will negate the limitations. Google ‘docker volume’ and move your data directory there.

18

I do not believe the message actually relates to the problem itself, as I have no configured nextcloud on several totally different computers, in all kind of different ways, mysql, sqlite, with tons of storages as docker containers or directly under the os disk as well as a dedicated brand new empty SSD disk.

I believe the problem might be in how (some script somewhere) tries to evaluate the disk space, or some access rights for the user/server or something.

The original docker installation worked perfectly until a week ago, now none work at all.

19

Did you try exactly the same on the other Docker instance you created?

Because using your local folders can cause permission issues, as the user in the docker will most likely run under a different user ID than the user in the home folder.

Please try the following situations (proceed down the list as they all work):

  1. New Docker, no volumes -> upload a file
  2. New Docker, data volume only (actual volume: nextcloud_data:/var/www/html/data)
  3. New Docker, these volumes:
            - nextcloud_apps:/var/www/html/apps
            - nextcloud_config:/var/www/html/config
            - nextcloud_data:/var/www/html/data

As you can see, there is also no html folder. That one is actually not needed for the docker and just creating folders for apps and config seems to provide a more easily upgradeable docker.

If these 3 all work, then the assumption there is a permissions issue can be made. So then you can try reintroducing the data path of your own config, however i would not recommend this.

If you have a system with SELinux installed, this could also be the culprid. Disabling SELinux should then see your uploading work, but that is also not recommended.

If you need to increase the partition of your /var/lib/docker/volumes I would recommend decreasing the size of your /home partition and creating a partition that mounts to /var/lib/docker/volumes instead.

20

I have now removed the docker container and erased any data volumes that goes with it.

And going through M_R list:

  1. No volumes = “Not enough free space”

Well as not even the first one does work I am stopping here to await new orders :-/

df -hl on the host:

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/bmw–vg-root 28G 13G 14G 47% /
/dev/mapper/bmw–vg-home 75G 3.5G 67G 5% /home
/dev/sdb2 1.9T 209G 1.7T 12% /media/theo/Untitled 3