I am having a problem that I’m hoping to get some help with. Recently, I began to use snap packages on my server. Currently using 2, one of which is the Nextcloud snap.
After about 2 days of a fresh Nextcloud snap install the free space on my hard drive mysteriously disappears. Typically, I expect to see about 85 gigs of free space after the install. Then over a two or three day period, the 85 gigs of free space fill to capacity rendering several processes on my server inoperable. It is hard to imagine that somewhere there is a log file hogging 85 gigs of free space. Has anyone encountered this problem and if so how did they resolve it?
I should clarify that under these circumstances, when I delete the Nextcloud snap, I get the 85 gigs back plus an additional 10 gigs to account for the install and my files in the data directory.
I’ve heard of the logs getting very large, but we recently introduced proper log rotations so that shouldn’t be causing this unless you’re getting unimaginably hammered with traffic or your logs are filling up with errors. Check the size of the files in /var/snap/nextcloud/current/logs/. If any seem huge, take a look, make sure it’s not filling up with error logs from a misbehaving app, for example.
Since my last post, I reinstalled the nextcloud snap. That was 2 days ago. Thought all was fine until this morning where I find that my drive has 0 space left. I checked the logs and found the nextcloud.log to be a wopping 84 gigs!
-rw-r–r-- 1 root root 6.0M Oct 23 13:12 apache_access.log
-rw-r–r-- 1 root root 52K Oct 23 12:52 apache_errors.log
-rw-r----- 1 root root 187K Oct 23 11:05 mysql_errors.log
-rw-r----- 1 root root 84G Oct 23 13:12 nextcloud.log
-rw-r----- 1 root root 2.6G Oct 23 00:07 nextcloud.log.1
-rw-r–r-- 1 root root 596K Oct 23 11:36 php_errors.log
-rw------- 1 root root 74K Oct 23 11:36 php-fpm_errors.log
-rw-r–r-- 1 root root 214K Oct 23 13:12 redis.log
Also, I didn’t enable the external storage app until last night on the new install as I have suspected this could the culprit. In the hours that followed enabling that app, the log grew overnight.
My disk space is exhausted this morning due to the same issue.
root@server:/var/snap/nextcloud/common/nextcloud/data# ls -alh | grep log
-rw-r----- 1 root root 0 Oct 23 13:09 nextcloud.log
-rw-r----- 1 root root 220G Oct 16 01:41 nextcloud.log.1
It’s filled with this:
,{“file”:"/snap/nextcloud/23750/htdocs/lib/private/Files/Cache/Scanner.php",“line”:391,“function”:“scanChildren”,“class”:“OC\Files\Cache\Scanner”,“type”:"->",“args”:["{dir1}/{dir2}/////////(repeating)"…
Where {dir1} and {dir2} are censored for privacy. That directory no longer exists, as the user has been completely removed from the system.
Oh brutal. This is definitely due to a misbehaving app. We might be able to update the snap to only take a percentage of available free space for logs, but that doesn’t change the fact that the apps are logging errors so quickly that your logs will be completely useless. I suggest logging bugs against the misbehaving apps, and fix whatever problems they’re complaining about (or disable them) to stop this silliness. For example, I logged a bug about this type of behavior to richdocumentscode and they’re working on it. You should be able to safely truncate that log, assuming you’re okay losing it:
For others who may come here, I seem to have resolved it, and will keep an eye on it to be certain.
I pulled the log file to my local PC for further analysis, erased it on the server, rebooted, and the snap started pounding on the logs again. I then re-added the user to the system get the root path (/home/{dir1:username} - reference above) reconnected. This stopped the logging and I was able to remove the new file to reclaim the space.
I have also used:
find {path} -follow --printf “”
to find any symbolic link loops in the filesystem.
Only the normal rpi4/ubuntu ones seem to exist.
Guess: if the reader has recently removed a “root mount” or a user that was using nextcloud, try re-creating the user and/or their home directory.
I will file a bug if it happens again, as the reason I logged into this system was to perform updates and security upgrades of all packages. Unfortunately, this means I no longer know what versions I was running at the time.
