NCP update 1.37.0 to 1.38.5 on Armbian | " error creating directory './var/log/prometheus': No space left on device"

Hi,

I tried to update by Odroid HC4 running armbian to ncp 1.38.5 but was met with an error that there’s not enough space to make a new log folder for prometheus in /var/log.

I’ve got / on an SSD in one of the sata slots, /data in the other sata slot with a HDD and a traditional swap pertition set up. /var/log seems to be on a zram partition, which is odd as I though I disabled this during the install.

system info
  ___      _           _     _   _   _  ____ _  _   
 / _ \  __| |_ __ ___ (_) __| | | | | |/ ___| || |  
| | | |/ _` | '__/ _ \| |/ _` | | |_| | |   | || |_ 
| |_| | (_| | | | (_) | | (_| | |  _  | |___|__   _|
 \___/ \__,_|_|  \___/|_|\__,_| |_| |_|\____|  |_|  
                                                    
Welcome to Armbian 21.05.6 Buster with Linux 5.10.43-meson64

System load:   3%               Up time:       28 days 14:01
Memory usage:  23% of 3.73G     Zram usage:    3% of 7.88G      IP:            *.*.*.*
CPU temp:      46°C             Usage of /:    14% of 104G   
RX today:      94.4 MiB  

[ General system configuration (beta): armbian-config ]

$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0 111.8G  0 disk 
├─sda1        8:1    0 103.9G  0 part /
└─sda2        8:2    0   7.9G  0 part [SWAP]
sdb           8:16   0   2.7T  0 disk /data
mmcblk0     179:0    0   7.2G  0 disk 
└─mmcblk0p1 179:1    0   7.1G  0 part /media/mmcboot
zram0       252:0    0    50M  0 disk /var/log


sudo ncp-update
$ sudo ncp-update
[sudo] password for admin: 
Downloading updates
Performing updates
Installing metrics
Hit:1 http://security.debian.org buster/updates InRelease
Hit:2 http://deb.debian.org/debian buster InRelease                                                      
Hit:3 http://deb.debian.org/debian buster-updates InRelease                                              
Hit:4 http://deb.debian.org/debian buster-backports InRelease                                            
Hit:5 https://apt.syncthing.net syncthing InRelease                                                      
Hit:6 http://armbian.systemonachip.net/apt buster InRelease               
Reading package lists... Done
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  prometheus-node-exporter
0 upgraded, 1 newly installed, 0 to remove and 10 not upgraded.
3 not fully installed or removed.
Need to get 0 B/2,716 kB of archives.
After this operation, 10.3 MB of additional disk space will be used.
(Reading database ... 49251 files and directories currently installed.)
Preparing to unpack .../prometheus-node-exporter_0.17.0+ds-3+b11_arm64.deb ...
Unpacking prometheus-node-exporter (0.17.0+ds-3+b11) ...
dpkg: error processing archive /var/cache/apt/archives/prometheus-node-exporter_0.17.0+ds-3+b11_arm64.deb (--unpack):
 error creating directory './var/log/prometheus': No space left on device
Errors were encountered while processing:
 /var/cache/apt/archives/prometheus-node-exporter_0.17.0+ds-3+b11_arm64.deb
E: Write error - ~LZMAFILE (28: No space left on device)
E: Sub-process /usr/bin/dpkg returned an error code (1)
ncp-zram info
$ sudo ncp-zram stop
rmmod: ERROR: Module zram is in use

So the command, armbian MOTD, and lsblk says it’s in use but the webinterface says it isnt:
image

Not sure why lsblk says /var/log is on zram0 but is only 50M, whereas armbian MOTD says zram is using the swap partition and is 8G.

This might be more of an armbian question so I’ll have a look over at that forum how to properly configure/disable zram again.

Found a thread on zram over on armbian forum. Someone suggested to run # armbianmonitor -u which uploaded som info here http://ix.io/3wHb
In the output there’s some interesting output from zramctl that differs from my output when I run it straight after:

### zramctl:

NAME       ALGORITHM DISKSIZE  DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram1 zstd           50M  132K  697B   36K       4 /var/log
/dev/zram0 lzo-rle       1.9G    4K   74B   12K       4 [SWAP]

when I run it right after:

$ sudo zramctl
NAME       ALGORITHM DISKSIZE  DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 zstd           50M 47.5M  6.7M  7.3M       4 /var/log

man zramctl has som info on how to setup the zram so might just delete this one and setup a bigger one

Seems like there was a log2ram setup initially which was removed at some point.
blkid showed that the zram /var/log ram partition had the label log2ram but couldnt find it on the system. I installed it, and added some configs from the last post here:
https://forum.armbian.com/topic/6626-var-easily-get-full-with-log2ram/
Basically shoten the log rotation and make the deletion more aggressive. The main culprit in /var/log was apache2 so might look into which errors to log in that config as well.