I can't update NCP since 0.67.11 ( /usr/local/etc/library.sh: line 117: echo: write error: No space left on device)

groby · January 10, 2019, 1:02am

i can’t update NCP since 0.67.11:

Tried to update to v1.0.1 through ‘ncp-config’ or type ‘sudo ncp-update’ but i always have:

root@nextcloudpi:~# sudo ncp-update
Downloading updates
Performing updates
Running nc-autoupdate-nc
/usr/local/etc/library.sh: line 117: echo: write error: No space left on device
root@nextcloudpi:~#

same with NextcloudPi Panel (green led but no output in the white window)
same after rebooting Odroid HC2

Tried to free some space with no luck:

apt-get autoremove -y
apt-get clean
rm /var/lib/apt/lists/* -r

what should i do? is my SDcard full? i don’t get it…

OliverV · January 10, 2019, 7:57am

Check and share output of:

df -h

Note: if you are logged in as root, there is no need for sudo
Sudo is used for user pi or other privileged users, that are members of sudo group.

Edit: @groby forgot to ask, also useful for devs to have logs, if possible to get output of ncp,log, on pastbin.com or similar, and add link to it below, thank you.

groby · January 10, 2019, 6:19pm

i did:
rm /var/log/apache2/* (something like that…it was late)

and i was able to ncp-update to 1.0.1
looks like logs are eating space?

here the output you ask:

root@nextcloudpi:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            932M     0  932M   0% /dev
tmpfs           200M   23M  178M  12% /run
/dev/mmcblk1p1   30G  3.9G   25G  14% /
tmpfs           999M     0  999M   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           999M     0  999M   0% /sys/fs/cgroup
tmpfs           999M  4.0K  999M   1% /tmp
/dev/zram0       49M   48M     0 100% /var/log
/dev/sda1       3.7T  391G  3.3T  11% /media/DATAcloud
tmpfs           200M     0  200M   0% /run/user/0

update: Now same thing is happening trying to update ncp-update to 1.0.2

ncp-report: https://ghostbin.com/paste/2co8e

cak89 · January 12, 2019, 3:19pm

Hello!
I got the same error. Can’t update NCP to v1.0.2 neither.

root@nextcloudpi:~# ncp-update
Downloading updates
Performing updates
Running nc-autoupdate-nc
/usr/local/etc/library.sh: line 117: echo: write error: No space left on device

When trying to open a log file, the following error appears:

[ Error writing lock file ./.ncp.log.swp: No space left on device ]

Log file seems to be completely empty but i’m not sure if it can’t be read because of the missing lock file (?)

df -h gives me:

Filesystem      Size  Used Avail Use% Mounted on
udev            932M     0  932M   0% /dev
tmpfs           200M   23M  178M  12% /run
/dev/mmcblk1p1   15G  5.9G  8.6G  41% /
tmpfs           999M  8.0K  999M   1% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           999M     0  999M   0% /sys/fs/cgroup
tmpfs           999M     0  999M   0% /tmp
/dev/zram0       49M   48M     0 100% /var/log
/dev/sdb1       233G  165G   67G  72% /media/clouddrive
/dev/sda2       2.8T  2.6T  208G  93% /media/Medien
tmpfs           200M     0  200M   0% /run/user/0

Any help would be great, thanks in advance!

nachoparker · January 12, 2019, 3:31pm

hi, you guys need to free up some space in /var/log

I would like to see what’s taking up so much space, could you do du -shc /var/log/*?

cak89 · January 12, 2019, 3:51pm

du -shc /var/log/* returns:

0 /var/log/alternatives.log
4.0K /var/log/apache2
4.0K /var/log/apt
0 /var/log/armbian-hardware-monitor.log
0 /var/log/armbian-services.log
0 /var/log/btmp
0 /var/log/daemon.log
0 /var/log/debug
0 /var/log/dpkg.log
0 /var/log/fail2ban.log
0 /var/log/kern.log
64K /var/log/letsencrypt
4.0K /var/log/mail.log
0 /var/log/messages
4.0K /var/log/mysql
0 /var/log/ncp.log
4.0K /var/log/ntpstats
0 /var/log/php7.0-fpm.log
0 /var/log/php7.2-fpm.log
4.0K /var/log/redis
0 /var/log/rpimonitor.log
16K /var/log/samba
0 /var/log/syslog
4.0K /var/log/sysstat
4.0K /var/log/unattended-upgrades
0 /var/log/user.log
4.0K /var/log/wicd
0 /var/log/wtmp
116K total

I already deleted some logfiles, without having any affect on the size of /dev/zram0.
Thanks for helping!

groby · January 12, 2019, 3:53pm

NextCloudPi v1.0.1 is outdated
update to v1.0.2 through 'ncp-config' or type 'sudo ncp-update'
You have new mail.

root@nextcloudpi:~# du -shc /var/log/*?
0	/var/log/alternatives.log
4.0K	/var/log/apache2
4.0K	/var/log/apt
0	/var/log/armbian-hardware-monitor.log
4.0K	/var/log/auth.log
0	/var/log/btmp
0	/var/log/daemon.log
0	/var/log/debug
0	/var/log/dpkg.log
0	/var/log/fail2ban.log
0	/var/log/kern.log
4.0K	/var/log/letsencrypt
0	/var/log/lynis.log
48K	/var/log/lynis-report.dat
48M	/var/log/mail.info
0	/var/log/mail.log
0	/var/log/mail.warn
0	/var/log/messages
4.0K	/var/log/mysql
0	/var/log/ncp.log
4.0K	/var/log/ntpstats
0	/var/log/php7.2-fpm.log
4.0K	/var/log/redis
68K	/var/log/samba
0	/var/log/syslog
4.0K	/var/log/sysstat
0	/var/log/ufw.log
4.0K	/var/log/unattended-upgrades
0	/var/log/user.log
4.0K	/var/log/wicd
0	/var/log/wtmp
48M	total
root@nextcloudpi:~#

i tried to delete “apache2 logs” and “ncp.log” but looks like “mai.info” is taking space.
Don’t know what to delete, i don’t want to mess everything.

OliverV · January 12, 2019, 4:05pm

To remove it run

sudo rm /var/log/mail.info

maybe check inside first with

sudo cat /var/log/mail.info

groby · January 12, 2019, 4:34pm

Looks like Fail2ban alert didn’t like my protonmail address, i had tons of:
Oct 4 03:33:37 nextcloudpi postfix/pickup[1887]: 5221A1950: uid=0 from=ncp-fail2ban@ownyourbits.com
Oct 4 03:33:37 nextcloudpi postfix/cleanup[14061]: 5221A1950: message-id=201810015516516221A1950@65f4123548
Oct 4 03:33:37 nextcloudpi postfix/qmgr[1888]: 5221A1950: from=ncp-fail2ban@ownyourbits.com, size=3005, nrcpt=1 (queue active)
Oct 4 03:34:07 nextcloudpi postfix/smtp[14063]: connect to mail.protonmail.ch[185.70.40.101]:25: Connection timed out
Oct 4 03:34:37 nextcloudpi postfix/smtp[14063]: connect to mailsec.protonmail.ch[185.70.40.102]:25: Connection timed out

i changed this to the email address i configured in nextcloud.
anyway i did sudo rm /var/log/mail.info

but still can’t update

:~# sudo rm /var/log/mail.info
You have new mail in /var/mail/root
root@nextcloudpi:~# ncp-update
Downloading updates
Performing updates
Running nc-autoupdate-nc
/usr/local/etc/library.sh: line 117: echo: write error: No space left on device
root@nextcloudpi:~# du -shc /var/log/*
0	/var/log/alternatives.log
4.0K	/var/log/apache2
4.0K	/var/log/apt
0	/var/log/armbian-hardware-monitor.log
4.0K	/var/log/auth.log
0	/var/log/btmp
0	/var/log/daemon.log
0	/var/log/debug
0	/var/log/dpkg.log
0	/var/log/fail2ban.log
0	/var/log/kern.log
4.0K	/var/log/letsencrypt
0	/var/log/lynis.log
48K	/var/log/lynis-report.dat
0	/var/log/mail.log
0	/var/log/mail.warn
0	/var/log/messages
4.0K	/var/log/mysql
0	/var/log/ncp.log
4.0K	/var/log/ntpstats
0	/var/log/php7.2-fpm.log
4.0K	/var/log/redis
68K	/var/log/samba
0	/var/log/syslog
4.0K	/var/log/sysstat
0	/var/log/ufw.log
4.0K	/var/log/unattended-upgrades
0	/var/log/user.log
4.0K	/var/log/wicd
0	/var/log/wtmp
156K	total

Then i rebooted and mail.info is back!

root@nextcloudpi:~#     df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            932M     0  932M   0% /dev
tmpfs           200M  5.0M  195M   3% /run
/dev/mmcblk1p1   30G  3.9G   25G  14% /
tmpfs           999M     0  999M   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           999M     0  999M   0% /sys/fs/cgroup
tmpfs           999M     0  999M   0% /tmp
/dev/zram0       49M   48M     0 100% /var/log
/dev/sda1       3.7T  392G  3.3T  11% /media/DATAcloud
tmpfs           200M     0  200M   0% /run/user/0
root@nextcloudpi:~# 
root@nextcloudpi:~# du -shc /var/log/*
0	/var/log/alternatives.log
4.0K	/var/log/apache2
4.0K	/var/log/apt
0	/var/log/armbian-hardware-monitor.log
4.0K	/var/log/auth.log
0	/var/log/btmp
0	/var/log/daemon.log
0	/var/log/debug
0	/var/log/dpkg.log
0	/var/log/fail2ban.log
0	/var/log/kern.log
4.0K	/var/log/letsencrypt
0	/var/log/lynis.log
48K	/var/log/lynis-report.dat
48M	/var/log/mail.info
4.0K	/var/log/mail.log
0	/var/log/mail.warn
4.0K	/var/log/messages
4.0K	/var/log/mysql
0	/var/log/ncp.log
4.0K	/var/log/ntpstats
0	/var/log/php7.2-fpm.log
4.0K	/var/log/redis
56K	/var/log/samba
4.0K	/var/log/syslog
4.0K	/var/log/sysstat
0	/var/log/ufw.log
4.0K	/var/log/unattended-upgrades
0	/var/log/user.log
4.0K	/var/log/wicd
0	/var/log/wtmp
48M	total
root@nextcloudpi:~#

OliverV · January 12, 2019, 5:04pm

Looks like your smtp is not configured correctly

cak89 · January 12, 2019, 5:30pm

I managed to solve the problem (for me).

I had to configure my mail setup correctly
I had to delete the big logfile stored under var/log.hdd (they reappeard in var/log after reboot?)
I deleted the same logfile in var/log
reboot

Hope it helps!

groby · January 12, 2019, 5:45pm

i 've checked my smtp setup now looks fine. and i just left one admin user in nextcloud.
(previously i’ve transfered and deleted “ncp” who was the default admin in nextcloud after the first setup, to another admin name).
what should i delete in /var/log.hdd?

root@nextcloudpi:/var/log.hdd# ls
alternatives.log		   fail2ban.log.4.gz  ntpstats
apache2				   kern.log	      php7.2-fpm.log
apt				   kern.log.1	      php7.2-fpm.log.1
armbian-hardware-monitor.log	   letsencrypt	      redis
armbian-hardware-monitor.log.1.gz  lynis.log	      samba
armbian-ramlog.log		   lynis-report.dat   syslog
auth.log			   mail.info	      syslog.1
auth.log.1			   mail.info.1	      syslog.2.gz
auth.log.2.gz			   mail.info.2.gz     syslog.3.gz
auth.log.3.gz			   mail.info.3.gz     syslog.4.gz
auth.log.4.gz			   mail.info.4.gz     syslog.5.gz
btmp				   mail.log	      syslog.6.gz
btmp.1.gz			   mail.log.1	      syslog.7.gz
daemon.log			   mail.log.2.gz      sysstat
daemon.log.1			   mail.log.3.gz      ufw.log
debug				   mail.log.4.gz      ufw.log.1
debug.1				   mail.warn	      unattended-upgrades
dpkg.log			   messages	      user.log
fail2ban.log			   messages.1	      user.log.1
fail2ban.log.1			   messages.2.gz      wicd
fail2ban.log.2.gz		   mysql	      wtmp
fail2ban.log.3.gz		   ncp.log	      wtmp.1.gz

cak89 · January 12, 2019, 6:37pm

with

ls -hs

you can see the filesize. in my case, i deleted the biggest ones.

groby · January 12, 2019, 7:17pm

Finally! thanks

root@nextcloudpi:/var/log#     ls -hs
total 472K
   0 alternatives.log		   4.0K letsencrypt	  4.0K redis
4.0K apache2			      0 lynis.log	  4.0K samba
4.0K apt			    48K lynis-report.dat  108K syslog
 44K armbian-hardware-monitor.log  8.0K mail.info	  4.0K sysstat
 16K auth.log			   8.0K mail.log	     0 ufw.log
8.0K btmp			      0 mail.warn	  4.0K unattended-upgrades
 40K daemon.log			    60K messages	   12K user.log
4.0K debug			   4.0K mysql		  4.0K wicd
   0 dpkg.log			   4.0K ncp.log		  4.0K wtmp
 12K fail2ban.log		   4.0K ntpstats
 52K kern.log			   4.0K php7.2-fpm.log
root@nextcloudpi:/var/log# cd ~
root@nextcloudpi:~# ncp-update
Downloading updates
Performing updates
Running nc-autoupdate-nc
automatic Nextcloud updates enabled
NextCloudPi updated to version v1.2.0
root@nextcloudpi:~#

is there a way to avoid this logs problems in the future or we just need to check
/var/log.hdd and /var/log sometimes?