NCP upgrade fails due to full /var/log

Hello knowledgeable people. I will start by saying that Nextcloudpi has served me well for years now, and I am grateful for the software. Previous upgrades haven’t failed me, but this time there was a snag.
I ran sudo ncp-update-nc while logged in to SSH. I started noticing complaints about /var/log being full, so I thought “OK, let apt find out of it and roll back to the previous version” - I believe PHP8 was the culprit. The messages indicated that was exactly what was happening, so I let it continue. After a while, however, I was thrown out of SSH and couldn’t ping the box anymore. Reboots don’t help.

I run Nextcloudpi on an Odroid HC2. As you know, this board runs from an SD card, has its own SATA port and a USB port for backups to a USB HD. This means I have access to all files, but as the board is headless, I can’t do a local login or see boot messages to figure out what’s wrong. I need it to boot so I can log in to SSH. I have backups of the files and of the Nextcloudpi config on the external USB drive, and the internal SATA drive is fairly new. I have little reason to suspect hardware failure.

What I do have reason to suspect, is failed logrotating which leads to a full /var/log. The partition (automatically created when installing Nextcloudpi) is fairly small to begin with, so that could also be a problem. I can reinstall and get up again, but it does take some time, and I wonder if I can get around that. This post, then, is to notify of a potential problem with /var/log, and also two questions:

  1. Does anyone know if I can get around reinstalling?
  2. A heretic question: Does anyone know if it would be possible to restore my config if I should choose to install Yunohost instead of Nextcloudpi, given the case that I have to reinstall?

Thanks in advance, and again: Thank you for making Nextcloudpi. I am happy and don’t complain.

1 Like

If you have access to another Linux computer that can read an SD card, you can plug it in there and remove the full log file that’s causing issues.

Note: If the SD card is in a “read-only” state when you plug it in, it’s at the end of its lifespan (unless of course you forgot to flip the switch on the adapter)

I don’t know about your second question :pray: you can always try and see. It should not be compatible with the nextcloud backup however

Thanks for answering. I have tried deleting some of the biggest logfiles in /var/log, but the board still doesn’t boot. My suspicion is that the interrupted upgrade process left the system unbootable, so I will probably have to reinstall. Also, as the HC2 is getting old, I might be left high and dry by current releases that don’t use armhf.

@eivode
Mm, there is an issue upstream in the armbian images, we’ve found another image with armhf which are made by the Debian team directly and will use those instead, so come the next release it should work better for you :pray:

If you want to give it a go right away, you can find the images here

https://www.debian.org/releases/bullseye/debian-installer/

1 Like

Thanks again. I don’t know how to install Debian on the HC2, as I would need to see what is going on to install Debian… And you can’t do that on a headless box like the Odroid HC2. With Armbian, you can, but I understand that Armbian is currently discouraged? If so, I can’t tell how to reinstall.

Of course you can :slight_smile: , Debian can be headless too and installed without a screen, though it requires some tinkering.

It is absolutely the easiest to connect a display to the device temporarily and complete the installation that way and de-select the graphical desktop environments when prompted during the install and select SSH server. I don’t have an odroid hc2 myself however :pray: so I don’t know if it is completely impossible to connect a display to it, according to this article however it should be possible.

You can also setup the image to be installed using something called a preseed configuration for Debian images, which means the image will install itself according to the instructions you set in a file, this way doesn’t require a display at all but is a bit more complicated.

Here are instructions for a preseed installation: DebianInstaller/Preseed - Debian Wiki

And yes, the armbian images are currently discouraged to use

Here is also a GitHub project that helps to make a remote headless installation of Debian easier

The HC2 doesn’t have any graphic output port, so a local display is out of the question. In the article you mentioned, they are using a different Odroid.
Yes, preseed is possible. I will look into that and see if it’s worth the effort. I do have decent backups, though, so I’ll also look into the possibilities of importing them into eg. Yunohost. Thanks for all your help so far.

1 Like

You’re welcome :pray: & good luck

1 Like

If you suspect some of the newly installed services, like php8.1-fpm or even updated apache2 are messing around that prevents login, you may disable them manually from default booting sequence in another Linux machine.

Systemd booting sequence is basically constructed by symbol link service unit files to specific booting target folders. Below operation of systemctl shows that:

$ sudo systemctl disable php8.1-fpm
Synchronizing state of php8.1-fpm.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable php8.1-fpm
Removed /etc/systemd/system/multi-user.target.wants/php8.1-fpm.service.

This is actually equality to manually remove /etc/systemd/system/multi-user.target.wants/php8.1-fpm.service, which is a symbol link from file system.

Hence you can pull off the SD Card, and insert and mount it on another Linux machine, remove the symbol link.
Be careful about the folder structure. It sits insides the etc folder of the mounted SD Card.

Once booted, you can enable it via systemctl as usual:

$ sudo systemctl enable php8.1-fpm
Synchronizing state of php8.1-fpm.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable php8.1-fpm
Created symlink /etc/systemd/system/multi-user.target.wants/php8.1-fpm.service → /lib/systemd/system/php8.1-fpm.service.

Follow the same logic, you may also check whether sshd.service behave normally.