Fixing Letsencrypt CA root certificate change → HTTPD service down - HPB service down → Nextcloud unreachable

Hi,

Letsencrypt have changed to a new provider for root CA certificate, the old one expired on 30092021. Some older libraries do not support it, like my Nextcloudpi server that is signed with a Letsencrypt cert.
I fixe my nextcloudpi instance by this quick procedure:
These notes may help.

The failures:

NextCloudPi v1.39.19 is outdated
update to v1.40.4 through ‘ncp-config’ or type ‘sudo ncp-update’
root@cloud:~# ncp-update
Downloading updates
Performing updates
Installing nc-restore
AH00526: Syntax error on line 5 of /etc/apache2/sites-enabled/ncp.conf:
SSLCertificateFile: file ‘/etc/letsencrypt/live/cloud.xxx.com/fullchain.pem’ does not exist or is empty
Action ‘-k graceful’ failed.
The Apache error log may have more information.
Error while applying update 1.40.0. Exiting…

root@cloud:~# systemctl status apache2
● apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset:
Active: failed (Result: exit-code) since Thu 2021-10-14 04:01:06 CEST; 6 days
Docs: https://httpd.apache.org/docs/2.4/
Process: 994 ExecStart=/usr/sbin/apachectl start (code=exited, status=1/FAILUR

root@cloud:~# systemctl restart apache2
Job for apache2.service failed because the control process exited with error code.
See “systemctl status apache2.service” and “journalctl -xe” for details.

One working solution:
1/ Update server
root@cloud:~# apt update && apt upgrade

2/ Disable letsencrypt
root@cloud:~# ncp-config
NextCloudPi installer configuration

  Enter configuration for letsencrypt

   ACTIVE        no
   DOMAIN        cloud.xxx.com
   OTHER_DOMAIN
   EMAIL         postmaster@xxx.com

                    <Start >          <Cancel>

Running letsencrypt
INFO: Metrics enabled: no
letsencrypt certificates disabled. Using self-signed certificates instead.
Done. Press any key…

root@cloud:~# ncp-config
Running nc-info
Gathering information…
NextCloudPi version v1.39.19
NextCloudPi image NextCloudPi_01-09-20
distribution Armbian 21.08.2 Buster \l . 5.4.151-odroidxu4 (armv7l)
.

Nextcloud check ok
Nextcloud version 20.0.12.1
HTTPD service down
PHP service up
MariaDB service up
Redis service up
HPB service down
Postfix service up
internet check ok
port check 80 closed
port check 443 closed
.

You should run Lets Encrypt for trusted encrypted access

3/ Unistall Letsencrypt:
Armbian distribution ships letsencrypt as a package, I would recommend using package manager to uninstall the client

root@cloud:~# apt-get remove --purge letsencrypt

4/ Edit ncp.conf file:
root@cloud:~# vi /etc/apache2/sites-enabled/ncp.conf
turn-off line 4
SSLEngine = on > off
Comment out lines 5 & 6
SLLCertificateFile /etc/letsencrypt/live/cloud.xxx.com/fullchain.pem > # …
SSLCertificateFileKeyFile /etc/letsencrypt/live/cloud.xxx.com/privkey.pem > # …

5/ Update nextcloudpi:
root@cloud:~# systemctl restart apache2
root@cloud:~# systemctl status apache2
● apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset:
Active: active (running) since Wed 2021-10-20 17:44:04 CEST; 23min ago
Docs: https://httpd.apache.org/docs/2.4/
Main PID: 13230 (apache2)
Tasks: 51 (limit: 4447)
Memory: 146.0M
CGroup: /system.slice/apache2.service
├─13230 /usr/sbin/apache2 -k start
├─15179 /usr/sbin/apache2 -k start
├─17912 /usr/sbin/apache2 -k start
├─17919 /usr/sbin/apache2 -k start
├─17945 /usr/sbin/apache2 -k start
├─19342 /usr/sbin/apache2 -k start
├─19725 /usr/sbin/apache2 -k start
├─20296 /usr/sbin/apache2 -k start
├─20497 /usr/sbin/apache2 -k start
├─20502 /usr/sbin/apache2 -k start
└─20507 /usr/sbin/apache2 -k start

Oct 20 17:44:04 cloud systemd[1]: Starting The Apache HTTP Server…
Oct 20 17:44:04 cloud systemd[1]: Started The Apache HTTP Server.
lines 1-22/22 (END)

Update nextcloudpi:
root@cloud:~# ncp-update
.

NextCloudPi updated to version v1.41.10

6/ Update nextcloud from 20.0.12.1 to a new version 21.0.5
root@cloud:~# ncp-config
NextCloudPi installer configuration

  Enter configuration for nc-update-nextcloud

   VERSION       21.0.5

                    <Start >          <Cancel>

Running nc-update-nextcloud
Current Nextcloud version 20.0.12.1
Available Nextcloud version 21.0.5
.

To activate the new configuration, you need to run:
systemctl restart apache2
Created symlink /etc/systemd/system/multi-user.target.wants/notify_push.service → /etc/systemd/system/notify_push.service.
System config value trusted_domains => 3 set to string cloud.xxx.com
System config value overwrite.cli.url set to string https://cloud.xxx.com/
System config value trusted_proxies => 11 set to string 127.0.0.1
System config value trusted_proxies => 12 set to string ::1
System config value trusted_proxies => 13 set to string cloud.xxx.com
System config value trusted_proxies => 14 set to string WAN.IP.WAN.IP
✓ redis is configured
✓ push server is receiving redis messages
✓ push server can load mount info from database
✓ push server can connect to the Nextcloud server
🗴 push server is not a trusted proxy, please add ‘LAN.IP.LAN.IP’ to the list of trusted proxies or configure any existing reverse proxy to forward the ‘x-forward ed-for’ send by the push server.
See https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/reverse_proxy_configuration.html#defining-trusted-proxies for how to set truste d proxies.
The following trusted proxies are currently configured: “127.0.0.1”, “::1”, “cloud.xxx.com”, “WAN.IP.WAN.IP”
of which the following seem to be invalid: “cloud.xxx.com
The following x-forwarded-for header was received by Nextcloud: 1.2.3.4
from the following remote: LAN.IP.LAN.IP

If you’re having issues getting the trusted proxy setup working, you can try bypassing any existing reverse proxy
in your setup by setting the NEXTCLOUD_URL environment variable to point directly to the internal Nextcloud webserver url
(You will still need the ip address of the push server added as trusted proxy)
notify_push setup failed. You are probably behind a proxy
Run ‘ncc config:system:set trusted_proxies 15 --value=<proxy_IP>’ and then 'nccnotify_push:setup https:///push to enable
Check https://help.nextcloud.com/tags/ncp for support

Fixe trusted_proxies /var/www/nextcloud/config/config.php

root@cloud:# ncc config:system:set trusted_proxies 15 --value=LAN.IP.LAN.IP
System config value trusted_proxies => 15 set to string LAN.IP.LAN.IP
root@cloud: ncc notify_push:setup https://cloud.xxx.com/push
✓ redis is configured
✓ push server is receiving redis messages
✓ push server can load mount info from database
✓ push server can connect to the Nextcloud server
✓ push server is a trusted proxy
✓ push server is running the same version as the app
configuration saved

7/ Check the system:

root@cloud: ncp-config
Running nc-info
Gathering information…
NextCloudPi version v1.41.10
NextCloudPi image NextCloudPi_01-09-20
OS Armbian 21.08.3 Buster \l . 5.4.151-odroidxu4 (armv7l)
.

Nextcloud check ok
Nextcloud version 21.0.5.1
HTTPD service up
PHP service up
MariaDB service up
Redis service up
HPB service up
Postfix service up
internet check ok
port check 80 open
port check 443 open
.

You should run Lets Encrypt for trusted encrypted access
Done. Press any key…

8/ All is functional, reinstall letsencrypt:
root@cloud:# apt-get install letsencrypt

9/ Enable letsencrypt for your nextcloud instance
root@cloud:# ncp-config
NextCloudPi installer configuration

   Enter configuration for letsencrypt

    ACTIVE        yes
    DOMAIN        cloud.xxx.com
    OTHER_DOMAIN
    EMAIL         postmaster@xxx.com


                      <Start >          <Cancel>

Running letsencrypt
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Obtaining a new certificate
IMPORTANT NOTES:

  • Congratulations! Your certificate and chain have been saved at:
    /etc/letsencrypt/live/cloud.xxx.com-0002/fullchain.pem
    Your key file has been saved at:
    /etc/letsencrypt/live/cloud.xxx.com-0002/privkey.pem

10/ Update your ncp.conf file:

root@cloud:~# vi /etc/apache2/sites-enabled/ncp.conf
turn-on line 4
SSLEngine = off > on
Uncomment & modify lines 5 & 6
SLLCertificateFile /etc/letsencrypt/live/cloud.xxx.com-0002/fullchain.pem
SSLCertificateFileKeyFile /etc/letsencrypt/live/cloud.xxx.com-0002/privkey.pem

Then restart apache2
root@cloud:~# systemctl restart apache2

Quick but perhaps dirty!

konki