Notify push (sometimes) causes 503-Error / HPB service down on reboot

This is related to this thread: 503-Error / HPB service down / notify push can't connect to the Nextcloud server

As NCP and php had updates and I’ve more experiences I open a new thread.

Nextcloud version (eg, 20.0.5): 27.0.1.2
Operating system and version (eg, Ubuntu 20.04): Debian GNU/Linux 11 (bullseye) (aarch64)
Apache or nginx version (eg, Apache 2.4.25): Apache 2.4.56-1~deb11u2
PHP version (eg, 7.4): 8.1

The issue you are facing: on reboot notify push (sometimes) causes 503-Error / HPB service down

Is this the first time you’ve seen this error? (Y/N): N

NextcloudPi diagnostics

NextcloudPi version  v1.52.4
NextcloudPi image    NextcloudPi_06-02-23
OS                   Debian GNU/Linux 11. 6.1.21-v8+ (aarch64)
automount            yes
USB devices          sda 
datadir              /media/WD_500GB
data in SD           no
data filesystem      ext2/ext3
data disk usage      100G/458G
rootfs usage         6.0G/58G
swapfile             /dev/zram3
dbdir                /var/lib/mysql
Nextcloud check      ok
Nextcloud version    27.0.1.2
HTTPD service        up
PHP service          up
MariaDB service      up
Redis service        up
HPB service          up
Postfix service      up
Internet check       ok
Public IPv4          ***REMOVED SENSITIVE VALUE***
Public IPv6          ***REMOVED SENSITIVE VALUE***
Port 80              open
Port 443             open
IP                   ***REMOVED SENSITIVE VALUE***
Gateway              ***REMOVED SENSITIVE VALUE***
Interface            eth0
Certificates         ***REMOVED SENSITIVE VALUE***
NAT loopback         yes
Uptime               2:28

Nextcloud configuration

{
    "system": {
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": {
            "1": "192.168.1.104",
            "4": "nextcloudpi.local",
            "5": "",
            "6": "nextcloudpi.lan",
            "2": "https:\/\/***REMOVED SENSITIVE VALUE***",
            "11": "109.250.64.197",
            "14": "nextcloudpi",
            "3": "nextcloudpi",
            "20": "***REMOVED SENSITIVE VALUE***",
            "21": "192.168.1.104",
            "22": "localhost"
        },
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "27.0.1.2",
        "overwrite.cli.url": "https:\/\/nextcloudpi\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "memcache.local": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "timeout": 0,
            "password": "***REMOVED SENSITIVE VALUE***"
        },
        "tempdirectory": "\/media\/WD_500GB\/tmp",
        "mail_smtpmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "preview_max_x": "2048",
        "preview_max_y": "2048",
        "jpeg_quality": "60",
        "overwriteprotocol": "https",
        "loglevel": "2",
        "log_type": "file",
        "maintenance": false,
        "theme": "",
        "mail_sendmailmode": "smtp",
        "mail_smtpsecure": "ssl",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "updater.release.channel": "stable",
        "data-fingerprint": "c43964c4e776c8cc78ea7cf4fb8f2ae1",
        "logfile": "\/media\/WD_500GB\/nextcloud.log",
        "app_install_overwrite": [
            "admin_notifications"
        ],
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "htaccess.RewriteBase": "\/"
    }
}

HTTPd logs

[Sun Aug 27 00:00:02.593984 2023] [ssl:warn] [pid 1078:tid 547970348096] AH01909: localhost:4443:0 server certificate does NOT include an ID which matches the server name
[Sun Aug 27 00:00:02.601319 2023] [mpm_event:notice] [pid 1078:tid 547970348096] AH00489: Apache/2.4.56 (Debian) OpenSSL/1.1.1n configured -- resuming normal operations
[Sun Aug 27 00:00:02.601397 2023] [core:notice] [pid 1078:tid 547970348096] AH00094: Command line: '/usr/sbin/apache2'
[Sun Aug 27 10:19:31.561483 2023] [mpm_event:notice] [pid 1078:tid 547970348096] AH00492: caught SIGWINCH, shutting down gracefully
[Sun Aug 27 10:19:48.809767 2023] [ssl:warn] [pid 1056:tid 547643728960] AH01909: localhost:4443:0 server certificate does NOT include an ID which matches the server name
[Sun Aug 27 10:19:48.867525 2023] [ssl:warn] [pid 1076:tid 547643728960] AH01909: localhost:4443:0 server certificate does NOT include an ID which matches the server name
[Sun Aug 27 10:19:48.874627 2023] [mpm_event:notice] [pid 1076:tid 547643728960] AH00489: Apache/2.4.56 (Debian) OpenSSL/1.1.1n configured -- resuming normal operations
[Sun Aug 27 10:19:48.874759 2023] [core:notice] [pid 1076:tid 547643728960] AH00094: Command line: '/usr/sbin/apache2'

Database logs

tail: cannot open '/var/log/mysql/*.log' for reading: No such file or directory

Nextcloud logs ```

On reboot notify push sometimes (more than every second time) doesn’t start correctly, which causes an 503-error. HPB service is down.

/var/www/nextcloud/occ notify_push:self-test is:

✓ redis is configured
✓ push server is receiving redis messages
✓ push server can load mount info from database
🗴 push server can’t connect to the Nextcloud server
Invalid response when getting test cookie

It seems like notify push somehow starts to early. I already changed /etc/systemd/system/multi-user.target.wants/notify_push.service to (this was a suggestion in 503-Error / HPB service down / notify push can't connect to the Nextcloud server - #13 by chrissi55):

[Unit]
Description = Push daemon for Nextcloud clients
After=apache2.service php8.1-fpm.service mysql.service redis.service
#After=mysql.service
#After=redis.service
Requires=redis.service

[Service]
Environment=PORT=7867
Environment=NEXTCLOUD_URL=https://localhost
ExecStart=“/var/www/nextcloud/apps/notify_push/bin/aarch64/notify_push” --allow>
User=www-data
Restart=on-failure
RestartSec=20

[Install]
WantedBy = multi-user.target

sudo systemctl restart php8.1-fpm doesn’t solve the problem.

When I run sudo ncp-config and choose clear-php-opcache in the “TOOLS”-section, everything runs well. /var/www/nextcloud/occ notify_push:self-test afterwards is:

✓ redis is configured
✓ push server is receiving redis messages
✓ push server can load mount info from database
✓ push server can connect to the Nextcloud server
✓ push server is a trusted proxy
✓ push server is running the same version as the app

This solution worked more than 20 times now.

I’d appreciate your feedback/ideas:

  • Does anybody have the same problem?
  • Are there maybe more changes to /etc/systemd/system/multi-user.target.wants/notify_push.service needed?
  • Does anybody know what commands clear-php-opcache in ncp-config includes? I couldn’t find anything about that. As a workaround, I would like to add that command to cron to schedule a reboot all x days.

For me this sometimes happened when the notify_push service was starting before the database service was running.

Adding the following line to the [UNIT] section in the systemd service file fixed it for me:

After=mariadb.service

This is how my complete file looks:

[Unit]
Description = Push daemon for Nextcloud clients
After=mariadb.service

[Service]
Environment=PORT=7867
Environment=NEXTCLOUD_URL=https://cloud.mydomain.tld
ExecStart=/path/to/nextcloud/apps/notify_push/bin/x86_64/notify_push /path/to/nextcloud/config/config.php
User=www-data

[Install]
WantedBy = multi-user.target

If the service stops or does not start at all, this may also be due to incorrect permissions for the notify_push binary. Make sure that it is executable and has the correct ownership and permissions:

sudo chmod ug+x path/to/nextcloud/apps/notify_push/bin/x86_64/notify_push

Hope that helps…

Thanks! Unfortunately adding After=mariadb.service in /etc/systemd/system/multi-user.target.wants/notify_push.service doesn’t fix the problem.

As far as I understand mysql and mariadb I already had that requirement with After=mysql.service in notify_push.service added.

Permissions look fine to me:

ls -la /var/www/nextcloud/apps/notify_push/bin/aarch64/
total 22520
drwxr-xr-x 2 www-data www-data 4096 Jul 27 06:31 .
drwxr-xr-x 5 www-data www-data 4096 Jul 27 06:31 …
-rwxr-xr-x 1 www-data www-data 23050936 Jul 27 06:31 notify_push

Since the update of notify_push last month to version 0.6.6 everything is working well again.