503-Error / HPB service down / notify push can't connect to the Nextcloud server

ℹ️ Support
1

Nextcloud version (eg, 20.0.5): 26.0.1.1
Operating system and version (eg, Ubuntu 20.04): Debian GNU/Linux 11. 6.1.21-v8+ (aarch64)
Apache or nginx version (eg, Apache 2.4.25): Apache 2.4.56-1~deb11u2
PHP version (eg, 7.4): 8.1

The issue you are facing: HPB service down / no connection to services on Nextcloud (503-errors)

Is this the first time you’ve seen this error? (Y/N): Y

NextcloudPi diagnostics 
NextcloudPi version  v1.52.0
NextcloudPi image    NextcloudPi_06-02-23
OS                   Debian GNU/Linux 11. 6.1.21-v8+ (aarch64)
automount            yes
USB devices          sda 
datadir              /media/WD_500GB
data in SD           no
data filesystem      ext2/ext3
data disk usage      83G/458G
rootfs usage         5.5G/58G
swapfile             /dev/zram3
dbdir                /var/lib/mysql
Nextcloud check      ok
Nextcloud version    26.0.1.1
HTTPD service        up
PHP service          up
MariaDB service      up
Redis service        up
HPB service          down
Postfix service      up
Internet check       ok
Public IPv4          ***REMOVED SENSITIVE VALUE***
Public IPv6          ***REMOVED SENSITIVE VALUE***
Port 80              open
Port 443             open
IP                   ***REMOVED SENSITIVE VALUE***
Gateway              ***REMOVED SENSITIVE VALUE***
Interface            eth0
Certificates         ***REMOVED SENSITIVE VALUE***
NAT loopback         yes
Uptime               2min
Nextcloud configuration 

{
    "system": {
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": {
            "1": "192.168.1.104",
            "4": "nextcloudpi.local",
            "5": "",
            "6": "nextcloudpi.lan",
            "2": "https:\/\/***REMOVED SENSITIVE VALUE***",
            "11": "109.250.64.111",
            "14": "nextcloudpi",
            "3": "nextcloudpi",
            "20": "***REMOVED SENSITIVE VALUE***",
            "21": "192.168.1.104"
        },
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "26.0.1.1",
        "overwrite.cli.url": "https:\/\/nextcloudpi\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "memcache.local": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "timeout": 0,
            "password": "***REMOVED SENSITIVE VALUE***"
        },
        "tempdirectory": "\/media\/WD_500GB\/tmp",
        "mail_smtpmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "preview_max_x": "2048",
        "preview_max_y": "2048",
        "jpeg_quality": "60",
        "overwriteprotocol": "https",
        "loglevel": "2",
        "log_type": "file",
        "maintenance": false,
        "theme": "",
        "mail_sendmailmode": "smtp",
        "mail_smtpsecure": "ssl",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "updater.release.channel": "stable",
        "data-fingerprint": "c43964c4e776c8cc78ea7cf4fb8f2ae1",
        "logfile": "\/media\/WD_500GB\/nextcloud.log",
        "app_install_overwrite": [
            "admin_notifications"
        ],
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "htaccess.RewriteBase": "\/"
    }
}
HTTPd logs 

[Wed Jun 28 19:53:25.126627 2023] [authz_host:error] [pid 1087:tid 547567423872] [remote 192.168.1.134:54168] AH01753: access check of 'localhost' to /img/toggle.svg failed, reason: unable to get the remote host name
[Wed Jun 28 19:53:25.127176 2023] [authz_host:error] [pid 1087:tid 547550605696] [remote 192.168.1.134:54168] AH01753: access check of 'localhost' to /img/defaults.svg failed, reason: unable to get the remote host name
[Wed Jun 28 19:53:25.127177 2023] [authz_host:error] [pid 1087:tid 547559014784] [remote 192.168.1.134:54168] AH01753: access check of 'localhost' to /img/poweroff.svg failed, reason: unable to get the remote host name
[Wed Jun 28 19:53:25.127399 2023] [authz_host:error] [pid 1087:tid 547542196608] [remote 192.168.1.134:54168] AH01753: access check of 'localhost' to /img/reboot.svg failed, reason: unable to get the remote host name
[Wed Jun 28 19:53:25.160284 2023] [authz_host:error] [pid 1087:tid 547533787520] [remote 192.168.1.134:54168] AH01753: access check of 'localhost' to /img/search.svg failed, reason: unable to get the remote host name, referer: https://192.168.1.104:4443/css/ncp.css
[Wed Jun 28 19:53:25.160487 2023] [authz_host:error] [pid 1087:tid 547500151168] [remote 192.168.1.134:54168] AH01753: access check of 'localhost' to /img/dashboard.svg failed, reason: unable to get the remote host name, referer: https://192.168.1.104:4443/css/ncp.css
[Wed Jun 28 19:53:25.160610 2023] [authz_host:error] [pid 1087:tid 547516969344] [remote 192.168.1.134:54168] AH01753: access check of 'localhost' to /img/nc-button.svg failed, reason: unable to get the remote host name, referer: https://192.168.1.104:4443/css/ncp.css
[Wed Jun 28 19:53:25.161041 2023] [authz_host:error] [pid 1087:tid 547508560256] [remote 192.168.1.134:54168] AH01753: access check of 'localhost' to /img/defaults-white.svg failed, reason: unable to get the remote host name, referer: https://192.168.1.104:4443/css/ncp.css
[Wed Jun 28 19:53:25.161848 2023] [authz_host:error] [pid 1087:tid 547592651136] [remote 192.168.1.134:54168] AH01753: access check of 'localhost' to /img/settings-white.svg failed, reason: unable to get the remote host name, referer: https://192.168.1.104:4443/css/ncp.css
[Wed Jun 28 19:53:25.161782 2023] [authz_host:error] [pid 1087:tid 547525378432] [remote 192.168.1.134:54168] AH01753: access check of 'localhost' to /img/logs.svg failed, reason: unable to get the remote host name, referer: https://192.168.1.104:4443/css/ncp.css
[Wed Jun 28 19:53:25.162820 2023] [authz_host:error] [pid 1087:tid 547516969344] [remote 192.168.1.134:54168] AH01753: access check of 'localhost' to /img/wizard.svg failed, reason: unable to get the remote host name, referer: https://192.168.1.104:4443/css/ncp.css
[Wed Jun 28 19:53:25.163002 2023] [authz_host:error] [pid 1087:tid 547533787520] [remote 192.168.1.134:54168] AH01753: access check of 'localhost' to /img/files.svg failed, reason: unable to get the remote host name, referer: https://192.168.1.104:4443/css/ncp.css
[Wed Jun 28 19:53:25.163335 2023] [authz_host:error] [pid 1087:tid 547491742080] [remote 192.168.1.134:54168] AH01753: access check of 'localhost' to /img/info-white.svg failed, reason: unable to get the remote host name, referer: https://192.168.1.104:4443/css/ncp.css
[Wed Jun 28 19:53:25.166397 2023] [authz_host:error] [pid 1087:tid 547634655616] [remote 192.168.1.134:54168] AH01753: access check of 'localhost' to /ncp-output.php failed, reason: unable to get the remote host name
[Wed Jun 28 19:53:25.169383 2023] [authz_host:error] [pid 1087:tid 547659833728] [remote 192.168.1.134:54168] AH01753: access check of 'localhost' to /ncp-launcher.php failed, reason: unable to get the remote host name
[Wed Jun 28 19:53:25.646759 2023] [authz_host:error] [pid 1087:tid 547668226432] [remote 192.168.1.134:54168] AH01753: access check of 'localhost' to /img/favicon.png failed, reason: unable to get the remote host name
[Wed Jun 28 19:53:44.978106 2023] [authz_host:error] [pid 1087:tid 547609469312] [remote 192.168.1.134:54168] AH01753: access check of 'localhost' to /ncp-launcher.php failed, reason: unable to get the remote host name
[Wed Jun 28 19:53:45.975462 2023] [authz_host:error] [pid 1087:tid 547643048320] [remote 192.168.1.134:54168] AH01753: access check of 'localhost' to /img/download.svg failed, reason: unable to get the remote host name
[Wed Jun 28 19:53:45.975806 2023] [authz_host:error] [pid 1087:tid 547575832960] [remote 192.168.1.134:54168] AH01753: access check of 'localhost' to /img/delete.svg failed, reason: unable to get the remote host name
[Wed Jun 28 19:53:45.976029 2023] [authz_host:error] [pid 1087:tid 547617870208] [remote 192.168.1.134:54168] AH01753: access check of 'localhost' to /ncp-launcher.php failed, reason: unable to get the remote host name
Database logs 

tail: cannot open '/var/log/mysql/*.log' for reading: No such file or directory
Nextcloud logs 

{"reqId":"vru70caTAOB1z8zpJvKs","level":2,"time":"2023-06-27T14:40:08+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE*** read error : Server error: `GET https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE***` resulted in a `503 Service Unavailable` response:\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>503 Service Unavailable</title>\n</head><body>\n<h1 (truncated...)\n","userAgent":"--","version":"26.0.1.1","data":{"app":"news"}}
{"reqId":"teZ38qy6BrJm9hzdsmN7","level":2,"time":"2023-06-27T15:45:10+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE*** read error : Server error: `GET https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE***` resulted in a `503 Service Unavailable` response:\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>503 Service Unavailable</title>\n</head><body>\n<h1 (truncated...)\n","userAgent":"--","version":"26.0.1.1","data":{"app":"news"}}
{"reqId":"Eu2f2eFn0ssaLRgG5910","level":2,"time":"2023-06-27T16:45:09+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE*** read error : Server error: `GET https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE***` resulted in a `503 Service Unavailable` response:\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>503 Service Unavailable</title>\n</head><body>\n<h1 (truncated...)\n","userAgent":"--","version":"26.0.1.1","data":{"app":"news"}}
{"reqId":"ZJsqrn1jaQwfjVxYCpKPcgAAAAQ","level":3,"time":"2023-06-27T18:30:06+00:00","remoteAddr":"109.250.64.111","user":"--","app":"core","method":"PROPFIND","url":"/remote.php/dav/files/***REMOVED SENSITIVE VALUE***/Photos/IMG_20230317_173503_121.jpg","message":"Redis server went away","userAgent":"Mozilla/5.0 (Android) Nextcloud-android/3.24.1","version":"26.0.1.1","exception":{"Exception":"RedisException","Message":"Redis server went away","Code":0,"Trace":[{"file":"/var/www/nextcloud/lib/private/Memcache/Redis.php","line":56,"function":"get","class":"Redis","type":"->"},{"file":"/var/www/nextcloud/lib/autoloader.php","line":154,"function":"get","class":"OC\\Memcache\\Redis","type":"->"},{"file":"/var/www/nextcloud/remote.php","line":62,"function":"load","class":"OC\\Autoloader","type":"->"},{"file":"/var/www/nextcloud/remote.php","line":174,"function":"handleException"}],"File":"/var/www/nextcloud/lib/private/Memcache/Redis.php","Line":56,"CustomMessage":"--"}}
{"reqId":"ZJsqrn1jaQwfjVxYCpKPcgAAAAQ","level":3,"time":"2023-06-27T18:30:07+00:00","remoteAddr":"109.250.64.111","user":"--","app":"core","method":"PROPFIND","url":"/remote.php/dav/files/***REMOVED SENSITIVE VALUE***/Photos/IMG_20230317_173503_121.jpg","message":"Redis server went away","userAgent":"Mozilla/5.0 (Android) Nextcloud-android/3.24.1","version":"26.0.1.1","exception":{"Exception":"RedisException","Message":"Redis server went away","Code":0,"Trace":[{"file":"/var/www/nextcloud/lib/private/Memcache/Redis.php","line":56,"function":"get","class":"Redis","type":"->"},{"file":"/var/www/nextcloud/lib/private/App/InfoParser.php","line":58,"function":"get","class":"OC\\Memcache\\Redis","type":"->"},{"file":"/var/www/nextcloud/lib/private/App/AppManager.php","line":506,"function":"parse","class":"OC\\App\\InfoParser","type":"->"},{"file":"/var/www/nextcloud/lib/private/App/AppManager.php","line":521,"function":"getAppInfo","class":"OC\\App\\AppManager","type":"->"},{"file":"/var/www/nextcloud/lib/private/legacy/OC_App.php","line":580,"function":"getAppVersion","class":"OC\\App\\AppManager","type":"->"},{"file":"/var/www/nextcloud/lib/private/legacy/OC_App.php","line":884,"function":"getAppVersion","class":"OC_App","type":"::"},{"file":"/var/www/nextcloud/lib/private/legacy/OC_Util.php","line":1224,"function":"shouldUpgrade","class":"OC_App","type":"::"},{"file":"/var/www/nextcloud/lib/public/Util.php","line":570,"function":"needUpgrade","class":"OC_Util","type":"::"},{"file":"/var/www/nextcloud/lib/private/legacy/OC_Template.php","line":113,"function":"needUpgrade","class":"OCP\\Util","type":"::"},{"file":"/var/www/nextcloud/lib/private/legacy/OC_Template.php","line":77,"function":"initTemplateEngine","class":"OC_Template","type":"::"},{"file":"/var/www/nextcloud/lib/private/legacy/OC_Template.php","line":321,"function":"__construct","class":"OC_Template","type":"->"},{"file":"/var/www/nextcloud/remote.php","line":92,"function":"printExceptionErrorPage","class":"OC_Template","type":"::"},{"file":"/var/www/nextcloud/remote.php","line":174,"function":"handleException"}],"File":"/var/www/nextcloud/lib/private/Memcache/Redis.php","Line":56,"CustomMessage":"--"}}
{"reqId":"ZJssaX1jaQwfjVxYCpKQYgAAABU","level":3,"time":"2023-06-27T18:37:29+00:00","remoteAddr":"192.168.1.134","user":"***REMOVED SENSITIVE VALUE***","app":"webdav","method":"POST","url":"/remote.php/dav/bulk","message":"Unknown error while seeking content","userAgent":"Mozilla/5.0 (Linux) mirall/3.9.0-20230616.170352.c6c6c61f5-1.0~jammy1 (Nextcloud, ubuntu-5.19.0-45-generic ClientArchitecture: x86_64 OsArchitecture: x86_64)","version":"26.0.1.1","exception":{"Exception":"Sabre\\DAV\\Exception","Message":"Unknown error while seeking content","Code":500,"Trace":[{"file":"/var/www/nextcloud/apps/dav/lib/BulkUpload/MultipartRequestParser.php","line":129,"function":"isAt","class":"OCA\\DAV\\BulkUpload\\MultipartRequestParser","type":"->"},{"file":"/var/www/nextcloud/apps/dav/lib/BulkUpload/BulkUploadPlugin.php","line":71,"function":"isAtLastBoundary","class":"OCA\\DAV\\BulkUpload\\MultipartRequestParser","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpPost","class":"OCA\\DAV\\BulkUpload\\BulkUploadPlugin","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":253,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":321,"function":"start","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/apps/dav/lib/Server.php","line":366,"function":"exec","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/apps/dav/appinfo/v2/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/remote.php","line":172,"args":["/var/www/nextcloud/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/nextcloud/apps/dav/lib/BulkUpload/MultipartRequestParser.php","Line":111,"message":"Unknown error while seeking content","exception":{},"CustomMessage":"Unknown error while seeking content"}}
{"reqId":"2fQ1jUdjohzGoq6zctle","level":2,"time":"2023-06-28T03:55:09+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE*** read error : Server error: `GET https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE***` resulted in a `503 Service Unavailable` response:\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>503 Service Unavailable</title>\n</head><body>\n<h1 (truncated...)\n","userAgent":"--","version":"26.0.1.1","data":{"app":"news"}}
{"reqId":"0SydFlo9m53abM4HIp8S","level":2,"time":"2023-06-28T05:00:09+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE*** read error : Server error: `GET https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE***` resulted in a `503 Service Unavailable` response:\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>503 Service Unavailable</title>\n</head><body>\n<h1 (truncated...)\n","userAgent":"--","version":"26.0.1.1","data":{"app":"news"}}
{"reqId":"xLMwEX7jl3kIzRC8hKQ6","level":2,"time":"2023-06-28T06:05:10+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE*** read error : Server error: `GET https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE***` resulted in a `503 Service Unavailable` response:\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>503 Service Unavailable</title>\n</head><body>\n<h1 (truncated...)\n","userAgent":"--","version":"26.0.1.1","data":{"app":"news"}}
{"reqId":"Dtxa1iUykirjQiSZ5roo","level":2,"time":"2023-06-28T07:10:08+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE*** read error : Server error: `GET https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE***` resulted in a `503 Service Unavailable` response:\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>503 Service Unavailable</title>\n</head><body>\n<h1 (truncated...)\n","userAgent":"--","version":"26.0.1.1","data":{"app":"news"}}
{"reqId":"0JveTxXTb6GFiaqEIfxD","level":2,"time":"2023-06-28T08:15:09+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE*** read error : Server error: `GET https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE***` resulted in a `503 Service Unavailable` response:\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>503 Service Unavailable</title>\n</head><body>\n<h1 (truncated...)\n","userAgent":"--","version":"26.0.1.1","data":{"app":"news"}}
{"reqId":"8yCeatLZ9GibatsmwETl","level":2,"time":"2023-06-28T09:20:10+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE*** read error : Server error: `GET https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE***` resulted in a `503 Service Unavailable` response:\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>503 Service Unavailable</title>\n</head><body>\n<h1 (truncated...)\n","userAgent":"--","version":"26.0.1.1","data":{"app":"news"}}
{"reqId":"SDncbcrZueFQs3xWyCrZ","level":2,"time":"2023-06-28T10:20:09+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE*** read error : Server error: `GET https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE***` resulted in a `503 Service Unavailable` response:\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>503 Service Unavailable</title>\n</head><body>\n<h1 (truncated...)\n","userAgent":"--","version":"26.0.1.1","data":{"app":"news"}}
{"reqId":"J2PpUdLvKQa0VKsEqbQN","level":2,"time":"2023-06-28T11:25:09+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE*** read error : Server error: `GET https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE***` resulted in a `503 Service Unavailable` response:\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>503 Service Unavailable</title>\n</head><body>\n<h1 (truncated...)\n","userAgent":"--","version":"26.0.1.1","data":{"app":"news"}}
{"reqId":"61FdFIGWNgcmJHLnq6Kl","level":2,"time":"2023-06-28T12:30:08+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE*** read error : Server error: `GET https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE***` resulted in a `503 Service Unavailable` response:\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>503 Service Unavailable</title>\n</head><body>\n<h1 (truncated...)\n","userAgent":"--","version":"26.0.1.1","data":{"app":"news"}}
{"reqId":"8PKF3ObDGrcusfbP1bVH","level":2,"time":"2023-06-28T13:30:09+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE*** read error : Server error: `GET https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE***` resulted in a `503 Service Unavailable` response:\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>503 Service Unavailable</title>\n</head><body>\n<h1 (truncated...)\n","userAgent":"--","version":"26.0.1.1","data":{"app":"news"}}
{"reqId":"nRyZAqjgcRxdzsnKLFIK","level":2,"time":"2023-06-28T14:35:11+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE*** read error : Server error: `GET https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE***` resulted in a `503 Service Unavailable` response:\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>503 Service Unavailable</title>\n</head><body>\n<h1 (truncated...)\n","userAgent":"--","version":"26.0.1.1","data":{"app":"news"}}
{"reqId":"9ZJRh99FTqyysNgBvEVt","level":2,"time":"2023-06-28T15:35:13+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE*** read error : Server error: `GET https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE***` resulted in a `503 Service Unavailable` response:\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>503 Service Unavailable</title>\n</head><body>\n<h1 (truncated...)\n","userAgent":"--","version":"26.0.1.1","data":{"app":"news"}}
{"reqId":"lmnoxe5LsrcYROcsTX7P","level":2,"time":"2023-06-28T16:40:07+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE*** read error : Server error: `GET https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE***` resulted in a `503 Service Unavailable` response:\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>503 Service Unavailable</title>\n</head><body>\n<h1 (truncated...)\n","userAgent":"--","version":"26.0.1.1","data":{"app":"news"}}
{"reqId":"hy9Dh4jyABNxTNIt8SKX","level":2,"time":"2023-06-28T17:40:09+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE*** read error : Server error: `GET https://***REMOVED SENSITIVE VALUE***/index.php/apps/activity/rss.php?token=***REMOVED SENSITIVE VALUE***` resulted in a `503 Service Unavailable` response:\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>503 Service Unavailable</title>\n</head><body>\n<h1 (truncated...)\n","userAgent":"--","version":"26.0.1.1","data":{"app":"news"}}

As the HPB service is connected to notify_push I tried sudo systemctl restart notify_push and sudo -u www-data php /var/www/nextcloud/occ notify_push:self-test. The result is:

✓ redis is configured
✓ push server is receiving redis messages
✓ push server can load mount info from database
🗴 push server can't connect to the Nextcloud server
  Invalid response when getting test cookie

The problem persists after sudo shutdown -r.

Any suggestions and ideas are welcome!

2

Here some more informations:

The output of sudo -u www-data php /var/www/nextcloud/occ notify_push:setup is:

Push binary seems to be running already
🗴 failed to run self-test.
test output: ✓ redis is configured
             🗴 using unencrypted http for push server is strongly discouraged
             🗴 push server url is set to localhost, the push server will not be reachable from other machines
             ✓ push server is receiving redis messages
             ✓ push server can load mount info from database
             🗴 push server can't connect to the Nextcloud server
               Invalid response when getting test cookie
             
  See the steps in the README for manual setup instructions: https://github.com/nextcloud/notify_push

/etc/systemd/system/notify_push.service looks like this:

[Unit]
Description = Push daemon for Nextcloud clients
After=mysql.service
After=redis.service
Requires=redis.service

[Service]
Environment=PORT=7867
Environment=NEXTCLOUD_URL=https://localhost
ExecStart="/var/www/nextcloud/apps/notify_push/bin/aarch64/notify_push" --allow-self-signed /var/www/nextcloud/config/config.php
User=www-data
Restart=on-failure
RestartSec=20

[Install]
WantedBy = multi-user.target

Changing Environment=NEXTCLOUD_URL=https://localhost to ...=https://my.domain.com doesn’t fix it.

Any ideas?

3

what do you get when using commands

systemctl status notify_push.service

and

systemctl start notify_push.service

Do you get any errors concerning access rights for the service file?
I ask because i had a similar prob and checked out that

/var/www/nextcloud/apps/notify_push/bin/aarch64/notify_push

was not accessible from /etc/systemd/multi-user.target.wants/notify_push.service

4

systemctl status notify_push.service looks fine to me:

● notify_push.service - Push daemon for Nextcloud clients
     Loaded: loaded (/etc/systemd/system/notify_push.service; enabled; vendor p>
     Active: active (running) since Thu 2023-06-29 07:53:57 CEST; 1min 48s ago
   Main PID: 3177 (notify_push)
      Tasks: 5 (limit: 3933)
        CPU: 36ms
     CGroup: /system.slice/notify_push.service
             └─3177 /var/www/nextcloud/apps/notify_push/bin/aarch64/notify_push>

systemctl start notify_push.service doesn’t have any output.

ls -la /var/www/nextcloud/apps/notify_push/bin/aarch64/ is:

total 22520
drwxr-xr-x 2 www-data www-data     4096 May 16 20:42 .
drwxr-xr-x 5 www-data www-data     4096 May 16 20:42 ..
-rwxr-xr-x 1 www-data www-data 23050936 May 16 20:42 notify_push
5

what does

sudo -u www-data php /var/www/nextcloud/occ app:enable notify_push

and after that try again to test notify_push

and of course php8.1-redis should be installed and running, but i think that is already be done

6

It’s still the same:

pi@nextcloudpi:~ $ sudo -u www-data php /var/www/nextcloud/occ app:enable notify_push
notify_push already enabled

pi@nextcloudpi:~ $ sudo -u www-data php /var/www/nextcloud/occ notify_push:self-test
✓ redis is configured
🗴 push server url is set to localhost, the push server will not be reachable from other machines
✓ push server is receiving redis messages
✓ push server can load mount info from database
🗴 push server can't connect to the Nextcloud server
  Invalid response when getting test cookie

php8.1-redis is installed an running.

7

my ExecStart does not contain any “”

what if you change the parameter --allow self-signed
to the form of the manual
→ notify_push/README.md at main · nextcloud/notify_push · GitHub

ALLOW_SELF_SIGNED=true

for the Server URL i use https://public_domain_name_NC-Server e.g. https://my.nextcloud.com

8

When I get error 503, then clean opcache always helps as opcache is closely linked to php. I’m not sure if this helps but it’s worth to give it a try.

9

Thanks! I tried that one, it doesn’t change anything. notify_push is still down and 503 on all services. I changed the config back to 503-Error / HPB service down / notify push can't connect to the Nextcloud server - #2 by marc_f.

10

Thanks for this idea. I tried that one already, unfortunately it didn’t fix the problem…

11

Okay. Things get weird. I just tried systemctl restart php8.1-fpm, now notify_push and Nextcloud are working fine - even after two reboots. I’ll monitor this the next days and will come back here…

12

After a scheduled reboot this morning notify_push was down and 503 on all services again:

sudo -u www-data php /var/www/nextcloud/occ notify_push:self-test
✓ redis is configured
✓ push server is receiving redis messages
✓ push server can load mount info from database
🗴 push server can't connect to the Nextcloud server
  Invalid response when getting test cookie

sudo -u www-data php /var/www/nextcloud/occ notify_push:setup didn’t help:

Push binary seems to be running already
🗴 failed to run self-test.
test output: ✓ redis is configured
             🗴 using unencrypted http for push server is strongly discouraged
             🗴 push server url is set to localhost, the push server will not be reachable from other machines
             ✓ push server is receiving redis messages
             ✓ push server can load mount info from database
             🗴 push server can't connect to the Nextcloud server
               Invalid response when getting test cookie
             
  See the steps in the README for manual setup instructions: https://github.com/nextcloud/notify_push

systemctl restart php8.1-fpm didn’t help, too.

I ran clear-php-opcache via sudo ncp-config, and now notify_push was working:

sudo -u www-data php /var/www/nextcloud/occ notify_push:self-test
✓ redis is configured
✓ push server is receiving redis messages
✓ push server can load mount info from database
✓ push server can connect to the Nextcloud server
✓ push server is a trusted proxy
✓ push server is running the same version as the app

I could reproduce this two times: After a reboot notify_push is down and all services respond 503. When clearing php-opcache manually, everything works fine. So @kjathome was right! Thanks again!

I guess on reboot notify_push somehow runs to early. Until this is fixed in NextCloudPi I see two options:

  1. delay the start of notify_push by adding ExecStartPre=/bin/sleep 60 in the [Service]-section of /etc/systemd/system/notify_push.service
  2. clear php-opcache after reboot with cron - which command would that be?

I’d appreciate some comments on these two options. Thanks alot!

13

With systemctl restart php8.1-fpm you flushed opcache as well and forced it to be rebuild on the first incoming request. (Source: How To Clear PHP’s Opcache)

I would try to set in /etc/systemd/system/multi-user.target.wants/notify_push.service the following line

After=nginx.service php8.1-fpm.service mysql.service redis.service

And comment out or delete the second “After=redis.service” line.

When you forget nginx.service it should be that notify_push.service starts to run before the webserver = Nextcloud Server is up and running.

14

I would try to set in /etc/systemd/system/multi-user.target.wants/notify_push.service the following line

After=nginx.service php8.1-fpm.service mysql.service redis.service

And comment out or delete the second “After=redis.service” line.

That works! Thank you, @chrissi55! I restarted the server five times. notify_push runs normal and no more 503-errors so far.
On my server it’s apache2.service instead of nginx.service.
/etc/systemd/system/multi-user.target.wants/notify_push.service now looks like this:

[Unit]
Description = Push daemon for Nextcloud clients
After=apache2.service php8.1-fpm.service mysql.service redis.service
#After=mysql.service
#After=redis.service
Requires=redis.service

[Service]
Environment=PORT=7867
Environment=NEXTCLOUD_URL=https://localhost
ExecStart="/var/www/nextcloud/apps/notify_push/bin/aarch64/notify_push" --allow>
User=www-data
Restart=on-failure
RestartSec=20

[Install]
WantedBy = multi-user.target

With systemctl restart php8.1-fpm you flushed opcache as well and forced it to be rebuild on the first incoming request. (Source: How To Clear PHP’s Opcache)

That’s good to know. Maybe I was too impatient when I tried this first to clear php-opcache. I’ll add systemctl restart php8.1-fpm to cron as php-opcache is kind of a bottleneck with Nextcloud.

15

The problem still persists. As NCP and php had updates and I’ve more experiences I opened a new thread: Notify push (sometimes) causes 503-Error / HPB service down on reboot