Raspberry Pi crashes caused by notifyPush setup?

[hoshymann]

NextcloudPi version: 1.52.2
Nextcloud version (eg, 20.0.5): 26.0.3.2
Operating system and version (eg, Ubuntu 20.04): Debian GNU/Linux 11.7 (bullseye)
Apache or nginx version (eg, Apache 2.4.25): Apache 2.4.56
PHP version (eg, 7.4): 8.1.21

My new installation of NextcloudPi on my RasPi 3 Model B continues to crash occasionally. If that happens, I get a ping the IP but can neither reach it through the web interface nor via SSH. The only fix is a hard reboot (disconnect power).

In the logs (NC and NCP) I can not find anything obviously related to the crashes. What stands out to me though is that the HPB service is always down according to the NCP web interface.
The NCP log shows a reoccurring error related to the push server:

[ nc-limits ] (Thu 27 Jul 13:51:46 CEST 2023)
Using 715339776 for PHP max memory
Installing template 'php/90-ncp.ini.sh'...
System config value trusted_domains => 3 set to string [MY_DOMAIN].de
Using 16 PHP threads
Installing template 'php/pool.d.www.conf.sh'...
System config value overwrite.cli.url set to string https://[MY_DOMAIN].de/
System config value trusted_proxies => 11 set to string 127.0.0.1
Installing template 'mysql/91-ncp.cnf.sh'...
System config value trusted_proxies => 12 set to string ::1
System config value trusted_proxies => 13 set to string [MY_DOMAIN].de

System config value trusted_proxies => 14 set to string [MY_PUBLIC_IP]
Setup notify_push (attempt 1/5)
✓ redis is configured
🗴 push server is not receiving redis messages (received 641554094, got 0)
Setup notify_push (attempt 2/5)
✓ redis is configured
🗴 push server is not receiving redis messages (received 191475829, got 0)
Setup notify_push (attempt 3/5)
✓ redis is configured
🗴 push server is not receiving redis messages (received 565106433, got 0)
Setup notify_push (attempt 4/5)
✓ redis is configured
🗴 push server is not receiving redis messages (received 1071285661, got 0)
Setup notify_push (attempt 5/5)
✓ redis is configured
🗴 push server is not receiving redis messages (received 67272037, got 0)

It looks to me like the system is trying to initialize the push server with [MY_DOMAIN].de but my DynDNS is reachable via cloud.[MY_DOMAIN].de

I deactivated the Client Push app in NC a few hours ago and have not since gotten this error.
It is too early to say if that also avoids the crashes.

Does the start of the push server failing because of the wrong domain it is set to? And how can I change the domain used for that?
Could all that be causing my crashes or how could I find the cause for those otherwise?

Thank you for your time and help!
Mark

[hoshymann]

Update: Since I deactivated the Client Push app through the Nextcloud web interface the mentioned error did in fact not reoccur and the system did not crash.

As I understand, the Client Push app greatly improves performance and reduces use of resources on both server and client machines, correct?
In that case it seems desirable to get this app working again.

Does anyone have a suggestion on how I could fix my problem with the push server?

Thanks
Mark

[chrissi55]

one question: your redis-server and redis-sentinel are both up and running?
…and reachable**?

For further info see **
https://arnowelzel.de/en/client-push-for-nextcloud

[Gorf]

Running 64bit or 32bit OS?

Depending on what is running there the Raspi 3b also might get short of RAM .

[hoshymann]

@chrissi55
Well, according to my log above it looks to me like redis-server is configured but not reachable:

✓ redis is configured
🗴 push server is not receiving redis messages (received 641554094, got 0)

htop shows several instances of the redis-server running:

Is there another way to check wheather it is reachable?
What is the redis-sentinel and how do I check if it is working?

Please excuse my ignorance. Like I said, I did not set this system up myself but am just using a preconfigurated NextcloudPi image.

@Gorf
My Linux kernel is 64 bit.
But at least currently there seem to be 450 M of RAM availabe

Could it still be that each time the redis-server tries to set up, the RAM is exceeded?

Couldn´t the problem be this line:

System config value overwrite.cli.url set to string https://[MY_DOMAIN].de/

Because my DynDNS that redirects to my RasPi is reachable via https://cloud.[MY_DOMAIN].de.
Or does that not matter?

[chrissi55]

have you read the linked article?

I had to do the following as well

The first step is to install and activate the Redis server as follows:

sudo apt install redis-server sudo systemctl start redis-server

The default configuration of Redis usually only allows local access. To be on the safe side, however, you should check whether the following line is present in /etc/redis/redis.conf:

bind 127.0.0.1 ::1

If this is not the case, or the line is commented out, this should be corrected and the Redis server restarted afterwards.

Then my config for redis in Nextcloud is as follows
(so i don’t use the ip+port number but instead the unix socket to connect to redis - a performance tip from the community that worked for a long time now for me

  'redis' => 
  array (
    'host' => '/var/run/redis/redis.sock',
    'port' => 0,
    'dbindex' => 0,
    'timeout' => 1.5,
  ),

and

‘overwrite.cli.url’ => ‘https://my.cloud.com’, #(the FQDN under which my NC Server is reachable from the internet)

I don’t know which ssl certs you are using (Let’s Encrypt or own??)
For notify_push it makes a difference as well

my service file for notify_push is as follows

[Unit]
Description = Push daemon for Nextcloud clients
After=nginx.service php8.2-fpm.service postgresql.service redis.service

[Service]
Environment = PORT=7867
Environment = NEXTCLOUD_URL=https://my.cloud.com
ExecStart = /var/www/nextcloud/apps/notify_push/bin/x86_64/notify_push /var/www/nextcloud/config/config.php
User=www-data

[Install]
WantedBy = multi-user.target

If you are using self signed certs please read this tutorial

GitHub - nextcloud/notify_push: Update notifications for nextcloud clients

and also from the linked article above:

In order for Nextcloud to accept incoming requests from the push server service, the IP address of the server must be added to the list of “trusted proxies” in config/config.php as follows. Instead of , enter the IP address of the server.

‘trusted_proxies’ =>
array (
0 => ‘Nextcloud IP address’,
1 => ‘127.0.0.1’,
2 => ‘::1’,
),

[hoshymann]

Thank you very much for the detailed description of your approach!

I did read the linked article before but was afraid to screw up my Nextcloud completely by meddling with the configuration.
With your additional information I felt brave enough and went through the steps described in the article.

My config/config.php looked like this:

'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
    'host' => '/var/run/redis/redis.sock',
    'port' => 0,
    'timeout' => 0.0,
    'password' => '[removed sensible(?) information]',
  ),

Compared to yours it has a different timeout (probaly the default one), dbindex is missing and instead it states some password (upper and lower case letters, numbers, special chars).
I added the 'dpindex' => 0, line and adjusted the timeout just in case and changed
'overwrite.cli.url' => 'https://MY_DOMAIN.de/',
to
'overwrite.cli.url' => 'https://cloud.MY_DOMAIN.de/',

After all this, the Nextcloud web interface is still working which the article rates as a sign of success. However, when I type in $ redis-cli I get this output:

Could not connect to Redis at 127.0.0.1:6379: Connection refused
not connected>

Something seems to be wrong here but I dont know what so I kept going with the next step.

I added cloud.MY_DOMAIN.de to trusted_proxies in config/config.php or should I use the RasPi´s local IP address here?

I am using Let´s Encrypt for SSL certificates and /etc/systemd/system/notify_push.service already existed on my system and looked like this:

[Unit]
Description = Push daemon for Nextcloud clients
After=mysql.service
After=redis.service
Requires=redis.service

[Service]
Environment=PORT=7867
Environment=NEXTCLOUD_URL=https://localhost
ExecStart="/var/www/nextcloud/apps/notify_push/bin/aarch64/notify_push" --allow-self-signed /var/www/nextcloud/config/config.php
User=www-data
Restart=on-failure
RestartSec=20

[Install]
WantedBy = multi-user.target

The main difference seems to be that NEXTCLOUD_URL is set to https://localhost in my case. Other than that I just have a different webserver and fewer dependencies, correct?
Should I change NEXTCLOUD_URL to my domain?

In the end, I can enable and start the notify_push service but when I run the self test mentioned in the article (only works as www-data for me):
sudo -u www-data php /var/www/nextcloud/occ notify_push:self-test
I still get the same result as in the ncp.log:

Everything I did not mention was already set up exactly as mentioned in the article.

What am I doing wrong?

[hoshymann]

Ha! Looks like I got it to work after all

After reeding about a setup wizard I started it with this:
sudo -u www-data php /var/www/nextcloud/occ notify_push:setup

Although it merely seemd to acknoledge that everything was already set up and running…

Push binary seems to be running already
✓ push server seems to be functioning correctly.
Reverse proxy seems to be configured already
✓ reverse proxy seems to be setup correctly.
  configuration saved

…the self test afterwards was successfull:

 $ sudo -u www-data php /var/www/nextcloud/occ notify_push:self-test
✓ redis is configured
✓ push server is receiving redis messages
✓ push server can load mount info from database
✓ push server can connect to the Nextcloud server
✓ push server is a trusted proxy
✓ push server is running the same version as the app

Still I am unsure if I should change the NEXTCLOUD_URL in the notify_push.service.
Or anything else?
What do you think?

EDIT:
redis-cli still returns Connection refused like before.
So might there still be an issue?

[chrissi55]

@hoshymann

'password' => '[removed sensible(?) information]',

I had this parameter in earlier config settings as well and commented it out.

Try to comment this line out in the NC config and restart nc, redis, php and try again.
I am not very sure but i think when talking via unix socket the password is obsolete?!

And check as well if www-data is part of redis

→ usermod -a -G redis www-data

For the NEXTCLOUD_URL see my info above → i have set my FQDN of the NC Server e.g. https://my.cloud.com = www address to reach my NC Instance from the internet

Another tip of the community was to flush redis from time to time
see: FLUSHALL | Redis

a little bash script can do this as well

#!/bin/bash
redis-cli -s /var/run/redis/redis.sock <<EOF
FLUSHALL
quit
EOF

[hoshymann]

Hey chrissi55,

thank you for your continued support an patience with me
I tried the suggested changes.

If I comment out the password Nextcloud stops working completely and I get plenty of those errors in the ncp.log:

RedisException: NOAUTH Authentication required.

So this does not seem to be a good idea

If I change NEXTCLOUD_URL to cloud.MY_DOMAIN.de I get this error on the redis self test:

✓ redis is configured
✓ push server is receiving redis messages
✓ push server can load mount info from database
✓ push server can connect to the Nextcloud server
🗴 push server is not a trusted proxy, please add '87.123.XX.XX' to the list of trusted proxies or configure any existing reverse proxy to forward the 'x-forwarded-for' send by the push server.
  See https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/reverse_proxy_configuration.html#defining-trusted-proxies for how to set trusted proxies.
  The following trusted proxies are currently configured: "cloud.MY_DOMAIN.de", "127.0.0.1", "::1", "cloud.MY_DOMAIN.de", "81.169.XXX.XX"
    of which the following seem to be invalid: "cloud.MY_DOMAIN.de", "cloud.MY_DOMAIN.de"
  The following x-forwarded-for header was received by Nextcloud: 1.2.3.4
    from the following remote: 87.123.XX.XX

  If you're having issues getting the trusted proxy setup working, you can try bypassing any existing reverse proxy
  in your setup by setting the `NEXTCLOUD_URL` environment variable to point directly to the internal Nextcloud webserver url
  (You will still need the ip address of the push server added as trusted proxy)

Do I always have to add my current public IP to trusted proxies? That can’t be right because it changes all the time, right? For now I changed NEXTCLOUD_URL back to local host and eveything seems to be working a before (redis self test passes, redis-cli comes back with Connection refused)

Because of that I am not able to run FLUSHALL so far.

So what am I getting wrong about the trusted proxy settings?

Thank you!
Mark

[chrissi55]

@hoshymann

If I comment out the password Nextcloud stops working completely and I get plenty of those errors in the ncp.log:

RedisException: NOAUTH Authentication required.

if you comment out the redis password in nc config you should do the same in redis.conf as well.
I have not mentioned that but it should bring your NC up without any pass !!

Go to /etc/redis/redis.conf and look into the security section …you should have the entry like

requirepass xxxxxxxxxxxxxxxxxxxxxxx

and comment it out as well. Then restart nc, redis, php …

@hoshymann

Do I always have to add my current public IP to trusted proxies?

of course not.
My trusted proxy looks like this

'trusted_proxies' => 
    array (
     0 => 'local-IP-of-my-Reverse-Proxy',
),

that is because in my case nc is running under eg 192.168.100.5 and the reverse proxy is localy under 192.168.100.1 … if you have both on the same server, 127.0.0.1 and / or local network IP of the nc = proxy server should do the same. (incl. ::1 if IPv6 support is enabled)

@hoshymann

(redis self test passes, redis-cli comes back with Connection refused)

same prob because of the not commented out pass in redis.conf

[hoshymann]

Sorry, I did not gather that from your previous post.
But now since I commented out the password in the Nextcloud config as well as requirepass in /etc/redis/redis.config Nextcloud is working as usual.

After adding my Raspberry Pi´s local IP address to the trusted proyies in the Nextcloud config (127.0.0.1 was already there) changing the NEXTCLOUD_URL to cloud.MY_DOMAIN.de does not cause problems anymore.

BUT any call to redis-cli still returns Connection refused:

[chrissi55]

/var/run should link directly to /run/…

so have a look at ls -la /run/redis and ls -la /run/redis/redis.sock if redis.sock is owned by redis:redis

Is www-data user part of the redis group

Question - Redis Unix Socket Connection Permission denied, TCP Connection...

[hoshymann]

Damn, it looks like some of my changes from yesterday caused my Raspberry Pi to crash again.
Since I am not going to be home for the next few weeks I will need to find someone to go to my house to do a hard reset for me.

So it will take a few days until I can inverstigat this issue any further.

I wonder what caused the Pi to crash again. Until now I thought the initial problem were the repeated errors during the initialization of redis but those errors were fixed when I left the system yesterday.

[hoshymann]

Ok, here we go again.
I am back at home and ready further persue this redis issue.

My problem is still, that any call to redis-cli returns Permission denied.

I checked what @chrissi55 suggested:

The ls -la outputs look like this

so it looks like redis.sock is owned by redis:redis who can read, write and execute.

The redis line in /etc/group looks like this

redis:x:118:www-data

so it looks like www-data is part of the redis group.

[chrissi55]

also look here

Permission denied for Redis?

The socket must be accessible for www-data. You first have to change the chmod of the socket to 770 that the redis group can access it, make changes in /etc/redis/redis.conf: unixsocketperm 770 and then add www-data to the redis group in /etc/group: redis:x:110:www-data

[hoshymann]

Ok, maybe I was missing a reboot.
Now if I run sudo redis-cli -s /var/run/redis/redis.sock it works.

Thank you for your great help!