Nextcloud Site Down, possible Redis issue. Internal Server Error

Hello,
I keep having issues with my cloud site. It was up and running and last week I restarted it and it failed shutting down due to REDIS issue, then it had trouble booting up because of REDIS. I noticed both MariaDB and REDIS trouble. I don’t have much experience with this and if you could assist me on how to troubleshoot or resolve, that would be appreciated.
Nextcloud Server version 30.0.6.2:
Apache/2.4.58 (Ubuntu)
Ubuntu 24.04.2 LTS
PHP version 8.3.6
Is this the first time you’ve seen this error?
No but I haven’t resolved it, and the site did work during the error at some points. I don’t know what changed.

  • When did this problem seem to first start?
    Recently, but it’s a new server
  • Installation method (e.g. AlO, NCP, Bare Metal/Archive, etc.)
    Hub

Summary of the issue you are facing:

The cloud server stopped being accessible. I get REDIS errors on shutdown.
Internal Server Error

The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.

Steps to replicate it (hint: details matter!):

  1. Log on to site, it’s inaccessible.

    GNU nano 7.2 /path/config.php *

<?php
$CONFIG = array (
  'instanceid' => '',
  'passwordsalt' => '/',
  'secret' => '',
  'trusted_domains' =>
   [
    'localhost',
    '.com',
    '10.0.0.43',
    'www.redacted.com'
  ],

 'datadirectory' => '/mnt',
  'tempdirectory' => '/mnt/A',
  'dbtype' => 'mysql',
  'memcache.local' => '\OC\Memcache\Redis',
  'memcache.locking' => '\OC\Memcache\Redis',
  'redis' => array(
   'host' => 'localhost',
    'port' => 6379,
),
  'version' => '30.0.6.2',
  'overwrite.cli.url' => 'https://.com',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => '',
  'dbpassword' => '',
'installed' => true,
  'maintenance_window_start' => '01:00',
  'default_phone_region' => 'US',
  'app_install_overwrite' => array (
    0 => 'backup'
  ),
  'maintenance' => false,
  'backgroundjobs.mode' => 'cron',
  'mail_smtpmode' => 'smtp',
  'mail_sendmailmode' => 'smtp',
  'mail_from_address' => 'cloud',
  'mail_domain' => '.com',
  'mail_smtphost' => '.com',
  'mail_smtpport' => '587',
  'mail_smtptimeout' => 30,
  'mail_smtpauth' => 1,
  'mail_smtpname' => '.com',
  'mail_smtppassword' => '',
);

Job for mariadb.service failed because the control process exited with error code.
See “systemctl status mariadb.service” and “journalctl -xeu mariadb.service” for details.

ps aux | grep mariadb
  0.0  0.0     pts/0    S+   09:19   0:00 grep --color=auto mariadb


sudo systemctl restart redis
Job for redis-server.service failed because the control process exited with error code.
See "systemctl status redis-server.service" and "journalctl -xeu redis-server.service" for details.

ps aux | grep redis
green       0.0  0.0    pts/0    S+   09:19   0:00 grep --color=auto redis


redis-cli -h localhost -p 6379 ping
Could not connect to Redis at localhost:6379: Connection refused

sudo systemctl status redis
× redis-server.service - Advanced key-value store
     Loaded: loaded (/usr/lib/systemd/system/redis-server.service; enabled; pre>
     Active: failed (Result: exit-code) since Tue 2025-04-01 09:18:51 EDT; 8min>
       Docs: http://redis.io/documentation,
             man:redis-server(1)
    Process: ExecStart=/usr/bin/redis-server /etc/redis/redis.conf --sup>
   Main PID: (code=exited, status=1/FAILURE)
        CPU: 42ms

Apr 01 09:18:51 systemd[1]: redis-server.service: Scheduled r>
Apr 01 09:18:51 systemd[1]: redis-server.service: Start reque>
Apr 01 09:18:51 systemd[1]: redis-server.service: Failed with>
Apr 01 09:18:51 systemd[1]: Failed to start redis-server.serv>
lines 1-13/13 (END)


sudo journalctl -xeu redis-server
░░ The job identifier is and the job result is failed.
Apr 01 09:18:51 systemd[1]: redis-server.service: Scheduled r>
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ Automatic restarting of the unit redis-server.service has been scheduled, as>
░░ the configured Restart= setting for the unit.
Apr 01 09:18:51 systemd[1]: redis-server.service: Start reque>
Apr 01 09:18:51 systemd[1]: redis-server.service: Failed with>
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit redis-server.service has entered the 'failed' state with result 'ex>
Apr 01 09:18:51 systemd[1]: Failed to start redis-server.serv>
░░ Subject: A start job for unit redis-server.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit redis-server.service has finished with a failure.
░░ 
░░ The job identifier is and the job result is failed.

Do you see more Information if you run redis-server on commandline?

To avoid the problem, you could first deactivate Redis (documentation). After that you can solve your problem.

Okay, I was able to get MariaDB and Redis working and running normal. But the site is still down.

Internal Server Error

The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.

Actually, upon rebooting I had a REDIS issue still:

Job redis-server.service/stop running Error trying to save the DB, can’t exit.

Upon boot-up:
Dependency failed for sssd-nss.socket,
Dependency failed for sssd-autofs.socket,
Dependency failed for sssd-pac.socket,
Dependency failed for sssd-pam-priv.socket,
Dependency failed for sssd-pam.socket,
Dependency failed for sssd-ssh.socket,
Dependency failed for sssd-sudo.socket

I don’t see if those sockets would matter to this.
But like I mentioned, the site did work with these issues prior.

Sounds like you don’t need the sssd service. If not you can stop and disable it. Or remove it completely.

Fixing redis was a permissions error. The site was still down.
Now the site is working on and off, but I didn’t change anything.

redis still has the reboot error, but I don’t want to reboot without figuring out why it can’t save to the db.

If I restart redis:
redis-server.service - Advanced key-value store
Loaded: loaded (redis-server.service; enabled; pre>
Active: deactivating (stop-sigterm) since Wed 2025-04-02 13:35:43 EDT; 1mi>
Main PID: 28967 (redis-server)
Status: “Error trying to save the DB, can’t exit.”
Tasks: 5 (limit: 9112)
Memory: 3.6M (peak: 4.4M)
CPU: 2.172s
CGroup: /system.slice/redis-server.service
└─28967 “/usr/bin/redis-server 127.0.0.1:6379”

then kill the PID
sudo kill -9 28967

then
sudo systemctl status redis-server.service
gets me:

redis-server.service - Advanced key-value store
Loaded: loaded (/usr/lib/systemd/system/redis-server.service; enabled; pre>
Active: active (running) since Wed 2025-04-02 13:42:00 EDT; 5s ago

sudo systemctl restart apache2

Now the site temporarily works. But Redis just cannot restart. and I still am not sure what causes the site to fail. The storage is okay for the entire drive. But is it maxing out a folder size maybe? I’m at a loss on what to do

log on redis
sudo tail -f /var/log/redis/redis-server.log
30898:C 02 Apr 2025 13:48:55.076 # Failed opening the temp RDB file temp-30898.rdb (in server root dir /mnt/storage/redis) for saving: Read-only file system
30645:M 02 Apr 2025 13:48:55.176 # Background saving error
30645:M 02 Apr 2025 13:49:01.090 * 100 changes in 300 seconds. Saving…
30645:M 02 Apr 2025 13:49:01.090 * Background saving started by pid 30901
30901:C 02 Apr 2025 13:49:01.090 # Failed opening the temp RDB file temp-30901.rdb (in server root dir /mnt/storage/redis) for saving: Read-only file system
30645:M 02 Apr 2025 13:49:01.191 # Background saving error
30645:M 02 Apr 2025 13:49:07.005 * 100 changes in 300 seconds. Saving…
30645:M 02 Apr 2025 13:49:07.005 * Background saving started by pid 30903
30903:C 02 Apr 2025 13:49:07.006 # Failed opening the temp RDB file temp-30903.rdb (in server root dir /mnt/storage/redis) for saving: Read-only file system
30645:M 02 Apr 2025 13:49:07.106 # Background saving error
30645:M 02 Apr 2025 13:49:13.021 * 100 changes in 300 seconds. Saving…
30645:M 02 Apr 2025 13:49:13.021 * Background saving started by pid 30909
30909:C 02 Apr 2025 13:49:13.021 # Failed opening the temp RDB file temp-30909.rdb (in server root dir /mnt/storage/redis) for saving: Read-only file system
30645:M 02 Apr 2025 13:49:13.122 # Background saving error

Additional info:

mount | grep /mnt/drive
/dev/sdb1 on /mnt/drive type ext4 (rw,relatime)

Is port 6379 closed or open in your firewall? And in redis.conf how are these lines set?

bind
requirepass
dir

I’m starting to think there’s a security problem.

1 Like

bind 127.0.0.1 ::1
dir /mnt/drive/redis

I’ll check on 6379

thank you for the suggestion

I noticed it was able to Pong back the 6379 for a few seconds, and then would error out and then the site would go down.

On immediate startup:
redis-cli -h localhost -p 6379 ping
PONG

After a few seconds:
redis-cli -h localhost -p 6379 ping
(error) MISCONF Redis is configured to save RDB snapshots, but it’s currently unable to persist to disk.

I changed dir to a different drive and it’s working (for now).
I had the correct permissions (read,write) on the directory but I can’t find why it was not working correctly with the one it was set to at first.

So this is fixed, but I still will evaluate in case the site goes down again.

I’m concerned about disk space on the new directory location. It only has 4gb which is why I was avoiding it. Any suggestions? Is that enough if I use it for large files often?

So you’re now seeing appendonly.aof and dump.rdb in the new directory? Maybe the previous one didn’t work because ownership was something like root:root instead of redis:redis.

aof files can grow out of control with high activity because by default there’s no memory limit and a no eviction policy, but they rotate and compress well and usually aren’t something to worry about. You can reduce redundancies in the active .aof somewhat with experimentation. In redis.conf, try maxmemory settings that are more conservative and finite than defaults, e.g.,

maxmemory 256mb
maxmemory-policy allkeys-lru

You might also need to experiment with the appendfsync setting for a slow or busy disk, but for now see how it goes set to everysec.

Btw, if remote clients aren’t accessing your redis server, you need to close port 6379 in your firewall and give requirepass in redis.conf a very strong password. You’ll need that password in Nextcloud’s config.php and when using redis-cli.

1 Like

Great, thank you. I added the policies you mentioned as well and security. I’ll look at the permission and appendfsync as well