Internal Server Error ... Found An Error!

[yupthatguy]

I am on Ubuntu 20.04 LAMP with Virtualmin 7.1-1 control panel, running php7.4-fpm, and NC 24.0.4.1, with a redis unix socket.

I just did the latest php updates and now my nextcloud instance is completely offline, with an internal server error.

My apache2 error contains no errors.
My nextcloud.log contains no errors.
My custom NC error log contains no errors.
UPDATE:
My php7.4-fpm.log also contains no errors…

My test server contain all the same php modules:

apcu
bcmath
bz2
calendar
Core
ctype
curl
date
dom
exif
FFI
fileinfo
filter
ftp
gd
gettext
gmp
hash
iconv
igbinary
imagick
json
ldap
libxml
mbstring
memcache
memcached
msgpack
mysqli
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
Phar
posix
readline
redis
Reflection
session
shmop
SimpleXML
sockets
sodium
SPL
ssh2
standard
sysvmsg
sysvsem
sysvshm
tokenizer
xml
xmlreader
xmlwriter
xsl
Zend OPcache
zip
zlib

[Zend Modules]
Zend OPcache

config.php

<?php
$CONFIG = array (
  'instanceid' => 'xxxxxxx',
  'passwordsalt' => 'password',
  'secret' => 'secret',
  'trusted_domains' => 
  array (
    0 => 'www.example.com',
    1 => 'example.com',
  ),
  'datadirectory' => '/home/username/private/data',
  'dbtype' => 'mysql',
  'version' => '24.0.4.1',
  'overwrite.cli.url' => 'https://www.example.com',
  'dbname' => 'username',
  'dbhost' => 'localhost:3306',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'username',
  'dbpassword' => 'password',
  'installed' => true,
  'memcache.local' => '\\OC\\Memcache\\Redis',
  'filelocking.enabled' => true,
  'memcache.locking' => '\\OC\\Memcache\\APCu',
  'redis' => 
  array (
    'host' => '/var/run/redis/redis-server.sock',
    'port' => 0,
    'timeout' => 0.0,
    'password' => 'redis',
  ),
  'default_phone_region' => 'CN',
  'log_type' => 'file',
  'logfile' => '/home/username/ncerrors/error.log',
  'loglevel' => 2,
  'logdateformat' => 'F d, Y H:i:s',
  'trashbin_retention_obligation' => '3, 7',
  'mail_smtpmode' => 'smtp',
  'mail_sendmailmode' => 'smtp',
  'mail_from_address' => 'notifications',
  'mail_domain' => 'example.com',
  'mail_smtpauthtype' => 'PLAIN',
  'mail_smtpauth' => 1,
  'mail_smtphost' => 'mail.example.com',
  'mail_smtpport' => '587',
  'mail_smtpname' => 'notifications@example.com',
  'mail_smtppassword' => 'password',
  'maintenance' => false,
  'auth.bruteforce.protection.enabled' => 'false',
  'theme' => '',
  'app_install_overwrite' => 
  array (
    0 => 'files_antivirus',
    1 => 'qownnotesapi',
    2 => 'appointments',
  ),
  'data-fingerprint' => 'fingerprint',
  'mail_smtpsecure' => 'tls',
  'trusted_proxies' => 
  array (
    0 => '888.888..88.88',
    1 => '127.0.0.1',
    2 => '::1',
  ),
);

I have rebooted the entire server and all services… just an internal server error.

How do I go about troubleshooting this?

I would really appreciate any help with this… I have tried every solution in this forum with no luck… It feels very unsafe to put my data and the data of others into something that can’t be repaired because there are no logs. Worked perfectly for 4+ months… before the php update, now completely dead and zero reasons why…

I finally found an error

Definitely a Nextcloud bug of some sorts:

85% of the time when I run systemctl status php7.4-fpm I see:

● php7.4-fpm.service - The PHP 7.4 FastCGI Process Manager
     Loaded: loaded (/lib/systemd/system/php7.4-fpm.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2022-08-26 15:39:18 CST; 2s ago
       Docs: man:php-fpm7.4(8)
    Process: 50963 ExecStartPost=/usr/lib/php/php-fpm-socket-helper install /run/php/php-fpm.sock /etc/php/7.4/fpm/pool.d/www.conf 74 (code=exited, status=0/SUCCESS)
   Main PID: 50940 (php-fpm7.4)
     Status: "Ready to handle connections"
      Tasks: 3 (limit: 4380)
     Memory: 18.7M
     CGroup: /system.slice/php7.4-fpm.service
             ├─50940 php-fpm: master process (/etc/php/7.4/fpm/php-fpm.conf)
             ├─50961 php-fpm: pool www
             └─50962 php-fpm: pool www

Aug 26 15:39:17 admin.example.com systemd[1]: Starting The PHP 7.4 FastCGI Process Manager...
Aug 26 15:39:18 admin.example.com systemd[1]: Started The PHP 7.4 FastCGI Process Manager.

I just ran the same command again and I got:

● php7.4-fpm.service - The PHP 7.4 FastCGI Process Manager
     Loaded: loaded (/lib/systemd/system/php7.4-fpm.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2022-08-26 14:57:00 CST; 40min ago
       Docs: man:php-fpm7.4(8)
    Process: 38966 ExecStartPost=/usr/lib/php/php-fpm-socket-helper install /run/php/php-fpm.sock /etc/php/7.4/fpm/pool.d/www.conf 74 (code=exited, status=0/SUCCESS)
   Main PID: 38943 (php-fpm7.4)
     Status: "Processes active: 0, idle: 2, Requests: 134, slow: 0, Traffic: 0req/sec"
      Tasks: 3 (limit: 4380)
     Memory: 32.2M
     CGroup: /system.slice/php7.4-fpm.service
             ├─38943 php-fpm: master process (/etc/php/7.4/fpm/php-fpm.conf)
             ├─38964 php-fpm: pool www
             └─38965 php-fpm: pool www

Aug 26 15:27:44 admin.example.com php[48909]:   thrown in /home/username/public_html/lib/private/Memcache/Factory.php on line 90
Aug 26 15:29:14 admin.example.com php[49083]: PHP Fatal error:  Uncaught OCP\HintException: [0]: Memcache \OC\Memcache\Redis not available for local cache (Is the >
Aug 26 15:29:14 admin.example.com php[49083]: 
Aug 26 15:29:14 admin.example.com php[49083]:   thrown in /home/username/public_html/lib/private/Memcache/Factory.php on line 90
Aug 26 15:29:35 admin.example.com php[49084]: PHP Fatal error:  Uncaught OCP\HintException: [0]: Memcache \OC\Memcache\Redis not available for local cache (Is the >
Aug 26 15:29:35 admin.example.com php[49084]: 
Aug 26 15:29:35 admin.example.com php[49084]:   thrown in /home/username/public_html/lib/private/Memcache/Factory.php on line 90
Aug 26 15:29:59 admin.example.com php[49089]: PHP Fatal error:  Uncaught OCP\HintException: [0]: Memcache \OC\Memcache\Redis not available for local cache (Is the >
Aug 26 15:29:59 admin.example.com php[49089]: 
Aug 26 15:29:59 admin.example.com php[49089]:   thrown in /home/username/public_html/lib/private/Memcache/Factory.php on line 90

[yupthatguy]

I have tried all of the solutions found here…

Internal server error without any logging

Hey mate, hope you are doing well… Let’s try to fix your issue here… But first, one question: Do you have any cache service running? Example: Redis. You can try this: 1) sudo -u www-data php /var/www/nextcloud/occ maintenance:mode --on 2) chown -R www-data:www-data /var/www/nextcloud/ 3) chmod -R 770 /var/www/nextcloud/ 4) sudo -u www-data php /var/www/nextcloud/occ maintenance:repair 5) sudo -u www-data php /var/www/nextcloud/occ db:add-missing-columns 6) sudo -u www-data php /var/www/…

Nextcloud Internal Server Error but no logs

Hey all, I’ve upgraded Nextcloud to 21 and it runs - everything ok. I’m using Apache2 with PHP7.3 FPM on Ubuntu 18.04 LTS. With this upgrade I removed the old php versions which are installed on the system with: apt purge php5.6* apt purge php7.0* apt purge php7.1* apt purge php7.2* apt autoclean apt autoremove Restart the server after these commands and Nextcloud throws an error 500 on opening a page in the browser (It seems by nextcloud itself - not by apache). What I did: I’ve a i…

[SOLVED] Internal server error - no log entries

Nextcloud version (eg, 20.0.5): 21.0.0-9 Operating system and version (eg, Ubuntu 20.04): arch Apache or nginx version (eg, Apache 2.4.25): Apache 2.4.46-3 PHP version (eg, 7.4): 8.0.3-1 The issue you are facing: Internal Server Error message when accessing via website after upgrading to v21 from v20. No details given, but there is a request ID. There is no corresponding entry in the nextcloud log—in fact there are no messages generated from accessing the website despite lowering the log_l…

no luck

[yupthatguy]

SOLVED

After finally finding an error I was able to determine that the php update mangled my php7.4-redis configs.

First off, I really hope that NC exception handling improves… its a scary moment when data becomes inaccessible to a completely invisible error… this is not a small issue. You guys are great, but this is makes people think twice about nextcloud usage.

Secondly, for anyone else who nextcloud throws internal errors without any logs… my advice is run:

systemctl (restart/status) apache2 mysql redis php*-fpm at least 20 to 30 times until you catch an error.

After finding my error above… simply ran sudo apt-get install --reinstall php7.4-redis and grep -R "redis" /etc/php against my test server and prod server and could easily see that php-redis was misconfigured after my php update/reinstall that these files were missing on my prod server:

7.4/cgi/conf.d/20-redis.ini:extension=redis.so
7.4/fpm/conf.d/20-redis.ini:extension=redis.so
7.4/cli/conf.d/20-redis.ini:extension=redis.so

I corrected the redis config and presto everything came back.

I’ve blown a whole web development work… but oh well…

error logging is important.