Can't access "Apps" and "Settings->System"

I have what I think are dire errors on my production nextcloud.
I’m looking for any guidance.
It’s a real problem because I can’t re-activate important apps after an upgrade, because I can’t access “Apps” anymore.
It throws the standard error screen with the following info :

Internal Server Error

The server was unable to complete your request.

If this happens again, please send the technical details below to the server administrator.

More details can be found in the server log.
Technical details

    Remote Address: 192.168.2.104
    Request ID: 3Dn5GoTmeBR3RBul3A08

You’ll find below the complete error log using the template provided by the forums.
Any ideas, even as to how to start over without loosing any user info (it’s a production server) are welcome, although I’d prefer finding the fix.

Nextcloud version : 22.2.8
Operating system and version : Debian 11
Apache or nginx version : Apache 2.4.38
PHP version : PHP 7.3.29

Is this the first time you’ve seen this error? : No

(Steps to replicate it: I wouldn't really know, see below)

  1. This all happened after a failed upgrade from 20 to 21. I continued using it with this bug, because it doesn’t prevent normal usage.

The output of your Nextcloud log in Admin > Logging:
This error is the one that appears when accessing "Apps" from the menu

Error	index	OCP\Files\NotPermittedException: 

    /var/www/nextcloud/lib/private/Files/SimpleFS/SimpleFile.php - line 90:

    OC\Files\Node\File->getContent()

    /var/www/nextcloud/lib/private/App/AppStore/Fetcher/Fetcher.php - line 154:

    OC\Files\SimpleFS\SimpleFile->getContent()

    /var/www/nextcloud/lib/private/Installer.php - line 432:

    OC\App\AppStore\Fetcher\Fetcher->get(false)

    /var/www/nextcloud/apps/serverinfo/lib/SystemStatistics.php - line 181:

    OC\Installer->isUpdateAvailable("accessibility")

    /var/www/nextcloud/apps/serverinfo/lib/SystemStatistics.php - line 90:

    OCA\ServerInfo\SystemStatistics->getAppsInfo()

    /var/www/nextcloud/apps/serverinfo/lib/Settings/AdminSettings.php - line 121:

    OCA\ServerInfo\SystemStatistics->getSystemStatistics()

    /var/www/nextcloud/apps/settings/lib/Controller/CommonSettingsTrait.php - line 137:

    OCA\ServerInfo\Settings\AdminSettings->getForm()

    /var/www/nextcloud/apps/settings/lib/Controller/AdminSettingsController.php - line 82:

    OCA\Settings\Controller\AdminSettingsController->formatSettings([ [ OCA\Serv ... ]])

    /var/www/nextcloud/apps/settings/lib/Controller/CommonSettingsTrait.php - line 148:

    OCA\Settings\Controller\AdminSettingsController->getSettings("serverinfo")

    /var/www/nextcloud/apps/settings/lib/Controller/AdminSettingsController.php - line 67:

    OCA\Settings\Controller\AdminSettingsController->getIndexResponse("admin", "serverinfo")

    /var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php - line 217:

    OCA\Settings\Controller\AdminSettingsController->index("serverinfo")

    /var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php - line 126:

    OC\AppFramework\Http\Dispatcher->executeController(OCA\Settings ... {}, "index")

    /var/www/nextcloud/lib/private/AppFramework/App.php - line 156:

    OC\AppFramework\Http\Dispatcher->dispatch(OCA\Settings ... {}, "index")

    /var/www/nextcloud/lib/private/Route/Router.php - line 302:

    OC\AppFramework\App::main("OCA\\Settin ... r", "index", OC\AppFramew ... {}, { section: " ... "})

    /var/www/nextcloud/lib/base.php - line 1008:

    OC\Route\Router->match("/settings/admin/serverinfo")

    /var/www/nextcloud/index.php - line 36:

    OC::handleRequest()

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => 'redacted',
  'passwordsalt' => 'redacted',
  'secret' => 'redacted',
  'trusted_domains' =>
  array (
    0 => 'redacted',
    1 => 'redacted',
    2 => 'redacted',
  ),
  'datadirectory' => '/mnt/livedata/nextcloud/data',
  'dbtype' => 'mysql',
  'version' => '22.2.8.1',
  'overwrite.cli.url' => 'redacted',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'ncu',
  'dbpassword' => 'redacted',
  'installed' => true,
//  'memcache.local' => '\\OC\\Memcache\\APCu',
  'filelocking.enabled' => true,
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
    'host' => 'localhost',
    'port' => 6379,
    'timeout' => 0,
    'password' => 'redacted',
  ),
  'knowledgebaseenable' => false,
  'lost_password_link' => 'disabled',
  'trashbin_retention_obligation' => 14,
  0 => 60,
  'default_language' => 'fr',
  'default_locale' => 'fr_CH',
  'maintenance' => false,
  'log_type' => 'file',
  'loglevel' => 2,
  'remember_login_cookie_lifetime' => 172800,
  'mail_smtpmode' => 'smtp',
  'mail_smtpsecure' => 'ssl',
  'mail_sendmailmode' => 'smtp',
  'mail_from_address' => 'mailrelay',
  'mail_domain' => 'redacted',
  'mail_smtpauthtype' => 'PLAIN',
  'mail_smtpauth' => 1,
  'mail_smtpname' => 'mailrelay',
  'mail_smtppassword' => 'redacted',
  'mail_smtphost' => 'redacted',
  'mail_smtpport' => '465',
  'default_phone_region' => 'CH',
);

The output of your Apache/nginx/system log in /var/log/____:

root@nc:/var/www/nextcloud# cat /var/log/apache2/error.log
[Sat May 21 00:00:02.452029 2022] [mpm_event:notice] [pid 897:tid 139651571831936] AH00489: Apache/2.4.38 (Debian) OpenSSL/1.1.1d configured -- resuming normal operations
[Sat May 21 00:00:02.452058 2022] [core:notice] [pid 897:tid 139651571831936] AH00094: Command line: '/usr/sbin/apache2'
[Sat May 21 00:46:24.130062 2022] [proxy_fcgi:error] [pid 31661:tid 139650475726592] [client 103.101.162.249:55096] AH01071: Got error 'Primary script unknown', referer: http://cloud.sight-sound.ch/wp-login.php
[Sat May 21 16:33:47.815671 2022] [mpm_event:notice] [pid 604:tid 140174739883136] AH00489: Apache/2.4.38 (Debian) OpenSSL/1.1.1d configured -- resuming normal operations
[Sat May 21 16:33:47.816494 2022] [core:notice] [pid 604:tid 140174739883136] AH00094: Command line: '/usr/sbin/apache2'
[Sat May 21 16:36:28.738270 2022] [mpm_event:notice] [pid 604:tid 140174739883136] AH00491: caught SIGTERM, shutting down
[Sat May 21 16:39:55.772855 2022] [mpm_event:notice] [pid 26768:tid 140040242033792] AH00489: Apache/2.4.38 (Debian) OpenSSL/1.1.1d configured -- resuming normal operations
[Sat May 21 16:39:55.773079 2022] [core:notice] [pid 26768:tid 140040242033792] AH00094: Command line: '/usr/sbin/apache2'
[Sat May 21 16:46:41.802259 2022] [mpm_event:notice] [pid 26768:tid 140040242033792] AH00491: caught SIGTERM, shutting down
[Sat May 21 16:46:41.896442 2022] [mpm_event:notice] [pid 26933:tid 140694429312128] AH00489: Apache/2.4.38 (Debian) OpenSSL/1.1.1d configured -- resuming normal operations
[Sat May 21 16:46:41.896573 2022] [core:notice] [pid 26933:tid 140694429312128] AH00094: Command line: '/usr/sbin/apache2'

Please check the permissions in your Nextcloud program folder and its sub folders as well as owner and group.

I just upgraded manually and reset all permissions to what is advised in this guide:

chown -R www-data:www-data nextcloud
find nextcloud/ -type d -exec chmod 750 {} \;
find nextcloud/ -type f -exec chmod 640 {} \;
root@ncd:/var/www/nextcloud# ls -la
total 164
drwxr-x--- 13 www-data www-data  4096 mai 21 16:48 .
drwxr-xr-x  5 root     root      4096 mai 21 16:37 ..
drwxr-x--- 44 www-data www-data  4096 mai 19 12:41 3rdparty
drwxr-x--- 55 www-data www-data  4096 mai 21 16:38 apps
-rw-r-----  1 www-data www-data 19327 mai 19 12:38 AUTHORS
drwxr-x---  2 www-data www-data  4096 mai 21 16:46 config
-rw-r-----  1 www-data www-data  3924 mai 19 12:38 console.php
-rw-r-----  1 www-data www-data 34520 mai 19 12:38 COPYING
drwxr-x--- 22 www-data www-data  4096 mai 19 12:42 core
-rw-r-----  1 www-data www-data  6197 mai 19 12:38 cron.php
-rw-r-----  1 www-data www-data  3396 mai 21 16:43 .htaccess
-rw-r-----  1 www-data www-data   156 mai 19 12:38 index.html
-rw-r-----  1 www-data www-data  3455 mai 19 12:38 index.php
drwxr-x---  6 www-data www-data  4096 mai 19 12:38 lib
-rw-r-----  1 www-data www-data   283 mai 19 12:38 occ
drwxr-x---  2 www-data www-data  4096 mai 19 12:38 ocm-provider
drwxr-x---  2 www-data www-data  4096 mai 19 12:38 ocs
drwxr-x---  2 www-data www-data  4096 mai 19 12:38 ocs-provider
-rw-r-----  1 www-data www-data  3139 mai 19 12:38 public.php
-rw-r-----  1 www-data www-data  5340 mai 19 12:38 remote.php
drwxr-x---  4 www-data www-data  4096 mai 19 12:38 resources
-rw-r-----  1 www-data www-data    26 mai 19 12:38 robots.txt
-rw-r-----  1 www-data www-data  2452 mai 19 12:38 status.php
drwxr-x---  3 www-data www-data  4096 mai 19 12:38 themes
drwxr-x---  2 www-data www-data  4096 mai 19 12:38 updater
-rw-r-----  1 www-data www-data   101 mai 19 12:38 .user.ini
-rw-r-----  1 www-data www-data   422 mai 19 12:41 version.php

With no effect, as far as I can tell.
Also note that (unsurprisingly), apps can’t be upgraded with occ either, the cli throwing the same exception, without details (wether I update one or all apps):

root@ncd:/var/www/nextcloud# sudo -u www-data php occ app:update --all

In File.php line 58:

  [OCP\Files\NotPermittedException]

app:install doesn’t work either, without any details about the error:

root@ncd:/var/www/nextcloud# sudo -u www-data php occ app:install impersonate
Error:

But all other app: commands work though !
I also tried maintenance:repair, which ran without errors, but found nothing as far as I can tell.

Is your /www/var/nextcloud owned by root? I think it must be owned by www-data:www-data

/var/www/nextcloud is owned by www-data, it’s .. (->/var/www) that is owned by root

Up!
No way to trace back these error logs to something?

Just an update :

I can, in facts, install apps with occ (although update --all does not work).
Also, the updater doesn’t work, but manual upgrade works.

maybe try this command?
occ files:scan-app-data

I would take a backup of database and data folder (everything underneath data), config.php file, the .user.ini file or .htaccess file or both.

Then I would replace all files EXCEPT the data folder with all files in the 30.0.1 release zip folder.

add the backedup config.php as: old.config.php under the config folder (this way you use a “custom” config file without overriding the core one).
Replace the .user.ini and .htaccess files.

Then change owner of all folders underneath that and change permissions to 750 for folders and 640 for files.

Then test if occ works as expected by running this one: files:scan --all