Accidentally enabled admin_audit app, now nextcloud Internal Server Error

Hi all!

So i clicked on the wrong row and now i cant access nextcloud, all i get is a internal server error even after running “sudo -u www-data php occ app:disable admin_audit”. Also the CPU gets hogged by a bunch of apache2 processes and mysql.

Nextcloud version: 23.0.4.1
Operating system and version: Ubuntu 20.10
Apache or nginx version: Apache/2.4.48
PHP version: PHP 8.0.8

The issue you are facing:
NC not reachable, only see Internal server error message.

Is this the first time you’ve seen this error?
Yes.

Steps to replicate it:
Just enable the mentioned app.

The output of your Nextcloud log in Admin > Logging:
Cant reach webui…

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => '',
  'passwordsalt' => '',
  'secret' => '',
  'trusted_domains' => 
  array (
    0 => '192.168.10.42',
    1 => '<domain_name>',
    2 => '10.125.210.1',
    3 => '192.168.10.1',
  ),
  'allow_local_remote_servers' => true,
  'datadirectory' => '/mnt/nfs/nc-udata',
  'dbtype' => 'mysql',
  'version' => '23.0.4.1',
  'overwrite.cli.url' => 'https://<domain_name>',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud',
  'dbpassword' => '',
  'installed' => true,
  'maintenance' => false,
  'default_phone_region' => 'HU',
  'theme' => '',
  'loglevel' => 2,
  'overwriteprotocol' => 'https',
  'twofactor_enforced' => 'true',
  'twofactor_enforced_groups' => 
  array (
    0 => 'admin',
    1 => 'family',
    2 => 'users',
  ),
  'twofactor_enforced_excluded_groups' => 
  array (
  ),
  'updater.secret' => '',
);

The output of your Apache/nginx/system log in /var/log/____:
https://content.section9.me/doc/ncc/apache2.log

dmesg
https://content.section9.me/doc/ncc/dmesg

Raw dump of logfile inside the data folder:
https://content.section9.me/doc/ncc/nextcloud.log.7z
(Had to compress it, pretty fat 255 MB logfile…)

Some of the errors, which you can search against this forum and the internet.

PHP Fatal error:  Uncaught Doctrine\\DBAL\\Exception: Failed to connect to the database: An exception occurred in the driver: SQLSTATE[08004] [1040] Too many connections in /var/www/nextcloud/lib/private/DB/Connection.php:87\nStack trace:\n#0 /var/www/nextcloud/3rdparty/doctrine/dbal/src/Connection.php(1519): OC\\DB\\Connection->connect()\n#1 /var/www/nextcloud/3rdparty/doctrine/dbal/src/Connection.php(1041): Doctrine\\DBAL\\Connection->getWrappedConnection()\n#2 /var/www/nextcloud/lib/private/DB/Connection.php(237): Doctrine\\DBAL\\Connection->executeQuery()\n#3 /var/www/nextcloud/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php(345): OC\\DB\\Connection->executeQuery()\n#4 /var/www/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php(287): Doctrine\\DBAL\\Query\\QueryBuilder->execute()\n#5 /var/www/nextcloud/lib/private/AppConfig.php(361): OC\\DB\\QueryBuilder\\QueryBuilder->execute()\n#6 /var/www/nextcloud/lib/private/AppConfig.php(126): OC\\AppConfig->loadConfigValues()\n#7 /var/www/nextcloud/lib/private/AppConfig.php(317): OC\\AppConfig->getApps()\n#8 /var/www/nextcloud/lib/private/legacy/OC_App.php(982): OC\\AppConfig->getValues()\n#9 /var/www/nextcloud/lib/private/Server.php(692): OC_App::getAppVersions()\n#10 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(160): OC\\Server->OC\\{closure}()\n#11 /var/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\\AppFramework\\Utility\\SimpleContainer->OC\\AppFramework\\Utility\\{closure}()\n#12 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(127): Pimple\\Container->offsetGet()\n#13 /var/www/nextcloud/lib/private/ServerContainer.php(136): OC\\AppFramework\\Utility\\SimpleContainer->query()\n#14 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(56): OC\\ServerContainer->query()\n#15 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(182): OC\\AppFramework\\Utility\\SimpleContainer->get()\n#16 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(160): OC\\AppFramework\\Utility\\SimpleContainer->OC\\AppFramework\\Utility\\{closure}()\n#17 /var/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(114): OC\\AppFramework\\Utility\\SimpleContainer->OC\\AppFramework\\Utility\\{closure}()\n#18 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(127): Pimple\\Container->offsetGet()\n#19 /var/www/nextcloud/lib/private/ServerContainer.php(136): OC\\AppFramework\\Utility\\SimpleContainer->query()\n#20 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(56): OC\\ServerContainer->query()\n#21 /var/www/nextcloud/lib/private/Server.php(1079): OC\\AppFramework\\Utility\\SimpleContainer->get()\n#22 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(160): OC\\Server->OC\\{closure}()\n#23 /var/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\\AppFramework\\Utility\\SimpleContainer->OC\\AppFramework\\Utility\\{closure}()\n#24 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(127): Pimple\\Container->offsetGet()\n#25 /var/www/nextcloud/lib/private/ServerContainer.php(136): OC\\AppFramework\\Utility\\SimpleContainer->query()\n#26 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(56): OC\\ServerContainer->query()\n#27 /var/www/nextcloud/lib/private/Server.php(2040): OC\\AppFramework\\Utility\\SimpleContainer->get()\n#28 /var/www/nextcloud/lib/private/Files/View.php(118): OC\\Server->getLockingProvider()\n#29 /var/www/nextcloud/lib/private/Server.php(439): OC\\Files\\View->__construct()\n#30 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(160): OC\\Server->OC\\{closure}()\n#31 /var/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\\AppFramework\\Utility\\SimpleContainer->OC\\AppFramework\\Utility\\{closure}()\n#32 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(127): Pimple\\Container->offsetGet()\n#33 /var/www/nextcloud/lib/private/ServerContainer.php(136): OC\\AppFramework\\Utility\\SimpleContainer->query()\n#34 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(56): OC\\ServerContainer->query()\n#35 /var/www/nextcloud/lib/private/Server.php(1403): OC\\AppFramework\\Utility\\SimpleContainer->get()\n#36 /var/www/nextcloud/lib/base.php(594): OC\\Server->boot()\n#37 /var/www/nextcloud/lib/base.php(1089): OC::init()\n#38 /var/www/nextcloud/ocs/v1.php(31): require_once('...')\n#39 /var/www/nextcloud/ocs/v2.php(23): require_once('...')\n#40 {main}\n  thrown in /var/www/nextcloud/lib/private/DB/Connection.php on line 87
"/apps/richdocumentscode/proxy.php?req=/hosting/capabilities","message":"Failed to connect to the database: An exception occurred in the driver: SQLSTATE[HY000] [1040] Too many connections","userAgent":"Nextcloud Server Crawler","version":"23.0.4.1","exception":{"Exception":"Doctrine\\DBAL\\Exception","Message":"Failed to connect to the database: An exception occurred in the driver: SQLSTATE[HY000] [1040] Too many connections"
"Could not boot files_trashbin: Could not resolve trashManager! Class \"trashManager\" does not exist","userAgent":"Nextcloud Server Crawler","version":"23.0.4.1"

Please paste in the exact error you are seeing. Did you do anything else on your system besides enable admin_audit?

Are you using the Nextcloud snap server installation?

I ended up here after my attempt to fix the issue by upping the limit failed. The mysql server used by nc is solely used by it. No other service accesses it.

Internal Server Error

The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.

I think i updated NC that day, but i was moving around things and using it right after the update so i dont think that is what caused the issue. It started when i accidentally enabled the admin_audit app…

No, manual install on ubuntu server with apache2, php8, and mysql.

What does top or similar reveal on system resource usage? It is totally possible the update is part of the problem. I’m unsure.

There are a ton of useful threads on understanding internal server errors.

You should be able to use occ commands to help you debug.

https://content.section9.me/doc/ncc/htop.png
(Its a bhyve VM, physical machine has an 1700x cpu.)

I was able to use NC without a hitch after the update so i didnt think it could be linked to that in light of the fact that enabling admin_audit had an immediate effect.

Yeah, strange. Is it possible php-mysql is no longer installed, or needs to be restarted?

php8.0-mysql was, php-mysql wasnt. Installed php-mysql manually then rebooted the VM, no effect…