All of a sudden my Nextcloud container is spitting out:
Internal Server Error
The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.
(this happened last week, but a bunch of unrelated stuff broke at work and on other projects so I haven’t been able to look at it).
The nginx error.log contains:
root@nixie:/vol/services/nextcloud# 2021/10/12 12:20:01 [error] 2410#2410: *44953 FastCGI sent in stderr: "PHP message: PHP Fatal error: Uncaught Doctrine\DBAL\Exception: Failed to connect to the database: An exception occurred in the driver: SQLSTATE[HY000] [2006] MySQL server has gone away in /config/www/nextcloud/lib/private/DB/Connection.php:85
> Stack trace:
> #0 /config/www/nextcloud/3rdparty/doctrine/dbal/src/Connection.php(1486): OC\DB\Connection->connect()
> #1 /config/www/nextcloud/3rdparty/doctrine/dbal/src/Connection.php(1014): Doctrine\DBAL\Connection->getWrappedConnection()
> #2 /config/www/nextcloud/lib/private/DB/Connection.php(226): Doctrine\DBAL\Connection->executeQuery()
> #3 /config/www/nextcloud/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php(210): OC\DB\Connection->executeQuery()
> #4 /config/www/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php(287): Doctrine\DBAL\Query\QueryBuilder->execute()
> #5 /config/www/nextcloud/lib/private/AppConfig.php(345): OC\DB\QueryBuilder\QueryBuilder->execute()
> #6 /config/www/nextcloud/lib/private/AppConfig.php(110): OC\AppConfig->loadConfigValues()
> #7" while reading response header from upstream, client: 172.25.29.40, server: _, request: "GET /ocs/v1.php/cloud/capabilities?format=json HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "nextcloud.home.honig.net"
-bash: 2021/10/12: No such file or directory
If I install mariadb-client into my nextcloud container and try to talk to the mariadb container it works:
root@nixie:/vol/services/nextcloud# docker-compose exec nextcloud /bin/bash
root@nextcloud.home.honig.net:/#
root@nextcloud.home.honig.net:/# mysql -h nextcloud_mariadb -u nextcloud --password=nextcloud --database nextcloud
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 11
Server version: 10.5.12-MariaDB-log MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [nextcloud]> status
--------------
mysql Ver 15.1 Distrib 10.5.12-MariaDB, for Linux (x86_64) using readline 5.1
Connection id: 11
Current database: nextcloud
Current user: nextcloud@nextcloud.nextcloud_default
SSL: Not in use
Current pager: stdout
Using outfile: ''
Using delimiter: ;
Server: MariaDB
Server version: 10.5.12-MariaDB-log MariaDB Server
Protocol version: 10
Connection: nextcloud_mariadb via TCP/IP
Server characterset: utf8mb4
Db characterset: latin1
Client characterset: utf8
Conn. characterset: utf8
TCP port: 3306
Uptime: 1 min 37 sec
Threads: 2 Questions: 226 Slow queries: 0 Opens: 114 Open tables: 108 Queries per second avg: 2.329
--------------
MariaDB [nextcloud]>
My docker-compose file looks like (It’s been running for quite a while, maybe a year)?
version: '3'
services:
nextcloud:
image: linuxserver/nextcloud:latest
container_name: nextcloud
hostname: nextcloud.home.honig.net
env_file: .env_uidgid
environment:
TZ: America/New_York
volumes:
- /vol/services/nextcloud/config:/config
- /vol/services/nextcloud/data:/data
- /home/jch:/home/jch
- /vol/library/books:/vol/library/books
- /vol/library/periodicals:/vol/library/periodicals
- /vol/library/software:/vol/library/software
- /vol/media:/vol/media
ports:
- 172.25.29.135:443:443
restart: unless-stopped
mariadb:
image: linuxserver/mariadb:latest
hostname: nextcloud_mariadb
container_name: nextcloud_mariadb
env_file: .env_uidgid
environment:
TZ: America/New_York
MYSQL_DATABASE: nextcloud
MYSQL_USER: nextcloud
MYSQL_ROOT_PASSWORD: REDACTED
MYSQL_PASSWORD: nextcloud
volumes:
- /vol/services/nextcloud/mariadb-config:/config
expose:
- 3306
restart: unless-stopped
The last entry of nextcloud.log (formatted, looks like:
{"reqId":"D2uuWzCsvjyj0NMIVKiE","level":3,"time":"2021-10-12T23:26:07+00:00","remoteAddr":"172.25.29.29","user":"--","app":"core","method":"GET","url":"/favicon.ico","message":{"Exception":"Doctrine\\DBAL\\Exception","Message":"Failed to connect to the database: An exception occurred in the driver: SQLSTATE[HY000] [2002] Connection refused","Code":2002,"Trace":[{"file":"/config/www/nextcloud/3rdparty/doctrine/dbal/src/Connection.php","line":1486,"function":"connect","class":"OC\\DB\\Connection","type":"->"},{"file":"/config/www/nextcloud/3rdparty/doctrine/dbal/src/Connection.php","line":1014,"function":"getWrappedConnection","class":"Doctrine\\DBAL\\Connection","type":"->"},{"file":"/config/www/nextcloud/lib/private/DB/Connection.php","line":226,"function":"executeQuery","class":"Doctrine\\DBAL\\Connection","type":"->"},{"file":"/config/www/nextcloud/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php","line":210,"function":"executeQuery","class":"OC\\DB\\Connection","type":"->"},{"file":"/config/www/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php","line":287,"function":"execute","class":"Doctrine\\DBAL\\Query\\QueryBuilder","type":"->"},{"file":"/config/www/nextcloud/lib/private/Files/Cache/Cache.php","line":162,"function":"execute","class":"OC\\DB\\QueryBuilder\\QueryBuilder","type":"->"},{"file":"/config/www/nextcloud/lib/private/Files/View.php","line":1339,"function":"get","class":"OC\\Files\\Cache\\Cache","type":"->"},{"file":"/config/www/nextcloud/lib/private/Files/View.php","line":1394,"function":"getCacheEntry","class":"OC\\Files\\View","type":"->"},{"file":"/config/www/nextcloud/lib/private/Files/Node/Root.php","line":201,"function":"getFileInfo","class":"OC\\Files\\View","type":"->"},{"file":"/config/www/nextcloud/lib/private/Files/Node/Folder.php","line":149,"function":"get","class":"OC\\Files\\Node\\Root","type":"->"},{"file":"/config/www/nextcloud/lib/private/Files/Node/Folder.php","line":158,"function":"get","class":"OC\\Files\\Node\\Folder","type":"->"},{"file":"/config/www/nextcloud/lib/private/Files/SimpleFS/SimpleFolder.php","line":72,"function":"nodeExists","class":"OC\\Files\\Node\\Folder","type":"->"},{"file":"/config/www/nextcloud/lib/private/Template/JSCombiner.php","line":117,"function":"fileExists","class":"OC\\Files\\SimpleFS\\SimpleFolder","type":"->"},{"file":"/config/www/nextcloud/lib/private/Template/JSCombiner.php","line":103,"function":"isCached","class":"OC\\Template\\JSCombiner","type":"->"},{"file":"/config/www/nextcloud/lib/private/Template/JSResourceLocator.php","line":116,"function":"process","class":"OC\\Template\\JSCombiner","type":"->"},{"file":"/config/www/nextcloud/lib/private/Template/JSResourceLocator.php","line":72,"function":"cacheAndAppendCombineJsonIfExist","class":"OC\\Template\\JSResourceLocator","type":"->"},{"file":"/config/www/nextcloud/lib/private/Template/ResourceLocator.php","line":79,"function":"doFind","class":"OC\\Template\\JSResourceLocator","type":"->"},{"file":"/config/www/nextcloud/lib/private/TemplateLayout.php","line":365,"function":"find","class":"OC\\Template\\ResourceLocator","type":"->"},{"file":"/config/www/nextcloud/lib/private/TemplateLayout.php","line":193,"function":"findJavascriptFiles","class":"OC\\TemplateLayout","type":"::"},{"file":"/config/www/nextcloud/lib/private/legacy/OC_Template.php","line":183,"function":"__construct","class":"OC\\TemplateLayout","type":"->"},{"file":"/config/www/nextcloud/lib/private/Template/Base.php","line":132,"function":"fetchPage","class":"OC_Template","type":"->"},{"file":"/config/www/nextcloud/lib/private/legacy/OC_Template.php","line":332,"function":"printPage","class":"OC\\Template\\Base","type":"->"},{"file":"/config/www/nextcloud/index.php","line":63,"function":"printExceptionErrorPage","class":"OC_Template","type":"::"}],"File":"/config/www/nextcloud/lib/private/DB/Connection.php","Line":85,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36","version":"21.0.4.1"}
My config.php is:
<?php
$CONFIG = array (
'memcache.local' => '\\OC\\Memcache\\APCu',
'datadirectory' => '/data',
'passwordsalt' => 'REDACTED',
'secret' => 'REDACTED',
'trusted_domains' =>
array (
0 => 'localhost',
1 => 'nextcloud.home.honig.net',
2 => 'home.honig.net',
),
'dbtype' => 'mysql',
'version' => '21.0.4.1',
'overwrite.cli.url' => 'http://localhost',
'dbname' => 'nextcloud',
'dbhost' => 'nextcloud_mariadb',
'dbport' => '',
'dbtableprefix' => 'oc_',
'mysql.utf8mb4' => true,
'dbuser' => 'nextcloud',
'dbpassword' => 'nextcloud',
'installed' => true,
'instanceid' => 'REDACTED',
'mail_smtpmode' => 'smtp',
'mail_smtphost' => 'relay.home.honig.net',
'mail_smtpsecure' => 'tls',
'mail_smtpauth' => true,
'mail_smtpauthtype' => 'LOGIN',
'mail_smtpname' => 'nextcloud',
'mail_smtppassword' => 'REDACTED',
'mail_from_address' => 'nextcloud',
'mail_sendmailmode' => 'smtp',
'mail_domain' => 'honig.net',
'maintenance' => false,
'theme' => '',
'loglevel' => 0,
'twofactor_enforced' => 'true',
'twofactor_enforced_groups' =>
array (
),
'twofactor_enforced_excluded_groups' =>
array (
0 => 'admin',
),
'ldapIgnoreNamingRules' => false,
'ldapProviderFactory' => 'OCA\\User_LDAP\\LDAPProviderFactory',
'updater.release.channel' => 'stable',
'default_phone_region' => 'US',
);
I’m using the linuxserver.io containers for nextcloud and mariadb. I had not updated nextcloud recently when this started happening.
TBH I don’t remember doing any container updates before this happened, but it was a week ago. It was 9AM on a Monday morning so I was more interested in coffee than updating software.
Anyone have any ideas to debug this?
Thanks.
Jeff