Internal Server Error after full-upgrade from Stretch to Buster

Support intro

Sorry to hear you’re facing problems :slightly_frowning_face:

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can :heart:

Nextcloud version (eg, 12.0.2): 15.0.10
Operating system and version (eg, Ubuntu 17.04): Linux Debian 9 Strech -> 10 Buster
Apache or nginx version (eg, Apache 2.4.25): Apache/2.4.38 (Raspbian)
PHP version (eg, 7.1): PHP 7.3.4-2 (cli) (built: Apr 13 2019 19:05:48) ( NTS )

The issue you are facing: After running a full (and successful) apt full-upgrade from Stretch to Buster, NC instance returns a internal server error.

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:

  1. apt full-upgrade
  2. connect to nextcloud instance

The output of your Nextcloud log in Admin > Logging:

unavailable

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

$CONFIG = array (
  'instanceid' => 'xxx',
  'passwordsalt' => 'xxx',
  'secret' => 'xxx',
  'trusted_domains' => 
  array (
    0 => 'xxx',
    1 => 'xxx',
    2 => 'xxx',
  ),
  'datadirectory' => '/media/xxx',
  'dbtype' => 'mysql',
  'version' => '15.0.10.0',
  'overwrite.cli.url' => 'https://xxx',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'nextcloud',
  'dbpassword' => 'xxx',
  'installed' => true,
  'maintenance' => false,
  'theme' => '',
  'loglevel' => 2,
  'enable_previews' => true,
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'updater.release.channel' => 'production',
  'updater.secret' => 'xxx',
);

The output of your Apache/nginx/system log in /var/log/____:

[Wed Jul 10 14:10:43.830605 2019] [:error] [pid 3633] [client xxx:51420] PHP Fatal error:  Uncaught Doctrine\\DBAL\\DBALException: Failed to connect to the database: An exception occured in driver: SQLSTATE[HY000] [2002] No such file or directory in /var/www/html/nextcloud/lib/private/DB/Connection.php:64\nStack trace:\n#0 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(429): OC\\DB\\Connection->connect()\n#1 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(389): Doctrine\\DBAL\\Connection->getDatabasePlatformVersion()\n#2 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(328): Doctrine\\DBAL\\Connection->detectDatabasePlatform()\n#3 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(623): Doctrine\\DBAL\\Connection->getDatabasePlatform()\n#4 /var/www/html/nextcloud/lib/private/DB/Connection.php(151): Doctrine\\DBAL\\Connection->setTransactionIsolation(2)\n#5 /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DriverManager.php(172): OC\\DB\\Connection->__construct(Array, Object(Doctrin in /var/www/html/nextcloud/lib/private/DB/Connection.php on line 64

[Wed Jul 10 15:36:29.029534 2019] [ssl:warn] [pid 591] AH01906: xxx:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Wed Jul 10 15:36:29.031969 2019] [ssl:warn] [pid 591] AH01909: xxx:443:0 server certificate does NOT include an ID which matches the server name
[Wed Jul 10 15:36:31.276409 2019] [ssl:warn] [pid 810] AH01906: xxx:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Wed Jul 10 15:36:31.276560 2019] [ssl:warn] [pid 810] AH01909: xxx:443:0 server certificate does NOT include an ID which matches the server name

Check your /etc/apache2/apache2.conf or included config where server_name is defined. It needs to match the domain where you received the SSL certificate for. Perhaps the dist upgrade has overwritten your Apache configs.

Ahh but that is not the reason for the failure. MariaDB connection fails. Check the status or MariaDB: journalctl -u mariadb

1 Like

Here is the dump from journalctl -u mariadb

Jul 10 15:36:24 xxx systemd[1]: Starting MariaDB 10.3.15 database server...
Jul 10 15:36:28 xxx mysqld[658]: 2019-07-10 15:36:28 0 [Note] /usr/sbin/mysqld (mysqld 10.3.15-MariaDB-1) starting as process 658 ...
Jul 10 15:36:32 xxx /etc/mysql/debian-start[951]: Upgrading MySQL tables if necessary.
Jul 10 15:36:32 xxx systemd[1]: Started MariaDB 10.3.15 database server.
Jul 10 15:36:32 xxx /etc/mysql/debian-start[955]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
Jul 10 15:36:32 xxx /etc/mysql/debian-start[955]: Looking for 'mysql' as: /usr/bin/mysql
Jul 10 15:36:32 xxx /etc/mysql/debian-start[955]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Jul 10 15:36:32 xxx /etc/mysql/debian-start[955]: Version check failed. Got the following error when calling the 'mysql' command line client
Jul 10 15:36:32 xxx /etc/mysql/debian-start[955]: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
Jul 10 15:36:32 xxx /etc/mysql/debian-start[955]: FATAL ERROR: Upgrade failed
Jul 10 15:36:32 xxx /etc/mysql/debian-start[969]: Checking for insecure root accounts.

Re-checking apache2.conf now for the ssl part…

That it is. So there is two things to check:

  • On Stretch already for a while and definitely on Buster, the default authentication is via unix socket, so “root” should be able to login without password.
  • However this can be overwritten or will be different if you migrated from an earlier version, e.g. Jessie that time.

You can check if accessing the mysql console works without password prompt: mysql -u root
If so, then unix socket authentication is active, if not, then the user has a password assigned (which does not match the root unix user, but was set some time when installing/accessing/setting up MariaDB the first time).

If password authentication is active, then cat /etc/mysql/debian.cnf, which is used by the debian-start script, needs to contain it, otherwise the password field needs to be empty. E.g. in my case with unix socket authentication:

2019-07-10 21:43:24 root@micha:/etc/mysql# cat debian.cnf
# Automatically generated for Debian scripts. DO NOT TOUCH!
[client]
host     = localhost
user     = root
password =
socket   = /var/run/mysqld/mysqld.sock
[mysql_upgrade]
host     = localhost
user     = root
password =
socket   = /var/run/mysqld/mysqld.sock
basedir  = /usr

What I can think of now, although this would be a bad update method (package/maintainer-wise), is that debian.cnf got updated with the distro to have an empty password field while the user database actually has a password set for “root”. In this case I hope you remember how you accessed (which password) MariaDB the first time when setting up the nextcloud database and user, so you can re-add it to debian.cnf.

1 Like

Hello,

Have a look here : General info, Debian Buster (10.0) is now available

I beleive you have missing/broken dependencies.

The issue here is a different one. Your error 500 was due to NC13 you came from with does not support PHP7.3. Before updating to Buster, one should update to NC15, the latest one that still supports PHP7.0 but as well already PHP7.3.

In my case, i had debian 9 + NC 16.0.1.
Then i updated to buster
Then i updated from php7.1 to 7.3
Then i did get a error 500
Then i fix my error
Then i updated to NC 16.0.3

In case that can help …

DEBIAN BUSTER, Apache 2.4.38, Php7.3, MariaDB 10.3, Python 3, Redis 5.0.3, Brotli
… and i may add stuff here later…


Server configuration detail

Operating system: Linux 4.19.0-5-amd64 #1 SMP Debian 4.19.37-5 (2019-06-19) x86_64

Webserver: Apache/2.4.38 (fpm-fcgi)

Database: mysql MariaDB 10.3.15 with Barracuda layout

PHP version:

7.3.4-2
Modules loaded: Core, date, libxml, openssl, pcre, zlib, filter, hash, Reflection, SPL, session, sodium, standard, cgi-fcgi, mysqlnd, PDO, xml, apcu, bz2, calendar, ctype, curl, dom, mbstring, fileinfo, ftp, gd, gettext, iconv, igbinary, imagick, intl, json, ldap, exif, mysqli, pdo_mysql, apc, posix, readline, redis, shmop, SimpleXML, smbclient, sockets, sysvmsg, sysvsem, sysvshm, tokenizer, wddx, xmlreader, xmlwriter, xsl, zip, Phar, libsmbclient, Zend OPcache

Nextcloud version: 16.0.1 - 16.0.1.1

Updated from an older Nextcloud/ownCloud or fresh install: Yes, Initialy a NC13 fresh install on a Debian Stretch 9 sometimes like 2 years ago. As it is a personnal test server, i do regular updates on BOTH the debian side, and the NC side. Almost every updates done through the web-interface without any troubles excepted for major debian stretch to buster update.

Where did you install Nextcloud from: sources.

List of activated apps
Enabled:
 - accessibility: 1.2.0
 - activity: 2.9.1
 - admin_audit: 1.6.0
 - announcementcenter: 3.5.1
 - apporder: 0.7.1
 - audioplayer: 2.7.2
 - browser_warning: 0.1.0
 - bruteforcesettings: 1.3.0
 - calendar: 1.7.0
 - camerarawpreviews: 0.7.0
 - checksum: 0.4.3
 - cloud_federation_api: 0.2.0
 - comments: 1.6.0
 - contacts: 3.1.3
 - data_request: 1.3.0
 - dav: 1.9.2
 - external: 3.3.0
 - federatedfilesharing: 1.6.0
 - federation: 1.6.0
 - files: 1.11.0
 - files_downloadactivity: 1.5.0
 - files_pdfviewer: 1.5.0
 - files_rightclick: 0.13.0
 - files_sharing: 1.8.0
 - files_texteditor: 2.8.0
 - files_trackdownloads: 1.5.0
 - files_trashbin: 1.6.0
 - files_versions: 1.9.0
 - files_videoplayer: 1.5.0
 - firstrunwizard: 2.5.0
 - flowupload: 0.1.2
 - gallery: 18.3.0
 - groupfolders: 4.0.3
 - impersonate: 1.3.0
 - issuetemplate: 0.5.0
 - logreader: 2.1.0
 - lookup_server_connector: 1.4.0
 - metadata: 0.9.0
 - notifications: 2.4.1
 - oauth2: 1.4.2
 - occweb: 0.0.4
 - password_policy: 1.6.0
 - previewgenerator: 2.1.0
 - provisioning_api: 1.6.0
 - quota_warning: 1.5.0
 - registration: 0.4.6
 - serverinfo: 1.6.0
 - sharebymail: 1.6.0
 - sharerenamer: 2.6.0
 - socialsharing_facebook: 1.0.4
 - support: 1.0.0
 - survey_client: 1.4.0
 - systemtags: 1.6.0
 - twofactor_backupcodes: 1.5.0
 - twofactor_totp: 2.1.2
 - twofactor_u2f: 3.0.0
 - twofactor_yubikey: 0.4.1
 - unsplash: 1.1.3
 - updatenotification: 1.6.0
 - user_usage_report: 1.1.2
 - viewer: 1.0.0
 - workflowengine: 1.6.0
Disabled:
 - circles
 - dashboard
 - encryption
 - files_antivirus
 - files_external
 - music
 - news
 - nextcloud_announcements
 - polls
 - privacy
 - recommendations
 - sociallogin
 - theming
 - user_ldap
 - user_saml

Configuration (config/config.php)
{
    "instanceid": "***REMOVED SENSITIVE VALUE***",
    "passwordsalt": "***REMOVED SENSITIVE VALUE***",
    "secret": "***REMOVED SENSITIVE VALUE***",
    "trusted_domains": [
        "nextcloud.rkn.ovh"
    ],
    "memcache.distributed": "\\OC\\Memcache\\Redis",
    "memcache.locking": "\\OC\\Memcache\\Redis",
    "memcache.local": "\\OC\\Memcache\\APCu",
    "redis": {
        "host": "***REMOVED SENSITIVE VALUE***",
        "port": 6379
    },
    "datadirectory": "***REMOVED SENSITIVE VALUE***",
    "overwrite.cli.url": "https:\/\/nextcloud.rkn.ovh",
    "dbtype": "mysql",
    "version": "16.0.1.1",
    "dbname": "***REMOVED SENSITIVE VALUE***",
    "dbhost": "***REMOVED SENSITIVE VALUE***",
    "dbport": "",
    "dbtableprefix": "oc_",
    "dbuser": "***REMOVED SENSITIVE VALUE***",
    "dbpassword": "***REMOVED SENSITIVE VALUE***",
    "installed": true,
    "mail_smtpmode": "smtp",
    "mail_smtpauthtype": "LOGIN",
    "mail_smtpsecure": "ssl",
    "mail_from_address": "***REMOVED SENSITIVE VALUE***",
    "mail_domain": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpauth": 1,
    "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpport": "465",
    "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
    "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
    "cache_chunk_gc_ttl": 86400,
    "filelocking.ttl": 3600,
    "auth.bruteforce.protection.enabled": false,
    "maintenance": false,
    "loglevel": 1,
    "ldapIgnoreNamingRules": false,
    "ldapProviderFactory": "\\OCA\\User_LDAP\\LDAPProviderFactory",
    "theme": "",
    "updater.release.channel": "stable",
    "twofactor_enforced": "true",
    "twofactor_enforced_groups": [],
    "twofactor_enforced_excluded_groups": [
        "Alpn",
        "admin",
        "pro",
        "scolaire",
        "users"
    ],
    "mysql.utf8mb4": true,
    "app_install_overwrite": [
        "external",
        "occweb",
        "files_downloadactivity",
        "admin_notifications",
        "twofactor_yubikey"
    ],
    "updater.secret": "***REMOVED SENSITIVE VALUE***"
}

Are you using external storage, if yes which one: none, but a raid6 mdadm/lvm2 mounted as /mnt/raid6/nextcloud

Are you using encryption: no, but HTTPS and UF2 Auth for clients.

Are you using an external user-backend, if yes which one: no


**Others configuration

No alp1/Alp2, i use openssl 1.1.1.c with Letsencrypt for SSL certs.

Sorry, no caddyfile. I manually install all. This is a dedicated homemade server…

Redis configuration:

bind 127.0.0.1 ::1
protected-mode yes
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 300
daemonize yes
supervised no
pidfile /var/run/redis/redis-server.pid
loglevel notice
logfile /var/log/redis/redis-server.log
syslog-enabled yes
syslog-ident redis
databases 16
always-show-logo yes
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /var/lib/redis
replica-serve-stale-data yes
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
replica-priority 100
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
replica-lazy-flush no
maxclients 10000
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
hll-sparse-max-bytes 3000
stream-node-max-bytes 4096
stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
dynamic-hz yes
aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes

Redis log:

1114:C 10 Jul 2019 02:23:58.869 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1114:C 10 Jul 2019 02:23:58.880 # Redis version=5.0.3, bits=64, commit=00000000, modified=0, pid=1114, just started
1114:C 10 Jul 2019 02:23:58.880 # Configuration loaded
                _._
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 5.0.3 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 1236
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           http://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'
              `-.__.-'

1236:M 10 Jul 2019 02:23:58.899 # Server initialized
1236:M 10 Jul 2019 02:23:58.999 * DB loaded from disk: 0.099 seconds
1236:M 10 Jul 2019 02:23:58.999 * Ready to accept connections
1236:M 10 Jul 2019 02:31:12.998 * 10 changes in 300 seconds. Saving...

Hope this could help.

Here is also a list of the package installed on my buster:

Additionnal Additional .ini files parsed:

/etc/php/7.3/fpm/conf.d/10-mysqlnd.ini, /etc/php/7.3/fpm/conf.d/10-opcache.ini, /etc/php/7.3/fpm/conf.d/10-pdo.ini, /etc/php/7.3/fpm/conf.d/15-xml.ini, /etc/php/7.3/fpm/conf.d/20-apcu.ini, /etc/php/7.3/fpm/conf.d/20-bz2.ini, /etc/php/7.3/fpm/conf.d/20-calendar.ini, /etc/php/7.3/fpm/conf.d/20-ctype.ini, /etc/php/7.3/fpm/conf.d/20-curl.ini, /etc/php/7.3/fpm/conf.d/20-dom.ini, /etc/php/7.3/fpm/conf.d/20-exif.ini, /etc/php/7.3/fpm/conf.d/20-fileinfo.ini, /etc/php/7.3/fpm/conf.d/20-ftp.ini, /etc/php/7.3/fpm/conf.d/20-gd.ini, /etc/php/7.3/fpm/conf.d/20-gettext.ini, /etc/php/7.3/fpm/conf.d/20-iconv.ini, /etc/php/7.3/fpm/conf.d/20-igbinary.ini, /etc/php/7.3/fpm/conf.d/20-imagick.ini, /etc/php/7.3/fpm/conf.d/20-intl.ini, /etc/php/7.3/fpm/conf.d/20-json.ini, /etc/php/7.3/fpm/conf.d/20-ldap.ini, /etc/php/7.3/fpm/conf.d/20-mbstring.ini, /etc/php/7.3/fpm/conf.d/20-mysqli.ini, /etc/php/7.3/fpm/conf.d/20-pdo_mysql.ini, /etc/php/7.3/fpm/conf.d/20-phar.ini, /etc/php/7.3/fpm/conf.d/20-posix.ini, /etc/php/7.3/fpm/conf.d/20-readline.ini, /etc/php/7.3/fpm/conf.d/20-redis.ini, /etc/php/7.3/fpm/conf.d/20-shmop.ini, /etc/php/7.3/fpm/conf.d/20-simplexml.ini, /etc/php/7.3/fpm/conf.d/20-smbclient.ini, /etc/php/7.3/fpm/conf.d/20-sockets.ini, /etc/php/7.3/fpm/conf.d/20-sysvmsg.ini, /etc/php/7.3/fpm/conf.d/20-sysvsem.ini, /etc/php/7.3/fpm/conf.d/20-sysvshm.ini, /etc/php/7.3/fpm/conf.d/20-tokenizer.ini, /etc/php/7.3/fpm/conf.d/20-wddx.ini, /etc/php/7.3/fpm/conf.d/20-xmlreader.ini, /etc/php/7.3/fpm/conf.d/20-xmlwriter.ini, /etc/php/7.3/fpm/conf.d/20-xsl.ini, /etc/php/7.3/fpm/conf.d/20-zip.ini, /etc/php/7.3/fpm/conf.d/25-apcu_bc.ini

Internal Server Error

After the upgrade to buster, I needed to install the php packages:

php7.3-bz2 php7.3-cli php7.3-common php7.3-curl php7.3-dev php7.3-fpm php7.3-gd php7.3-intl php7.3-json php7.3-ldap php7.3-mbstring php7.3-mysql php7.3-xml php7.3-zip php7.3

Activate php 7.3

sudo a2dismod php7.0
sudo a2enmod php7.3

manual Upgrade to 16.3
https://docs.nextcloud.com/server/16/admin_manual/maintenance/upgrade.html

2 Likes

this is what did it for me after ending up with internal error after upgrade from Stretch to Buster on Raspberry Pi. I also upgraded libapache2-mod-php7.3, not sure if that one mattered or not though. Thanks so much Andre39!