Cannot login after upgrading MariaDB from 10.2 to 10.5

jptrue · February 21, 2021, 12:33am

Nextcloud version 20.0.7
Operating system and version Ubuntu 18.04
Apache or nginx version Apache 2.4.29
PHP version 7.4

The issue you are facing:
I am completely stuck. Been at it for days… any help or tips would be greatly appreciated.
I upgraded MariaDB to 10.5. shortly after upgrading to Nextcloud 20.0.7, and now I am unable to log in.
I was able to login to Nextclound 20.0.7 before the MariaDB upgrade without issues.
Both the upgrade to Nextcloud 20.0.7 and MariaDB 10.5 seems to go smoothly.
Database is up and running, and everything seems to check out ok.
I was logged in to the web ui, and refreshed the page after the upgrade. That worked first time, then on second refresh I got the enter you updater.secret in plaintext message, and couldn’t do anything else. I followed the suggested procedure to replace that, and now cannot login at all. The logs seem to give no indication of what the problem is.

Other things I’ve tried:

enabling and disabling users
adding a new admintest user
resetting user passwords
running maintenece:repair

Is this the first time you’ve seen this error? Y:

Steps to replicate it:

Go to Login page
Enter correct user credentials
Login page says “Wrong username or password.”

The output of your Nextcloud log in Admin > Logging:

{"reqId":"L2MbbfaN3K9zNtpNihwp","level":2,"time":"2021-02-20T17:40:19+00:00","remoteAddr":"192.168.1.1","user":"--","app":"no app in context","method":"POST","url":"/index.php/login","message":"Login failed: admintest (Remote IP: 192.168.1.1)","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36","version":"20.0.7.1"}

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => 'xxx',
  'passwordsalt' => 'xxx',
  'secret' => 'xxx',
  'trusted_domains' =>
  array (
    0 => 'cloud.xxx.com',
  ),
  'datadirectory' => '/var/www/cloud.xxx.com/data',
  'dbtype' => 'mysql',
  'version' => '20.0.7.1',
  'overwrite.cli.url' => 'https://cloud.xxx.com',
  'dbname' => 'nextclouddb',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'nextcloud',
  'dbpassword' => 'xxx',
  'installed' => true,
  'mail_smtpdebug' => true,
  'maintenance' => false,
  'theme' => '',
  'loglevel' => 0,
  'mail_from_address' => 'xxx',
  'mail_smtpmode' => 'smtp',
  'updater.secret' => 'xxxxxxxxx',
  'mail_sendmailmode' => 'smtp',
  'mail_domain' => 'xxx.com',
);

The output of your Apache/nginx/system log in /var/log/____:

192.168.1.1 - - [20/Feb/2021:09:42:49 -0500] "OPTIONS /remote.php/dav/principals/users/yyy/ HTTP/1.1" 401 557
192.168.1.1 - - [20/Feb/2021:09:42:49 -0500] "PROPFIND /remote.php/dav/principals/users/yyy/ HTTP/1.1" 401 557
192.168.1.1 - - [20/Feb/2021:11:08:13 -0500] "GET /index.php/csrftoken HTTP/1.1" 200 122
192.168.1.1 - - [20/Feb/2021:11:08:13 -0500] "GET /index.php/csrftoken HTTP/1.1" 200 123
192.168.1.1 - - [20/Feb/2021:11:20:13 -0500] "GET /index.php/csrftoken HTTP/1.1" 200 122
192.168.1.1 - - [20/Feb/2021:11:32:13 -0500] "GET /index.php/csrftoken HTTP/1.1" 200 122
192.168.1.1 - - [20/Feb/2021:11:44:13 -0500] "GET /index.php/csrftoken HTTP/1.1" 200 121
192.168.1.1 - - [20/Feb/2021:11:56:13 -0500] "GET /index.php/csrftoken HTTP/1.1" 200 122
192.168.1.1 - - [20/Feb/2021:12:01:12 -0500] "POST /index.php/login HTTP/1.1" 303 -
192.168.1.1 - - [20/Feb/2021:12:01:27 -0500] "GET /index.php/apps/dashboard/ HTTP/1.1" 303 -
192.168.1.1 - - [20/Feb/2021:12:01:27 -0500] "GET /index.php/login?redirect_url=/index.php/apps/dashboard/ HTTP/1.1" 200 4954
192.168.1.1 - - [20/Feb/2021:12:01:30 -0500] "GET /index.php/apps/theming/icon?v=0 HTTP/1.1" 200 24609
192.168.1.1 - - [20/Feb/2021:12:01:36 -0500] "POST /index.php/login HTTP/1.1" 303 -
192.168.1.1 - - [20/Feb/2021:12:01:39 -0500] "GET /index.php/login?user=xxx@xxx.com&redirect_url=/index.php/apps/dashboard/ HTTP/1.1" 200 4998
192.168.1.1 - - [20/Feb/2021:12:13:40 -0500] "GET /index.php/csrftoken HTTP/1.1" 200 122
192.168.1.1 - - [20/Feb/2021:12:25:40 -0500] "GET /index.php/csrftoken HTTP/1.1" 200 121
192.168.1.1 - - [20/Feb/2021:12:37:40 -0500] "GET /index.php/csrftoken HTTP/1.1" 200 122
192.168.1.1 - - [20/Feb/2021:12:38:34 -0500] "GET / HTTP/1.1" 302 -
192.168.1.1 - - [20/Feb/2021:12:38:34 -0500] "GET /index.php/login HTTP/1.1" 200 4918
192.168.1.1 - - [20/Feb/2021:12:40:01 -0500] "GET / HTTP/1.1" 302 -
192.168.1.1 - - [20/Feb/2021:12:40:01 -0500] "GET /index.php/login HTTP/1.1" 200 4921
192.168.1.1 - - [20/Feb/2021:12:40:02 -0500] "GET /core/img/background.png?v=2 HTTP/1.1" 200 9621
192.168.1.1 - - [20/Feb/2021:12:40:02 -0500] "GET /core/img/logo/logo.svg?v=1 HTTP/1.1" 200 321
192.168.1.1 - - [20/Feb/2021:12:40:01 -0500] "GET /index.php/apps/theming/styles?v=0 HTTP/1.1" 200 468
192.168.1.1 - - [20/Feb/2021:12:40:02 -0500] "GET /core/img/background.png?v=0 HTTP/1.1" 304 -
192.168.1.1 - - [20/Feb/2021:12:40:02 -0500] "GET /index.php/apps/theming/manifest?v=0 HTTP/1.1" 200 188
192.168.1.1 - - [20/Feb/2021:12:40:16 -0500] "POST /index.php/login HTTP/1.1" 303 -
192.168.1.1 - - [20/Feb/2021:12:40:23 -0500] "GET /index.php/login?user=admintest HTTP/1.1" 200 4960
192.168.1.1 - - [20/Feb/2021:12:49:40 -0500] "GET /index.php/csrftoken HTTP/1.1" 200 122
192.168.1.1 - - [20/Feb/2021:12:50:35 -0500] "GET /index.php/csrftoken HTTP/1.1" 200 122
192.168.1.1 - - [20/Feb/2021:12:52:23 -0500] "GET /index.php/csrftoken HTTP/1.1" 200 122

tflidd · February 21, 2021, 12:36am

Can you run the occ upgrade procedure?

bb77 · February 21, 2021, 9:53am

How did you “upgrade” your MariaDB?

Did you stop mariaDB and uninstall the old version before you installed the new packages?

sudo systemctl stop mariadb
sudo apt remove "mariadb-*"

Did you upgrade your MariaDB datafolder after the upgrade?

sudo mariadb-upgrade

Is the MariaDB service running?

sudo systemctl status mariadb

…if not, maybe you should start over with this tutorial:
https://mariadb.com/docs/deploy/upgrade-community-server-cs105-ubuntu18/

Hope that helps.

jptrue · February 21, 2021, 6:01pm

Hey bb77,

Thanks for the reply. The link you provided was the exact one I used for the upgrade procedure. Maria DB is up and running and I am able to execute sql command against the database.

I tried running the mariadb-upgrade and it seemed to go off without a hitch.
I even re-ran it with the -force switch during troubleshooting.

xxx@ubuntu2:~$ sudo systemctl status mariadb

● mariadb.service - MariaDB 10.5.8 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: active (running) since Sat 2021-02-20 12:01:03 EST; 24h ago
Docs: man:mariadbd(8)
https://mariadb.com/kb/en/library/systemd/
Process: 2386 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
Process: 2384 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 1650 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/…; /usr/bin/galera_re
Process: 1628 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 1582 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 1722 (mariadbd)
Status: “Taking your SQL requests now…”
Tasks: 10 (limit: 4606)
CGroup: /system.slice/mariadb.service
└─1722 /usr/sbin/mariadbd

Thanks,
Jeff

jptrue · February 21, 2021, 6:04pm

Hey tflidd,

I tried the upgrade, but it tells me I am at the latest.

$ sudo -u www-data /usr/bin/php7.4 occ upgrade
Nextcloud is already latest version

Is there a way to force reapply the upgrade?

Thanks,
Jeff

tflidd · February 22, 2021, 5:02pm

No, I just wanted to make sure the update ran completely and didn’t stop half-way.

Stupid question: do you have other Nextcloud instances installed, perhaps you are linking to the wrong one? If there were database issues, it shouldn’t be able to connect or query the database.

What apps are you using? In case you have more “exotic” once, perhaps disable them… And you don’t have problems with the bruteforce-app?

bb77 · February 22, 2021, 5:42pm

hmm I am out of ideas, to be honest. In addition to what @tflidd suggested, you colud also check if your .htaccess is working correctly…

sudo -u www-data php occ maintenance:update:htaccess

And maybe check if the file ownerships and permissions are correct.

Good luck.

jptrue · February 22, 2021, 9:22pm

Hey @tflidd and @bb77,

Thanks again for the replies and suggestions.

My install is pretty generic with minimal apps. I’ve been doing incremental upgrades for years now (some had issues in the past when switching from sqlite to MariaDB, but I was always able to recover).

Unfortunately, the .htaccess upgrade did not solve the issue.

Bummer, I am out of ideas as well.

Is there a way to get better debug logging on failed logins besides the generic “Login failed: admintest (Remote IP: 192.168.1.1)”?

jptrue · February 23, 2021, 1:58am

Well, it was an adventure, but I’ve got it fixed.

Thought I’d report the solution incase anyone else ran into this issue.

First I did a manual upgrade to version 21, which actually made things worse, but gave me more information. For some reason version 21 was complaining about php 7.2 being install and that I should be using php 7.3 or later. Wierd.

So I went back and started to check and reinstall all of the php prerequisites.

One or more of the following commands fixed the issue.

sudo a2dismod php7.3
sudo a2dismod php7.2
sudo a2enmod php7.4

sudo update-alternatives --set php /usr/bin/php7.4
systemctl restart apache2

sudo apt install php7.4-common -y
sudo apt install php7.4-ctype -y
sudo apt install php7.4-curl -y
sudo apt install php7.4-dom -y
sudo apt install php7.4-gd -y
sudo apt install php7.4-iconv -y
sudo apt install php7.4-json -y
sudo apt install php7.4-libxml -y
sudo apt install php7.4-posix -y
sudo apt install php7.4-simplexml -y
sudo apt install php7.4-smlreader -y
sudo apt install php7.4-xmlreader -y
sudo apt install php7.4-xmlwriter -y
sudo apt install php7.4-zip -y
sudo apt install php7.4-mysql -y
sudo apt install php7.4-xml -y
sudo apt install php7.4-xmlrpc -y
sudo apt install php7.4-imagick -y
sudo apt install php7.4-cli -y
sudo apt install php7.4-dev -y
sudo apt install php7.4-imap -y
sudo apt install php7.4-mbstring -y
sudo apt install php7.4-opcache -y
sudo apt install php7.4-soap -y
sudo apt install php7.4-intl -y
sudo apt install libxml2 -y

systemctl restart apache2

bb77 · February 23, 2021, 7:22am

Thanks for getting back to us and telling us the solution.

Your solution makes sense, but It didn’t come to my mind, that there could be a problem with an older PHP version beeing active, since you are using Ubuntu 20.04 wich allready ships with PHP7.4. Anyways… I’m glad it worked out for you.

BTW. Nextcloud 21 now also supports PHP8 wich should give you even better performance. I will be testing this on my test server in the next few days…