All logins broken after auto-upgrade to NC 23

Support intro

Sorry to hear you’re facing problems :slightly_frowning_face:

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can :heart:

Nextcloud version (eg, 20.0.5): 23.0.2 installed via Snap
Operating system and version (eg, Ubuntu 20.04): Ubuntu 16.04
Apache or nginx version (eg, Apache 2.4.25): Unknown (included in snap package)
PHP version (eg, 7.4): Unknown (included in snap package)

The issue you are facing:

  • After snap decided to auto-upgrade Nextcloud from version 22 to 23, no users are able to login, since we are depending on the user_saml app which was auto-disabled during the upgrade process.
  • I have learned that the user_saml app is not compatible with Nextcloud 23 yet.
  • I also understand that downgrading to a previous version of Nextcloud is not supported.
  • On the Github issue for adding NC 23 support to user_saml, there is a comment about running occ app:enable -f user_saml to force the app to be enabled anyway. However, there is also a comment upcoming config format changes that need to be resolved before the app is ready for NC 23, which makes me hesitant to run this command.

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:

  1. Install Nextcloud 22 or earlier
  2. Enable user_saml app and create a new user with only SAML login
  3. Delete the initial admin user
  4. Upgrade to Nextcloud 23
  5. There are now no users in your system, nobody can login. nextcloud.occ user:list shows that the SAML-users do not exist anymore (but their files still seem to be in the storage folder.)

The output of your Nextcloud log in Admin > Logging:

{"reqId":"X5TUPGsKEkBMSD9UafGr","level":1,"time":"2022-03-16T05:09:35+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Updater::maintenanceEnabled: Turned on maintenance mode","userAgent":"--","version":"22.2.5.1"}
{"reqId":"X5TUPGsKEkBMSD9UafGr","level":0,"time":"2022-03-16T05:09:35+00:00","remoteAddr":"","user":"--","app":"core","method":"","url":"--","message":"starting upgrade from 22.2.5.1 to 23.0.2.1","userAgent":"--","version":"22.2.5.1"}
{"reqId":"X5TUPGsKEkBMSD9UafGr","level":1,"time":"2022-03-16T05:09:35+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Deprecated event type for \\OC\\Repair::step: Symfony\\Component\\EventDispatcher\\GenericEvent is used","userAgent":"--","version":"22.2.5.1"}
{"reqId":"X5TUPGsKEkBMSD9UafGr","level":1,"time":"2022-03-16T05:09:35+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Repair::step: Repair step: Repair MySQL collation","userAgent":"--","version":"22.2.5.1"}
{"reqId":"X5TUPGsKEkBMSD9UafGr","level":1,"time":"2022-03-16T05:09:35+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Deprecated event type for \\OC\\Repair::info: Symfony\\Component\\EventDispatcher\\GenericEvent is used","userAgent":"--","version":"22.2.5.1"}
{"reqId":"X5TUPGsKEkBMSD9UafGr","level":1,"time":"2022-03-16T05:09:35+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Repair::info: Repair info: All tables already have the correct collation -> nothing to do","userAgent":"--","version":"22.2.5.1"}
{"reqId":"X5TUPGsKEkBMSD9UafGr","level":1,"time":"2022-03-16T05:09:35+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Deprecated event type for \\OC\\Repair::step: Symfony\\Component\\EventDispatcher\\GenericEvent is used","userAgent":"--","version":"22.2.5.1"}
{"reqId":"X5TUPGsKEkBMSD9UafGr","level":1,"time":"2022-03-16T05:09:35+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Repair::step: Repair step: Repair SQLite autoincrement","userAgent":"--","version":"22.2.5.1"}
{"reqId":"X5TUPGsKEkBMSD9UafGr","level":1,"time":"2022-03-16T05:09:35+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Deprecated event type for \\OC\\Repair::step: Symfony\\Component\\EventDispatcher\\GenericEvent is used","userAgent":"--","version":"22.2.5.1"}
{"reqId":"X5TUPGsKEkBMSD9UafGr","level":1,"time":"2022-03-16T05:09:35+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Repair::step: Repair step: Copy data from accounts table when migrating from ownCloud","userAgent":"--","version":"22.2.5.1"}
{"reqId":"X5TUPGsKEkBMSD9UafGr","level":1,"time":"2022-03-16T05:09:36+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Deprecated event type for \\OC\\Repair::step: Symfony\\Component\\EventDispatcher\\GenericEvent is used","userAgent":"--","version":"22.2.5.1"}
{"reqId":"X5TUPGsKEkBMSD9UafGr","level":1,"time":"2022-03-16T05:09:36+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Repair::step: Repair step: Drop account terms table when migrating from ownCloud","userAgent":"--","version":"22.2.5.1"}
{"reqId":"X5TUPGsKEkBMSD9UafGr","level":1,"time":"2022-03-16T05:09:36+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Updater::dbUpgradeBefore: Updating database schema","userAgent":"--","version":"22.2.5.1"}
{"reqId":"X5TUPGsKEkBMSD9UafGr","level":1,"time":"2022-03-16T05:09:45+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Updater::dbUpgrade: Updated database","userAgent":"--","version":"22.2.5.1"}
{"reqId":"X5TUPGsKEkBMSD9UafGr","level":1,"time":"2022-03-16T05:09:45+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Updater::incompatibleAppDisabled: Disabled incompatible app: files_automatedtagging","userAgent":"--","version":"22.2.5.1"}
{"reqId":"X5TUPGsKEkBMSD9UafGr","level":1,"time":"2022-03-16T05:09:45+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Updater::incompatibleAppDisabled: Disabled incompatible app: files_retention","userAgent":"--","version":"22.2.5.1"}
{"reqId":"X5TUPGsKEkBMSD9UafGr","level":1,"time":"2022-03-16T05:09:45+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Updater::incompatibleAppDisabled: Disabled incompatible app: user_saml","userAgent":"--","version":"22.2.5.1"}

...

{"reqId":"X5TUPGsKEkBMSD9UafGr","level":0,"time":"2022-03-16T05:09:50+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Updater::checkAppStoreAppBefore: Checking for update of app \"user_saml\" in appstore","userAgent":"--","version":"22.2.5.1"}
{"reqId":"X5TUPGsKEkBMSD9UafGr","level":0,"time":"2022-03-16T05:09:50+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Updater::checkAppStoreApp: Checked for update of app \"user_saml\" in appstore","userAgent":"--","version":"22.2.5.1"}
{"reqId":"X5TUPGsKEkBMSD9UafGr","level":3,"time":"2022-03-16T05:09:50+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"App \"SSO & SAML authentication\" cannot be installed because it is not compatible with this version of the server.","userAgent":"--","version":"22.2.5.1","exception":{"Exception":"Exception","Message":"App \"SSO & SAML authentication\" cannot be installed because it is not compatible with this version of the server.","Code":0,"Trace":[{"file":"/snap/nextcloud/29929/htdocs/lib/private/legacy/OC_App.php","line":440,"function":"installApp","class":"OC\\Installer","type":"->"},{"file":"/snap/nextcloud/29929/htdocs/lib/private/Updater.php","line":424,"function":"enable","class":"OC_App","type":"->"},{"file":"/snap/nextcloud/29929/htdocs/lib/private/Updater.php","line":277,"function":"upgradeAppStoreApps","class":"OC\\Updater","type":"->"},{"file":"/snap/nextcloud/29929/htdocs/lib/private/Updater.php","line":133,"function":"doUpgrade","class":"OC\\Updater","type":"->"},{"file":"/snap/nextcloud/29929/htdocs/core/Command/Upgrade.php","line":235,"function":"upgrade","class":"OC\\Updater","type":"->"},{"file":"/snap/nextcloud/29929/htdocs/3rdparty/symfony/console/Command/Command.php","line":255,"function":"execute","class":"OC\\Core\\Command\\Upgrade","type":"->"},{"file":"/snap/nextcloud/29929/htdocs/3rdparty/symfony/console/Application.php","line":1009,"function":"run","class":"Symfony\\Component\\Console\\Command\\Command","type":"->"},{"file":"/snap/nextcloud/29929/htdocs/3rdparty/symfony/console/Application.php","line":273,"function":"doRunCommand","class":"Symfony\\Component\\Console\\Application","type":"->"},{"file":"/snap/nextcloud/29929/htdocs/3rdparty/symfony/console/Application.php","line":149,"function":"doRun","class":"Symfony\\Component\\Console\\Application","type":"->"},{"file":"/snap/nextcloud/29929/htdocs/lib/private/Console/Application.php","line":211,"function":"run","class":"Symfony\\Component\\Console\\Application","type":"->"},{"file":"/snap/nextcloud/29929/htdocs/console.php","line":99,"function":"run","class":"OC\\Console\\Application","type":"->"},{"file":"/snap/nextcloud/29929/htdocs/occ","line":11,"args":["/snap/nextcloud/29929/htdocs/console.php"],"function":"require_once"}],"File":"/snap/nextcloud/29929/htdocs/lib/private/Installer.php","Line":132,"CustomMessage":"App \"SSO & SAML authentication\" cannot be installed because it is not compatible with this version of the server."}}

Are there any ways out of this situation that will not bring more problems later? Other than waiting for a new version of the app?

Why are you unable to restore a backup?

That makes no sense. Are you sure you did you not make backups as part of your auto-update?

As you now know, you need to restore the backup to safely restore saml and you must backup to protect yourself. Nothing else can protect you If you do not backup. Backup before every single update.

I’ve got backups of both the files and the database. But how does that help me? Even if I set up a brand new server with NC 22 and the data from backup, Snap will auto-upgrade to NC 23 within a day and then I’m back to this situation again.

In that case you’ll want to disable the snap auto-update and rollback. I have no idea how snap works, but should be easy enough to google.

Haaa, snap, complexity over complexity when it goes wrong …

snap list yourappnamehere  --all

you get

Name                    Version   Rev  Tracking       Publisher   Notes
yourappnamehere  2019.3.4  212  latest/stable  jetbrains✓  disabled,classic
yourappnamehere  2020.1    216  latest/stable  jetbrains✓  classic

you do

sudo snap revert yourappnamehere   # revert to previous one
 
or 

sudo snap revert yourappnamehere --revision 212

sudo snap revert nextcloud seems to have fixed it. Thank you!

I also found that snap auto-updates cannot be permanently disabled, only temporarily for up to 90 days:
snap set system refresh.hold="$(date --date='today+90 days' --iso-8601=seconds)"

I would really like some way to auto-install minor upgrades (most importantly, security fixes) but not upgrades to new major versions. But that doesn’t seem possible with Snap.

Would it be better to use the Docker image? That would mean we need to manually keep track of new versions and re-deploy our container using the new image, right?

Snap/ docker …

I tried to used those “tools who help to simplify the administration of your server” to realise in fact they add complexity over complexity and are not a good solution for me.

Having docker app talking to another docker app, or to a snap app, having multiples conf/ vm/ vhost, subnets … what a F hell !!!

lets take the official docker app for redis for example: the official docker for redis log the event withing syslog… not even respecting the usual /var/log/redis/*.log

Yes, i would love to have a magical install/conf process but my experience say’s at one point, something will goes bad, like for you… So, in 90 day’s from now, if you didn’t fix your bug ( if it is your fault, of course ), will you be able to upgrade ?! Maybe it the snap-app who is broken, or maybe it is a snap-core bug…or else…

Anyway, happy snap revert helped fix your problem temporary…

All my servers now are bare-metal, all managed using apt or manual builds…
It’s more work, more headache sometimes, but they all works the way i want, and don’t depends overs specifics hidden snap/docker/VM’s bugs/values/conf…