Semi-functional instance after container upgrade

claydough · September 19, 2022, 5:02am

NextCloudPi version: 1.49.1
Nextcloud version (eg, 20.0.5): 24.0.3.2
Operating system and version (eg, Ubuntu 20.04): Debian 11
Apache or nginx version (eg, Apache 2.4.25): 2.4.54
PHP version (eg, 7.4): 7.4.30

The issue you are facing:

I’m running Nextcloud in an NextCloudPi Docker container. I also have auto-updates running.

The other day I got a notification that the version of Nextcloud that my system had just auto-updated to couldn’t run on the old PHP version my container was running. I figured that was probably a sign that it was time to just update the container.

When I tried to run nc-backup as a precaution in case something went wrong with the new container, I couldn’t because Nextcloud couldn’t run (because of the PHP issue). So I figured I had to roll the dice.

Unfortunately, after deploying the latest container, I’m encountering some issues. First, in the main web interface (not the NCP admin interface) I’m getting sporadic internal server errors. If I hit refresh 1-5 times, whatever page I’m accessing eventually loads.

The other major issue is that most of my apps stopped working. Most importantly, the 2FA TOTP app stopped functioning. For the time being I’m able to use my backup codes, but I’d really like to get that working again. Perhaps related, I can’t access the App Store. I can get to the Apps page, but I can only see apps that are already on my system–I appear to be able to enable/disable those, but I haven’t tried.

I’m hoping/wondering if this is a common issue with the latest upgrade or newest container, and am crossing my fingers that there’s a known way to resolve it. (I’ve tried looking through the forums, but sadly haven’t found anything.)

I’m very grateful for any advice or help!

Is this the first time you’ve seen this error? Y

Steps to replicate it:

Auto-upgrade to latest NextCloud version
Receive notification that PHP is out of date
Update NextCloudPi container to latest

The output of your Nextcloud log in Admin > Logging (contains thousands of lines like the following):

{"reqId":"YyftW-6NkQ4LxomA3uI46gAAABU","level":3,"time":"2022-09-19T04:17:31+00:00","remoteAddr":"router address","user":"--","app":"core","method":"GET","url":"/index.php/apps/serverinfo/update","message":"NOAUTH Authentication required.","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36","version":"24.0.3.2","exception":{"Exception":"RedisException","Message":"NOAUTH Authentication required.","Code":0,"Trace":[{"file":"/data/nextcloud/lib/private/Memcache/Redis.php","line":48,"function":"get","class":"Redis","type":"->"},{"file":"/data/nextcloud/lib/private/App/InfoParser.php","line":58,"function":"get","class":"OC\\Memcache\\Redis","type":"->"},{"file":"/data/nextcloud/lib/private/App/AppManager.php","line":506,"function":"parse","class":"OC\\App\\InfoParser","type":"->"},{"file":"/data/nextcloud/lib/private/App/AppManager.php","line":521,"function":"getAppInfo","class":"OC\\App\\AppManager","type":"->"},{"file":"/data/nextcloud/lib/private/legacy/OC_App.php","line":568,"function":"getAppVersion","class":"OC\\App\\AppManager","type":"->"},{"file":"/data/nextcloud/lib/private/legacy/OC_App.php","line":891,"function":"getAppVersion","class":"OC_App","type":"::"},{"file":"/data/nextcloud/lib/private/legacy/OC_Util.php","line":1225,"function":"shouldUpgrade","class":"OC_App","type":"::"},{"file":"/data/nextcloud/lib/public/Util.php","line":579,"function":"needUpgrade","class":"OC_Util","type":"::"},{"file":"/data/nextcloud/lib/private/legacy/OC_Template.php","line":116,"function":"needUpgrade","class":"OCP\\Util","type":"::"},{"file":"/data/nextcloud/lib/private/legacy/OC_Template.php","line":78,"function":"initTemplateEngine","class":"OC_Template","type":"::"},{"file":"/data/nextcloud/lib/private/legacy/OC_Template.php","line":324,"function":"__construct","class":"OC_Template","type":"->"},{"file":"/data/nextcloud/index.php","line":74,"function":"printExceptionErrorPage","class":"OC_Template","type":"::"}],"File":"/data/nextcloud/lib/private/Memcache/Redis.php","Line":48,"CustomMessage":"--"}

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

$CONFIG = array (
  'passwordsalt' => 'XYZ,
  'secret' => 'XYZ,
  'trusted_domains' => 
  array (
    0 => 'localhost',
    11 => 'XYZ',
    1 => 'XYZ',
    5 => 'nextcloudpi.local',
    7 => 'nextcloudpi',
    8 => 'nextcloudpi.lan',
    6 => 'XYZ',
    20 => 'XYZ',
    21 => 'XYZ',
    22 => '_',
    3 => 'localhost',
    14 => 'XYZ',
    '' => 'localhost',
  ),
  'datadirectory' => '/data/nextcloud/data',
  'dbtype' => 'mysql',
  'version' => '24.0.3.2',
  'overwrite.cli.url' => 'https://localhost/',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'ncadmin',
  'dbpassword' => 'XYZ',
  'installed' => true,
  'instanceid' => 'XYZ',
  'memcache.local' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' => 
  array (
    'host' => '/var/run/redis/redis.sock',
    'port' => 0,
    'timeout' => 0.0,
    'password' => 'XYZ',
  ),
'tempdirectory' => '/var/www/nextcloud/data/tmp',
  'mail_smtpmode' => 'sendmail',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_from_address' => 'admin',
  'mail_domain' => 'ownyourbits.com',
  'preview_max_x' => '2048',
  'preview_max_y' => '2048',
  'jpeg_quality' => '60',
  'overwriteprotocol' => 'https',
  'trusted_proxies' => 
  array (
    0 => 'XYZ',
    1 => 'XYZ',
    2 => '_',
    11 => 'XYZ',
    12 => '::1',
    13 => 'localhost',
    14 => '',
  ),
  'twofactor_enforced' => 'true',
  'twofactor_enforced_groups' => 
  array (
    0 => '2FA',
  ),
  'twofactor_enforced_excluded_groups' => 
  array (
  ),
  'maintenance' => false,
  'loglevel' => 2,
  'theme' => '',
);

The output of your Apache/nginx/system log in /var/log/____ (error.log contains thousands of lines as follows):

[Sun Sep 18 20:29:03.626843 2022] [proxy_fcgi:error] [pid 123:tid 140074737964800] [client 'container IP':46588] AH01071: Got error 'PHP message: PHP Fatal error:  Uncaught Doctrine\\DBAL\\Exception: Failed to connect to the database: An exception occurred in the driver: SQLSTATE[HY000] [1045] Access denied for user 'ncadmin'@'localhost' (using password: YES) in /data/nextcloud/lib/private/DB/Connection.php:139\nStack trace:\n#0 /data/nextcloud/3rdparty/doctrine/dbal/src/Connection.php(1519): OC\\DB\\Connection->connect()\n#1 /data/nextcloud/3rdparty/doctrine/dbal/src/Connection.php(1041): Doctrine\\DBAL\\Connection->getWrappedConnection()\n#2 /data/nextcloud/lib/private/DB/Connection.php(261): Doctrine\\DBAL\\Connection->executeQuery()\n#3 /data/nextcloud/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php(345): OC\\DB\\Connection->executeQuery()\n#4 /data/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php(281): Doctrine\\DBAL\\Query\\QueryBuilder->execute()\n#5 /data/nextcloud/lib/private/AppConfig.php(411): OC\\DB\\QueryBuilder\\QueryBuilder->execute()\n#6 /data/nextcloud/lib/private/AppConfig.php(176): OC\\AppConfig->loadConfigValues()\n#7 /data/n...'

ZendaiOwl · September 23, 2022, 6:39pm

Yes, there is currently an issue with updating/upgrading to the latest version (1.50.0), theCalcaholic is working on fixing the workflow for the release so it’ll take a little while longer before it’s working. There is currently an issue (#1568) on GitHub where I’ve outlined a test workflow going through things that currently go wrong when performing the upgrade and a couple ways you can restore it.

Go through it carefully & take care so you don’t accidentally delete your data if you want to give it a try to restore it yourself

EDIT: Another user posted about update/upgrade issues as well with a more fitting title, so let’s use that topic to track & gather information regarding the upgrade issues and troubleshooting, keeping it gathered in one place as much as possible, thank you

ZendaiOwl · September 23, 2022, 11:38pm

@claydough
If you restore your instance again carefully with the method mentioned in the article about the docker volume, it should make it work again for you. When the docker image is updated on the docker registry you can update by pulling the new image, stopping & removing the old then starting with the new image it’s still v1.49.1 when I checked

claydough · September 25, 2022, 2:54am

Thanks so much @ZendaiOwl ! The level of detail in your GitHub posts is incredible, and all the more amazing given your busyness with school work. Your time and efforts are very much appreciated!

Unfortunately, I have to admit that I’m getting a bit lost in the various procedures you experimented with, and can’t quite hone in on which is the procedure that’s most likely to help my situation.

Sadly, I wasn’t able to run a backup procedure because the PHP issue wouldn’t let NextCloud start at all, which drove my decision to just take a shot at directly upgrading the whole container, which then seems to have made things worse.

On the positive side, my data is safe and well backed up, but my NextCloud instance (settings, etc.) is not.

If it’s not too much trouble, any help you can give pointing me to the right starting point in your write-up would be greatly appreciated. I’m also okay to wait if you think an automated fix may be forthcoming. Thanks again!

ZendaiOwl · September 25, 2022, 3:04am

Thank you

If you check your data volume for a backup and your data, these are separate, move it outside of the docker volume directory, you can simply stop & remove container, images & volume afterwards and then recreate it.

If you don’t remove your files they will be deleted when you remove the volume, you can try remove container & images first if you want but still move your data first

When it’s back up you can restore Nextcloud with the backup, move your data files back in place, use ncc to scan for external files and they will be added again

Edit: If you move your files back before you restore the backup, your files will also be deleted. Be careful

claydough · September 25, 2022, 3:09am

Very cool! Thanks again for the kind (and fast!) help. I’ll take a shot at this and see if I can get myself back up and running.

claydough · September 25, 2022, 3:38am

My apologies, I’m usually not the helpless type, but I seem to be running into unusual hiccups here. From what I can see, it doesn’t look like NCP has generated any backups in a long time (almost 2 years), and even that file seems to be in a weird location. Here’s what I’m seeing:

Here’s where I’d expect to find the “ncp-update-backups” directory, but none exists

XYZ@<server>:/var/lib/docker/volumes/nextcloudpi/_data/nextcloud/data# ls -al
total 106912
drwxrwx---  6 www-data www-data      4096 Sep 24 19:40 .
drwxr-x--- 15 www-data www-data      4096 Sep 18 20:57 ..
drwxr-xr-x 10 www-data www-data      4096 Sep 22 15:05 appdata_oc64n71zslhy
-rw-r--r--  1 www-data www-data         0 Sep 16 13:06 audit.log
drwxr-xr-x  4 www-data www-data      4096 Sep 16 14:08 <my user name>
drwxr-xr-x  2 www-data www-data      4096 Sep 16 13:06 files_external
-rw-r--r--  1 www-data www-data       542 Sep 16 13:05 .htaccess
-rw-r--r--  1 www-data www-data         0 Sep 16 13:05 index.html
-rw-r-----  1 www-data www-data    665336 Sep 24 19:51 nextcloud.log
-rw-r-----  1 www-data www-data 108767875 Sep 24 19:40 nextcloud.log.1
-rw-r--r--  1 root     root          1024 Sep 17 14:30 .nextcloud.log.1.swp
-rw-r--r--  1 www-data www-data         0 Sep 16 13:05 .ocdata
drwxr-xr-x  2 www-data www-data      4096 Sep 24 19:51 tmp

A couple directories up is where I’m seeing a backup tarball, but it’s from almost 2 years ago

XYZ@<server>:/var/lib/docker/volumes/nextcloudpi/_data# ls -al
total 10545628
drwxr-xr-x  7 root             root                    4096 Sep 16 13:04 .
drwxr-xr-x  3 root             root                    4096 Oct 30  2020 ..
drwxr-xr-x  3 root             root                    4096 Sep 14 12:50 bin
drwxr-xr-x  5 systemd-timesync systemd-timesync        4096 Sep 18 20:29 database
drwxr-xr-x  9 root             root                    4096 Apr 19 06:26 etc
drwxr-xr-x  3 root             www-data                4096 Sep 24 06:39 ncp
drwxr-x--- 15 www-data         www-data                4096 Sep 18 20:57 nextcloud
-rw-r-----  1 root             root             10674999877 Oct 29  2020 nextcloud-bkp_20201029_1604018086.tar.gz
-rw-r--r--  1 root             root               123680045 Jul 18 04:35 nextcloud.tar.bz2

I’m getting a strong feeling that I’m just going to have to rebuild from scratch, but I hoped to see if someone with much more expertise and experience may have any last ditch efforts that could save me the pain.

ZendaiOwl · September 25, 2022, 7:38pm

I went through the backup during the test workflow it should have created one in the following possible locations. (Notice there is a typo in the first one, I don’t know if that is just the debug message itself, or the backup functions output. So check with and without the typo)

backup /var/www//nextcloud-bkp_20220923_1663943738.tar.gz generated
Backup stored at /data/nextcloud/data/ncp-update-backups/nextcloud-bkp_20220923_1663943738-24.0.3.2.tar.gz

You can find this part here in the test workflow

Can you check and see if you find anything there or in a similar location?

Edit2: You need to go deeper in your file structure, your need to cd into the nextcloud directory

Edit: Otherwise, you should be able to fix it by pulling the new image when it’s done and pushed to the docker hub. I have not tested this however so I don’t know this for sure

claydough · September 26, 2022, 12:10am

Such a bummer… I hunted all through the volume directory structure in the host and, just for thoroughness, in the container container itself (in case I was missing something simple). From within the container I also ran a find . -name *.tar.gz, and the only file found was that 2-year old backup. So strange that the upgrade didn’t trigger an automatic backup.

I guess the next best option is to sit tight for the new image and hope for the best.

I really appreciate all your time and follow-ups, @ZendaiOwl !

ZendaiOwl · September 26, 2022, 1:42am

You’re welcome and sorry it couldn’t be found. Yes, using the upgraded image when it’s available should resolve it for you, I don’t know tho as I haven’t tested it myself, but theoretically it should work considering how my tests went with the upgrade process on docker.

Heading off to bed now, good night

claydough · October 4, 2022, 2:49am

Popping in here to close the loop, in case anyone else was running into this issue (there are also some notes that may be helpful if the fix caused some new issues–more on that in a minute…).

I was extremely grateful to see that just as @ZendaiOwl expected, the new container release that came out 4-5 days ago fixed the issue for me without the need to restore from a backup (that sadly I didn’t have available).

One snag I ran into was that a new error started hitting my log file multiple times per minute. The error was Error: disk_free_space(): No such file or directory at /data/nextcloud/lib/private/Files/Storage/Local.php#395. Fortunately, the container update that just came out a couple hours ago seems to have addressed that.

So now all appears to have been worked out. If you’re finding this through a search, hopefully you find it helpful and are relieved to hear that things can be recovered without too much pain or worry!

Many thanks again to @ZendaiOwl for the awesome troubleshooting and documentation, and also to any devs who may stumble across this post for your help integrating the fixes into the releases!