Stuck in Maintenance Mode after trying to upgrade apps

ℹ️ Support 📦 Appliances (Docker, Snappy, VM, NCP, AIO)
1

Hello!

I’ve updated my Nextcloud instance from 19 to 20 and with that came a bunch of updates to the apps.
When I tried to start updating them earlier this evening my instance went into Maintenance Mode and I’ve not been able to get out of it since.

I’m running it in Docker, so I’ve tried to get it out of Maintenance Mode by running:
docker exec --user www-data Nextcloud php occ maintenance:mode --off
However it just gets stuck and stays there for about 20 minutes until it times out.

I’ve also tried to change the maintenance mode from true to false in the config file, but it just quickly changes it back to true.

In a log file I was able to find this:

{“reqId”:“HCr9g3MTYx7aoJL5MtGD”,“level”:3,“time”:“2020-10-12T18:47:23+00:00”,“remoteAddr”:“192.168.1.202”,“user”:“–”,“app”:“remote”,“method”:“GET”,“url”:“/status.php”,“message”:{“Exception”:“Doctrine\DBAL\DBALException”,“Message”:“Failed to connect to the database: An exception occurred in driver: SQLSTATE[HY000] [2002] Connection refused”,“Code”:0,“Trace”:[{“file”:“/var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php”,“line”:1449,“function”:“connect”,“class”:“OC\DB\Connection”,“type”:“->”,“args”:},{“file”:“/var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php”,“line”:892,“function”:“getWrappedConnection”,“class”:“Doctrine\DBAL\Connection”,“type”:“->”,“args”:},{“file”:“/var/www/html/lib/private/DB/Connection.php”,“line”:194,“function”:“executeQuery”,“class”:“Doctrine\DBAL\Connection”,“type”:“->”,“args”:[“SELECT * FROM oc_appconfig”,,,null]},{“file”:“/var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Query/QueryBuilder.php”,“line”:206,“function”:“executeQuery”,“class”:“OC\DB\Connection”,“type”:“->”,“args”:[“SELECT * FROM oc_appconfig”,,]},{“file”:“/var/www/html/lib/private/DB/QueryBuilder/QueryBuilder.php”,“line”:217,“function”:“execute”,“class”:“Doctrine\DBAL\Query\QueryBuilder”,“type”:“->”,“args”:},{“file”:“/var/www/html/lib/private/AppConfig.php”,“line”:332,“function”:“execute”,“class”:“OC\DB\QueryBuilder\QueryBuilder”,“type”:“->”,“args”:},{“file”:“/var/www/html/lib/private/AppConfig.php”,“line”:110,“function”:“loadConfigValues”,“class”:“OC\AppConfig”,“type”:“->”,“args”:},{“file”:“/var/www/html/lib/private/AppConfig.php”,“line”:288,“function”:“getApps”,“class”:“OC\AppConfig”,“type”:“->”,“args”:},{“file”:“/var/www/html/lib/private/legacy/OC_App.php”,“line”:949,“function”:“getValues”,“class”:“OC\AppConfig”,“type”:“->”,“args”:[false,“installed_version”]},{“file”:“/var/www/html/lib/private/Server.php”,“line”:673,“function”:“getAppVersions”,“class”:“OC_App”,“type”:“::”,“args”:},{“file”:“/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php”,“line”:155,“function”:“OC\{closure}”,“class”:“OC\Server”,“type”:“->”,“args”:[“*** sensitive parameters replaced “]},{“file”:”/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php",“line”:118,“function”:“OC\AppFramework\Utility\{closure}”,“class”:“OC\AppFramework\Utility\SimpleContainer”,“type”:“->”,“args”:[" sensitive parameters replaced “]},{“file”:”/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php",“line”:122,“function”:“offsetGet”,“class”:“Pimple\Container”,“type”:“->”,“args”:[“OC\Memcache\Factory”]},{“file”:“/var/www/html/lib/private/ServerContainer.php”,“line”:156,“function”:“query”,“class”:“OC\AppFramework\Utility\SimpleContainer”,“type”:“->”,“args”:[“OC\Memcache\Factory”,true]},{“file”:“/var/www/html/lib/private/Server.php”,“line”:1682,“function”:“query”,“class”:“OC\ServerContainer”,“type”:“->”,“args”:[“OC\Memcache\Factory”]},{“file”:“/var/www/html/lib/private/Server.php”,“line”:1022,“function”:“getMemCacheFactory”,“class”:“OC\Server”,“type”:“->”,“args”:[]},{“file”:“/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php”,“line”:155,“function”:“OC\{closure}”,“class”:“OC\Server”,“type”:“->”,“args”:[" sensitive parameters replaced “]},{“file”:”/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php",“line”:118,“function”:“OC\AppFramework\Utility\{closure}”,“class”:“OC\AppFramework\Utility\SimpleContainer”,“type”:“->”,“args”:[" sensitive parameters replaced “]},{“file”:”/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php",“line”:122,“function”:“offsetGet”,“class”:“Pimple\Container”,“type”:“->”,“args”:[“OCP\Lock\ILockingProvider”]},{“file”:“/var/www/html/lib/private/ServerContainer.php”,“line”:156,“function”:“query”,“class”:“OC\AppFramework\Utility\SimpleContainer”,“type”:“->”,“args”:[“OCP\Lock\ILockingProvider”,true]},{“file”:“/var/www/html/lib/private/Server.php”,“line”:1982,“function”:“query”,“class”:“OC\ServerContainer”,“type”:“->”,“args”:[“OCP\Lock\ILockingProvider”]},{“file”:“/var/www/html/lib/private/Files/View.php”,“line”:118,“function”:“getLockingProvider”,“class”:“OC\Server”,“type”:“->”,“args”:[]},{“file”:“/var/www/html/lib/private/Server.php”,“line”:396,“function”:“__construct”,“class”:“OC\Files\View”,“type”:“->”,“args”:[]},{“file”:“/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php”,“line”:155,“function”:“OC\{closure}”,“class”:“OC\Server”,“type”:“->”,“args”:[" sensitive parameters replaced “]},{“file”:”/var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php",“line”:118,“function”:“OC\AppFramework\Utility\{closure}”,“class”:“OC\AppFramework\Utility\SimpleContainer”,“type”:“->”,“args”:[" sensitive parameters replaced ***”]},{“file”:“/var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php”,“line”:122,“function”:“offsetGet”,“class”:“Pimple\Container”,“type”:“->”,“args”:[“OC\Files\Node\HookConnector”]},{“file”:“/var/www/html/lib/private/ServerContainer.php”,“line”:156,“function”:“query”,“class”:“OC\AppFramework\Utility\SimpleContainer”,“type”:“->”,“args”:[“OC\Files\Node\HookConnector”,true]},{“file”:“/var/www/html/lib/private/Server.php”,“line”:1329,“function”:“query”,“class”:“OC\ServerContainer”,“type”:“->”,“args”:[“OC\Files\Node\HookConnector”]},{“file”:“/var/www/html/lib/base.php”,“line”:596,“function”:“boot”,“class”:“OC\Server”,“type”:“->”,“args”:},{“file”:“/var/www/html/lib/base.php”,“line”:1092,“function”:“init”,“class”:“OC”,“type”:“::”,“args”:},{“file”:“/var/www/html/status.php”,“line”:37,“args”:[“/var/www/html/lib/base.php”],“function”:“require_once”}],“File”:“/var/www/html/lib/private/DB/Connection.php”,“Line”:67,“CustomMessage”:“–”},“userAgent”:“Mozilla/5.0 (Windows) mirall/3.0.2stable-Win64 (build 20200924) (Nextcloud)”,“version”:“20.0.0.9”}

Any idea what the issue could be?

2

to close the maintenance mode you have to run an specific occ statement in your NC root directory.

" sudo -u www-data php occ maintenance:mode --off"

As I understood you running docker, so in your case you have to run this inside your docker container. At least I guess because i only used standalone solutions for operation with NC.

Whatever you have to run this statement usually in /var/www/nextcloud.

3

Make sure that your database is up and running and can be accessed from Nextcloud.

1 Like
4

I mentioned in my original post that I already tried this.

5

Thanks for the suggestion. I’ve just checked up on it and the database is accessible.

6

So the problem has been solved and the error message isn’t shown anymore, right?

8

Correct. The error has not shown up in the log since yesterday.

9

However it is still stuck in Maintenance Mode and if I try to change anything in the config file, it just reverts back to it’s previous state in a matter of seconds.

10

That’s not normal AFAIK. The config file should not change unless you edit it or run occ. Do you have some kind of things going on that reset the state of the filesystem?

11

Not as far as I know?
It’s a docker running on an Unraid server.

And I’m not able to run anything with occ as it just locks up, when I try to use it.

12

You posted a log file in the first post but said that you had solved the error already. It is not useful anymore. Delete or move the logfile somewhere else and then try to do something with your instance. Then post the fresh logfile so we can actually see what is currently going on.

Also, did you try ./occ upgrade already?

13

I finally got the Nextcloud instance working again. Turns out that I had to close down the docker instance and then change the maintenance to false in the config file.
When I then started the docker instance again, it was no longer in Maintenance Mode.

However, there is still an issue with occ.
It doesn’t really matter what I try to run with occ (upgrade or maintenance:mode --off) it just stalls and nothing happens.
Nothing visible in the logs either.

I’ve also tried to run a cron with:
docker exec Nextcloud php -f /var/www/html/cron.php
Which after a while gives me this:

PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 29360256 bytes) in /var/www/html/lib/private/Log/ExceptionSerializer.php on line 137
PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 29360256 bytes) in /var/www/html/lib/private/Log.php on line 208
Script Finished Oct 13, 2020 19:13.13

All of this worked just fine before I upgraded from 19 to 20.

Edit:
I should probably mention that when I run anything related to occ it must do something, because it takes up 2 full cores of my CPU for a very long time, when I run for example occ upgrade or occ maintenance:mode --off

14

I have now tried to run the following and just let it run until something happened:
docker exec --user www-data Nextcloud php occ files:cleanup

I started it yesterday at 21:42 and at 04:53 this morning it stopped with:

PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 29360256 bytes) in /var/www/html/lib/private/Log/ExceptionSerializer.php on line 137
PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 29360256 bytes) in /var/www/html/lib/private/Log.php on line 208

15

I tried to run a maintenance:repair at 7:35 and at 11:40 it spit out the exact same php fatal error as above. I don’t know exactly what it is trying to do, but whatever it is, it is taking up more memory than I have available!

16

Since it was asking for more memory, I changed the memory limit to 2GB:
image

However, the error remains the same, when I try to run a repair after about 4 hours and 20 minutes:

PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 29360256 bytes) in /var/www/html/lib/private/Log/ExceptionSerializer.php on line 137
PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 29360256 bytes) in /var/www/html/lib/private/Log.php on line 208

17

I’ve yet to find a solution to this problem and it is rendering some features of Nextcloud useless! :frowning:

18

I too am stuck on it with upgrade from 22.2.0 to 22.2.2, and since I have it on a shared hosting, I do not have root access to run any commands. What can I do?
Edit: I scanned my folders in NC root, and could see: config/config.php file, where the Maintenance Mode was true. I changed the same to false, then refreshed my cloud page, and it ran the upgrade without any issues other than the “circles” app giving the internal error, which I solved by renaming/deleting the app and folder.

19

Updated from 19.0.1 to 19.0.13 to 20.0.14
faced the same issue. used occ to remove mail app
php occ app:remove mail
then restarted the upgrade. and it worked

1 Like
20

similar issue here:

Nextcloud: 25.0.5
PHP 8.1.17
OS: RHEL 8.7

doing automatic app updates during night:

2023.04.21 05:22:06 - occ app: drop_account - starting update
2023.04.21 05:22:11 - occ app: gpxpod - starting update
2023.04.21 05:22:43 - occ app: breezedark - starting update

This Morning users did report, that the Cloud is in Maint-Mode.

So, i had to turn it off manually.

There must be an issue either in the Cloud-Server stack or in the app update procedure itself.

There were NO ISSUES on:

  • Network
  • Database
  • other services (like PHP-FPM,Redis and so on)

There needs a mechanism in place, where apps make always sure, that the Maint.-Mode is disabled after updating - otherwise this means: Server Downtime for many users