[System Failure] App Update with web-interface went sideways and now system is unusable

Background information

Over time the updates and upgrades have been going worse for me; often getting stuck or failing in unexpected ways. I might have been in here since NC10 or NC11? I’m a bit of a hack-admin so I don’t know what php occ commands to run to give any log info because I nearly only use the GUI. Also I may use the wrong words for tings at times, especially if they start with the same letter and seem synonymous; upgrade/update. And also, this happened almost a week ago, and I’ve been very busy (and I’m old) so the precision of my testimony could be questionable. I’m one version behind the latest stable version because that also won’t install correctly…

While updating Apps, I had it fail during the contacts update. Typically, I can get going again with Ctrl+Shift+R and then updating one app at a time till it’s all done. The last few app update sessions I’ve also have to do the php occ app:update --all to clear the queue.

I manage my bare metal outward facing web server with the Virtualmin system. I have a second internal server managed with Webmin just for running MySQL because reasons, and my NC reaches over to that server for database functions. Both servers use Ubuntu for the OS.

I’ve lost access to probably all may data with any NC web interface or app both on mobile devices and desktop computers. Weirdly, the password browser extension on Windows 11 and the linked folders didn’t wipe data like all the apps on Ubuntu systems did or I’d be really ducked-royally.

A synopsis of what happened

Logged in as the administrator of Nextcloud, I did the normal App Update. As usual it got stuck and failed to finish as it does most times. This time it was stuck in Maintenance mode. I did other things for about 2 hours to see if it would finish, but it didn’t.

I decided to go to a shell and run the occ commands to finish. A lot of error messages got thrown while trying to install an update to duplicatefinder and the php occ app:update --all would crap out and fail to finish throwing back to the command prompt. I tried the last command again and think I got the same plethora of error messages.

It seemed to be wise to just get rid of duplicate finder because I rarely used it, so php occ app:remove duplicatefinder. The rest of things seemed to update correctly.

At my next login I got this screen:

Internal Server Error
The server was unable to complete your request.

If this happens again, please send the technical details below to the server administrator.

More details can be found in the server log.

Technical details
Remote Address: 192.168.1.1
Request ID: Z4RCmI7ZqLgUSSifefif5QAAAAk

Image

What I tried since then

Because I though it might be some problem with cookied hash brownies somewhere I decided to log out of everywhere and reboot both my servers. My Android device cannot log back into any services; not the main NC app, nor Passwords, nor Deck, Cookbook, Notes, Talk, Tasks, DAVx^5.

Weirdly, Phone-Track seems to be logging points just fine.

On my Ubuntu desktop (at home where the server is) I can’t login via cloud.ruppssites.com/index.php/login, nor can I use the operating systems Online Accounts feature to login so calendar, contacts and files are all out of reach to access. Outside logins don’t work either, but outside is a Windows 10 PC. Windows 11 at home also can’t login.

I ran the following commands:

  • php occ upgrade
  • php occ maintenance:repair
  • php8.2 occ preview:pre-generate
  • php8.2 occ db:add-missing-primary-keys
  • php8.2 occ integrity:check-core
  • php8.2 occ app:update --all
  • php8.2 occ maintenance:mimetype:update-db
  • php8.2 occ maintenance:mimetype:update-js
  • php8.2 occ maintenance:update:htaccess
  • php8.2 occ serverinfo:update-storage-statistics
  • php8.2 occ app:update contacts this one didn’t work when running the --all but did update after a couple tries
  • php8.2 occ app:update passwords this didn’t need updating, but

So you can see I’m just “spraying random / spray & pray” running OCC commands.

I did try searching the errors and it didn’t lead to anything meaningful. I tried restarting the Redis server based on one solution from the search but as far as I can tell, that didn’t help at all.

Some closing notes

Because I’m annoyingly ignorant, I’ll need to be explicitly told what OCC commands to run to give any log or system info you’ll need to un-duck what I’ve done over the years…

I have a long term plan to spin up a new bare metal server and migrate all my sites and systems to that, but I don’t even know how to robustly migrate the data from the current Nextcloud system to a new fresh install. All I got now is the Virtualmin server backup option where on the new Virtualmin I could restore the current system with all it’s buggy flaws. Ideally, I would serialize all the individual apps data and then import them to a new NC system one at a time. That way I could drop the MySQL legacy problem I have and rely on the MariaDB server that’s running on my main webserver.

P.S. I live in the EST [GMT -5:00] time zone and work a full time day job so responses will be delayed appropriately.

This solved everything.

How to safely update Nextcloud through console

For those who prefer to read rather than watch a video, this is also included in the official documentation: Upgrade via built-in updater — Nextcloud latest Administration Manual latest documentation

This topic was automatically closed 8 days after the last reply. New replies are no longer allowed.