Nextcloud in zombie state - possible systemd issue or upgrade issue?

The Basics

  • Nextcloud Server version: 30.0.8 (maybe - see below)
  • Operating system and version: Ubuntu 22.04.5 LTS
  • Web server and version: Apache - not sure which version, managed by snap
  • Reverse proxy and version: Apache (I guess? Not sure how to check)
  • PHP version: 8.2.26
  • Is this the first time you’ve seen this error? (Yes / No): Yes
  • When did this problem seem to first start?: This morning
  • Installation method: snap
  • Are you using Cloudflare, mod_security, or similar? (Yes / No): I don’t think so. Everything is local

Summary of the issue you are facing:

Basic symptom - Nextcloud is unreachable through browser.

More detailed symptoms - please note that I’m a beginner Linux admin. I know enough CLI to read logs and run commands - so ELIF is much appreciated. I keep a running text file of what I’m doing, if you need any additional details, I can provide them. Pastebin rejected my dump of my logs from the troubleshooting I’ve done. Something about a smartfilter… :?

  1. System was extremely sluggish to ssh into and I could not do a “sudo reboot now” so I had to hard power cycle the box.

  2. When it came up, Nextcloud was still unreachable.

  3. Reviewing /var/log/syslog told me that at some point, during routine shutdown/startup of nextcloud snap, systemd core dumped.

  4. Doing a “sudo snap list --all” shows me that I have two nextclouds on my box:

||nextcloud  30.0.8snap1    46970  latest/stable  nextcloud✓  disabled|
|---|---|
||nextcloud  31.0.2snap1    47222  latest/stable  nextcloud✓  -|
  1. Doing a “sudo snap start nextcloud” reports “Started.”

  2. Looking at “sudo snap info nextcloud” says that it’s still disabled?

	name:      nextcloud
	summary:   Nextcloud Server - A safe home for all your data
	publisher: Nextcloud✓
	store-url: https://snapcraft.io/nextcloud
	contact:   https://github.com/nextcloud/nextcloud-snap
	license:   unset
	description: |
	  Access, share and protect your files, calendars, contacts, communication and
	  more at home and in your enterprise.
	commands:
	  - nextcloud.disable-https
	  - nextcloud.enable-https
	  - nextcloud.export
	  - nextcloud.import
	  - nextcloud.manual-install
	  - nextcloud.mysql-client
	  - nextcloud.mysqldump
	  - nextcloud.occ
	services:
	  nextcloud.apache:          simple, disabled, inactive
	  nextcloud.logrotate:       simple, disabled, inactive
	  nextcloud.mysql:           simple, disabled, inactive
	  nextcloud.nextcloud-cron:  simple, disabled, inactive
	  nextcloud.nextcloud-fixer: simple, disabled, inactive
	  nextcloud.php-fpm:         simple, disabled, inactive
	  nextcloud.redis-server:    simple, disabled, inactive
	  nextcloud.renew-certs:     simple, disabled, inactive
  1. “ps -aux” says that various nextcloud bits are running, so I try to access my server via https. No luck

  2. On a guess, I tried to access nextcloud via http and I see the following error on the browser:

App directory "/snap/nextcloud/current/htdocs/apps" not found! Please put the Nextcloud apps folder in the Nextcloud folder. You can also configure the location in the config.php file.

  1. I looked at /var/snap/nextcloud and this is what I see:

46970 47222 common

At this point, I’m stumped. Is it a failed upgrade? When systemd coredumped, did something get left undone and I’m in this weird middle state?

Most importantly, how do I recover?

Fortunately, my data is still there, I checked those directories - and I’m synced on my local workstation. The only downside is loss of my primary calendar at the moment, but I have it on my mobile device, so there’s that for the moment.

Help?

PHEW Nextcloud is back up. Reverting back to the previous snap restored my “current” folder and - cautiously - everything seems to be back to normal.

Shout out to @markusr@mastodon.social for pointing me in the right direction. I’ve updated my linux admin journal with lots of things should this happen again.

Helpful links, in case you have this problem:

you were too impatient, no need to revert! we’ve tested this on many instances large and small! some took as much as 10 minutes to migrate the database, especially if the instance was installed pre Nextcloud snap v. 19.x.x . Newer version migrate quicker because database row formats are already converted to “DYNAMIC”.

why did it happen?

upstream Nextcloud would expect users to make changes on the database manually, which the snap team have automated for the snap community and will be migrated by the Nextcloud snap fixer service. The process may take some time, so be patient and don’t panic!

IMPORTANT
Depending on the size of your database, number of users and system resources, this procedure may take some time! So be patient…

so go ahead, sudo snap refresh nextcloud and let it run until the system quietens down… you’ll certainly be glad you updated because its smooth sailing!

Welp, #3123 wasn’t my symptoms, mine were related to systemd coredumping in the middle of an upgrade and my current folder vanishing so… idk about being impatient. Hopefully the next auto upgrade won’t have systemd scramming…

nope haven’t seen that and we’ve certainly done a lot of testing

would be great if you’d collect your logs (run the debugging script) and creating an issue on our GitHub (GitHub · Where software is built) if you should give it another try

this must have been the issue, the latest version is nextcloud 31.0.2snap2

take a walk on the wild side and give it a shot :star_struck: sudo snap refresh nextcloud

This topic was automatically closed 8 days after the last reply. New replies are no longer allowed.