*.Nextcloud.com - connectivity issues

Hi everyone,

the App Store should be up again.

Regarding the download server: it was hammered due to an update announcement to the desktop clients. We immediately rolled back the notification to free up bandwidth on our download server, but the notification was out already and we basically triggered a DDoS with the desktop clients on our download server. (overwhelmed by our own success I guess :see_no_evil:) This took quite some while but the load gets lower in the last hours.

Sorry for the inconvenience. We are about to change the distribution of the desktop client a bit so that this does not happen again, but we also only have a limited time so this will take some time.

Regarding the app store: there was a different issue where one process went havoc. We fixed this , but didn’t found the root cause yet and we are still investigating.

Sorry for the trouble here.

Cheers
Morris

10 Likes

apps.nextcloud.com loads here. :thinking:

Hello @MorrisJobke,

nice to hear from you.

Down from my home as well :slightly_frowning_face:

Now it is availible for me as well. But it loads slow and it was up before.
So i assume there still is an issue ongoing.

I can hit it now, but it doesn’t seem very healthy. Very slow and thumbnails aren’t loading, but that’s a significant improvement over flat-out down. @MorrisJobke would you folks mind writing a post-mortem once you get to the bottom of this?

2 Likes

Thanks @MorrisJobke !

Quite a few people had difficulty updating because of the self-ddos, which thankfully is resolved, however it seems that using the built-in nextcloud updater would leave people in a broken state when this happened, the only possible method is to restore from backup.

Are you folks aware of this problem, and tracking it so it will not be a problem for the future?

1 Like

What do you mean with broken state? Because the updater should protect from that. It first downloads, then extracts, verifies and only then starts to go into maintenance mode and replaces files. So all that is broken is that the updater never finishes the download and then it’s how it is. Not ideal, but at least the instance itself is not broken or down.

The site would get put into maintenance mode, which is expected. However, the PHP timeout would be reached because the download would never complete. The state file would have state 4, but the site would be still in maintenance mode. If you either tried to remove maintenance mode, or tried to remove the state file, you would have a broken site, as some of the files were moved before they should have been moved. The only way to recover is to restore from backups.

You will find a number of threads in this forum where people who failed update got stuck in this state.

I can’t really figure out if its caused by using different update methods (admin panel or occ comand) or timouts in the php processor. I used the occ command and it took around half an hour downloading the update at 12kb/s but it worked in the end without timeout or going into maintenance before the download finished.

The only thing i can think of currently is a timeout while updating the apps because at this point we are in maintenance mode.

Right … this might be the only one, but then again it would just skip the update check for this app and take long. Again: if this happens in the context of a web request, then the webserver could kill the process and then the instance is indeed in an intermediate state. That’s why we recommend to use the CLI for updates as there the webserver part is ruled out. Sorry for the trouble.

Even using the CLI you end up upgrading Nextcloud without upgrading any of the apps, so any incompatible app is disabled and can’t be re-enabled.

Seems to be up again! I’m able to install/update in normal speed.

I only used the occ method, but that uses php as well. I think the timeout will depend on what you have set in your php configuration.

That is greatly exaggerated.
When stuck on the download step, simply running

sudo -u www-data php occ maintenance:repair

will reset the update process… Worked for me every single time since v.14 (I think)…

1 Like

apps.nextcloud.com is down again @MorrisJobke

#edit: and it’s back up :slight_smile: thanks

Nope,
It’s 4 am.
https://apps.nextcloud.com/ still is not working.

1 Like

Still down. Managing apps really shouldn’t be dependent on the app store being online. As it stands now, one cannot even enable/disable currently installed apps without occ access.

1 Like

Still down for me,stuck with a brand new installation trying to configure the apps. Wondering looking at this post if I should consider something else… Scracthing my head for a day, until I saw the logs and forums. I thought no way, online access blocks me at looking at local apps? but then this… please help with new installation. This is a complete blocker for installation, I can confirm

    {"reqId":"0WvW0b8qqaZo8U2s5njd","level":2,"time":"2020-10-11T07:38:51+00:00","remoteAddr":"172.19.0.3","user":"admin","app":"appstoreFetcher","method":"GET","url":"/settings/apps/list","message":"Could not connect to appstore: cURL error 7: Failed to connect to apps.nextcloud.com port 443: Connection timed out (see https://curl.haxx.se/libcurl/c/libcurl-errors.html)","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36","version":"20.0.0.9"}

I have the exact same problem, just installed nextcloud on my VPS, and I can’t really do anything with it because appstore is down