Nextcloud and its planned update improvements

I just blogged about Nextcloud and it’s planned update improvements, you can find the blog post at https://statuscode.ch/2016/06/nextcloud-and-its-planned-update-improvements/ or below.

Any kind of feedback welcome! :rocket:


In the past, the update experiences with ownCloud have been difficult. It was not always clear when updates would be released for the updater app or how to move to a new major release. Apps disappeared after an update or apps were updated to an incompatible version (e.g. with a broken PHP dependency), or simply the updater had a bug and broke the whole instance. We hear you and in fact, we share the same concerns! Our goal has always been to get you the best possible update experience but there was and is room for improvement.

With Nextcloud one of our primary goals is to allow people to easily and securely host their own data. Reliable updates are a key part of the user experience and we consider it a top priority at Nextcloud. While we can’t promise to magically have fixed all existing updater problems directly from the beginning we assure you that all update issues are of critical priority to us.

We would like to share some of the changes that we’re planning for the updating process in Nextcloud. Please note that some of these are about being more open in the process, which means improvements depend on your help to be successful. Moreover, these are our thoughts and ideas and we are very open to constructive feedback, other ideas or practical help!

1. Open-sourcing all components related to the upgrade process

Some key components like the “update-server” as used by ownCloud right now are closed-source components. This means that the community has no way to improve the update experience.

A well-known example being https://github.com/owncloud/appstore-issues/issues/4, an issue open since 2014 which leads to the update server delivering wrong versions to the ownCloud server. Effectively, resulting in a broken instance.

This issue has been fixed in February 2016. Two years after it broke hundreds, if not thousands of ownCloud servers. We believe that such critical components have to be open-sourced so more people can chip in and help out.

2. Make updater ready for shared hosting and low-end hardware

The updater has in the past usually only been tested on dedicated machines. In such cases PHP has many additional functions such as executing Linux shell commands.

In many shared hosting environments these additional functions are however not available which breaks the updating process. To make the updater compatible with such environments as well we want to devise tests for the updater which also cover those scenarios.

3. Perform maintenance tasks live in the background

In the past the upgrading process executed most maintenance tasks while upgrading. This made upgrading in some scenarios an unnecessary slow experience.

We’re aiming to move some of the maintenance tasks to background jobs. This is often possible, such as for example when a new filetype icon gets added to Nextcloud there is no need to have this one already appearing directly after the update. It doesn’t harm the user if it appears some hours after the upgrade. So instead of the upgrade taking a long time some not critical changes would just propagate later, while your Nextcloud is just working as normal.

4. Add the ability to skip releases

We don’t consider it appropriate that users are not able to skip releases when updating. So for example when you want to update from Nextcloud 10 to Nextcloud 13 we want you to be able to directly go from 10 to 13 and not have to update to 11 and 12 as well. This will require some re-architecturing and, of course, more (automated) testing. We’re discussing our infrastructure and your thoughts are welcome on the forums!

5. Do not disable compatible apps

Apps like Calendar or Contacts are what make your Nextcloud your very own! Nextcloud is not only about files but also about Calendar, Contacts and all the other apps you need to do your work. Right now on every minor updates these apps get disabled and have to be enabled by an administrator again.

On the other hand, not only the Nextcloud dependency of an app should be considered when updating an app. They can specify dependencies on, for instance PHP version. If an app update requires a higher version than is installed on the server, an update should not be executed. Since the update may include security fixes though, the app should be disabled in this case and a warning sent out to the administrator. In short, this is a complicated subject but we think improving this is very important.

6. Provide the ability to easily use daily stable releases

In many cases it’s totally okay to use the daily stable branches of one of our releases. They contain all the bug fixes but no new features. However, also bug fixes from time to time can contain bugs. We want to make it easier for people to catch bugs and easier for us to debug them. Find a bug and you can tell us on which date is has been added? Awesome. You just saved another contributor a lot of time! Another thing with daily stable releases is that they can help you test a fix that we developed or find out if a problem has already been fixed before reporting it.

It’s important to add the ability to use daily branches and have updates of these running in an easy and uncomplicated way and this will be something we’ll work on.

7. Make it easier to subscribe to update notifications

While administrators right now get a little update notification when they are logged-in into Nextcloud this can be improved. We’re thinking about adding notifications using the notifications API as well as other channels such as email. (for example by sending an Nextcloud administrator automatically emails about new releases).

8. Make updating apps a breeze

Updating apps in ownCloud is something barely anybody has ever done since you manually would have to go to the admin menu, select each app and press “Update”. We believe that for many apps a kind of semi-automated or even completely automated approach is the way to go.

9. No more big delays until updates are available using the built-in updater

Once we have at least partially implemented above mentioned changes we’re believing that we’re able to push updates using the updater app in a way quicker and more stable manner. And while we should always be careful and wait a bit with releasing automated updates, having the update server open source and managing it with pull requests allows users and other contributors to see what is going on, when a release comes or why not, and participate in the process and be part of the decision making.

We’re looking forward to implementing changes in the Nextcloud updater and make it fit better with your needs. If you have any feedback leave it in your forums at help.nextcloud.com as, of course, we’re sharing OUR ideas here but we want and need YOUR input to make the result better than what we can come up with on our own!

7 Likes

Automated-apps-updating is a big deal to me. Wordpress does it very nice, take a look at them maybe?

I’m really happy that the update server is open-sourced, and that you address the issue about the long delay. On the mailing list this was always explained very vague to the end-users.

About the auto-update: will it be possible to disable this? I can imagine that some users want to upgrade the apps by them self.

1 Like

Very good point. Yes, this will very likely be the case as well. If we get
to such a point it probably will be the default but offering an opt-out
possibility is absolutely sensible.

In fact, for security reasons it makes sense it not to do for every kind of
app anyhow. For example we only might want to do this for signed
applications. The reason being here: If somebody hacks the appstore no
malware should be distributed to our users.

(as the signing happens locally on dev machines and no key material is on
the app store)

As a sysadmin I’m always torn when I consider upgrading a critical system.
Upgrading is a choice between 2 kind of risks :

  • don’t upgrade, and bear the risk of having (maybe) an unfixed bug/security hole bite you later
  • upgrade, and bear the risk of having bugs in new code bite you (maybe but probably right now)

It can happen to everyone. I’m reminded of a big fail at VMware several years ago, where an update was shipped with a timebomb that the devs forgot to remove : as a result, people that installed it saw their systems shutting down after a month or so. Ugly stuff, and we can hardly say that VMware is underfunded.

My point is that it would be very nice if that the updater displays to the administrator information more information about the version he’s about to install.
This way you could provide incentive to upgrade, and foster trust to do so.
I’m not talking about static release notes, but something dynamically fetched.

Some examples :

  • When the main version was released, and how many fixes it had since initial release.
  • Current known issues present in this version (or lack thereof). Not necessarily all of them, only the most critical ones.
  • Something along how many bugs would be fixed if he updates would be great too. Maybe a nice summary : “10 critical fixes, 30 standard fixes are included in this update”, and some links for further details. This summary would take into account the version currently installed.
2 Likes

That would be useful as very often there is a substantial list for a major upgrade, but most will be fixed in the next patch release.
That way, conscientious sysadmins could update NC to get rid of all published vulnerabilities and patch the known bugs they feel are important for their users.

We did put such lists on the forum.

True and people maintaining those threads do a good job.

Automating it would relieve you of this burden, and would provide this information to all admins, not just the ones that goes to the forums.

I was invited via twitter to share my update view here.

I’ve been running OC since ca. OC4. I’ve since had installations that I updated via
a) unzip + upload/overwrite
b) git checkout (stable branch, mostly)
c) owncloud integrated updater
d) packages on debian

None of these have worked reliably enough. I’m at a point where I only update OC If i can set aside half a day or more for that task. I had to make use of this emergency buffer more often than I’d like (last occurrence: today, package update from 9.0.1-1 to 9.0.2-1 on debian. After the update, all web pages failed with a 404 as the .htaccess had not been updated for some reason).

Long story short: I cannot recommend ownCloud to other people in the same way that I recommend, say, WordPress. The latter I suggest to pretty much anyone who can unzip a zipfile. The former is a fine piece of software, but once you need to update, things can get ugly.

This is not to bash OwnCloud. I still recommend it to many people. Not as many as I’d like, though.

3 Likes

As we’re sharing ideas about update improvements…

I’ve been running ownCloud since it’s early days (1.x probably) and have to agree with @amenthes regarding the updating.

So far the most reliable way I managed to find was to:

  1. use Gentoo’s packages
  2. then use Gentoo’s webapp-config to set handle the vserver
  3. and finally run occ from the CLI, because otherwise on my poor little ARMv5 ownCloud’s web-based update simply times out all the time.

I will update my server’s HW in the near future, but if oC somehow manages to run on a, the updater doesn’t. As such I very much welcome the relevant @LukasReschke’s suggestions :smile:

I would also welcome if exporting data (e.g. calendars or contacts in vCal/vCard) could be done via the CLI.

(As an aside, ownCloud works perfectly fine with Let’s Encrypt certs, so in the official VM and/or physical device being sold, that might be a very good idea to implement so users have automatically renewed and valid SSL certs.)

2 Likes

Another agreement here. I’ve been using owncloud since git repo’s were first made available. I love it, but generally I found the need to assume that it’s going to break after being updated. I’ve been pleasantly surprised a few times, but only a few.

Early on, many cases were either permissions problems or apps incompatible with the new version.

Later were versions being incompatible with the newer version of PHP.

I had a couple of corrupted databases along the way, and a few more issues which I don’t even remember what the solution was.

I’m really happy to hear about Nextcloud dev’s commitment to smooth upgrades for both core and apps.

What I would seriously love, would be to see some system similar to Wordpress where basic tests are run to report the percentage of people that the app worked for with specific versions of Nextcloud. In the Wordpress ecosystem, this helps to weed out badly programmed apps which are commonly causing breakage.

1 Like

Great idea. Even if apps are restricted to N+1, there is no guarantee that it will work on that version until someone tries.

That would be great, yes!

It should be an option you can enable/disable though, as some people might distrust Nc “phoning home”.

Agreed. Any phone home should be clear and optin.

  1. Add the ability to skip releases

:+1: great to see this as an idea. I think this would contribute tremendously to the possible downstream inclusion into various GNU/Linux distributions

1 Like

Faster updates (or at least more reliable information about when they’ll be released) via updater app would be a great thing.
As I have problems with the occ upgrade command (on a Synology NAS), I have to rely on the updater app.
So I’m still on OC 8.2.5 waiting for 9.0.3 via updater app. There were almost no information since 9.0.0.
I hope this will be better with Nextcloud (I’ll start from scratch to have a clean install).

That’s one of the goals. :wink:

Until then, you can always do what I did, which is to subscribe to the RSS feed of the Github releases page, so you get a notification the moment a new release is tagged, instead of waiting the weeks/months I used to in ownCloud, for the release to be announced in the updater app.

1 Like

that’s a really good hint!

Btw. the feed-url is
https://github.com/nextcloud/server/releases.atom

2 Likes