Production channel removal

It forces all open-source users using the channels to use the current stable channel and test the newest version for enterprise customers ?

In reality, for most users it probably isnā€™t that important. In enterprise environments, the setup is generally more complex, you have a storage backends, external authentication mechanisms ā€¦, there is other software and you are a bit less flexible with software versions.

But Iā€™m with @tanghus, I donā€™t see the point of removing this channel.

2 Likes

you have a storage backends, external authentication mechanisms

Iā€™m using nextcloud - I am the only user - I use multiple storage backends and authentication mechanisms, incl. Imap authentication. My primary storage backend is S3.

Iā€™m not going to even touch the enterprise comment but if youā€™re not making enough money off enterprise users to provide Q&A testing without forcing everyone to be your bug testers in the stable channel then Nextcloud needs to come up with a new business model.

2 Likes

Sure, thatā€™s not for enterprise customers exclusively in contrast to the old owncloud days.

This comment wasnā€™t really serious :wink: Seeing the development over the last few months and all the new public contracts, they canā€™t be so desperate.

Hmm. I have guess but Iā€™m not 100% sure. Take a look at the updater_server (https://github.com/nextcloud/updater_server). Iā€™ve checked plenty commits and most of the time they update stable and production at the same time. Actually I donā€™t see a huge difference. Usually they push a new major release a bit earlier to stable.

But thats only the update announcement. If you donā€™t want to update to a new major release just ignore the notification? I have done this for 17 myself and still received 16.0.6 even 17.0.1 was already announce to my instance.

I really wonder about the point here. A lot of posts here are users asking for the Nextcloud 17 update. I would assume that home users are the early adopters. If they would ship the update to their enterprise customers first the other half of the board would complain :wink:

For the record, my home setup, which I have taken very seriously and even run on actual server hardware, was using the production channel prior to this update paywalling it.

The argument that home users donā€™t benefit is invalid. I used it as a marker for when the next release was thoroughly tested.

Iā€™ve compensated by using the 17-Apache tag on Docker so I can delay when it moves to NC18.

Hi team,

So the reason behind the Production channel/Stable channel thing is rooted in some history, so let me try to explain:

  • We initially planned the Stable channel to get new releases right away, and the Production channel to get them over time, as the description explains on nextcloud.com/release-channels
  • But when we build our updater server, one or another clever person (I think it was lukas or morris) came up with the idea of incremental roll-out, that is, we only roll out Nc to a percentage of the users via the updater to catch problems before they hit everyone.
  • On top of that, our team quickly developed a habit of waiting a week or two before the Stable channel got the update, and instead using the Beta channel to get the newest version to users. You might have noticed how much our team cares about delivering stable software - I often pushed for putting things on stable quickly, as people were asking where the update was, but then I got push-back - ā€œbut what if it breaks a user systemā€ and ā€œI donā€™t want to call it stable until we know it isā€ and ā€œyou know, we can never test every scenario so letā€™s wait a few more weeksā€. Well, makes sense, right?
  • Usually, the .1 or even the .2 release was the first that was put on stable, and only then incrementally. Once I think we only did the .3 releaseā€¦
  • So in reality, we were treating the Stable channel as Production and Beta as Stable
  • Worse, we started to forget updating Production, so it was outdated and that introduced a security risk for users on there.
  • at this point, we had a Stable channel being treated as Production and a Production channel that was actually worse than Stable.

So we wanted to retire ā€˜Productionā€™ completely and just remove it.

But then, recently, we started a special build for our customers (see https://nextcloud.com/faq/ ) and that needed a channel to roll out to them. Re-purposing the now-obsolete production channel seemed to make most sense.

And so, thatā€™s what we did.

Iā€™m sorry for the mess and confusion here, we had a nice plan in the beginning but we didnā€™t really follow it so at some point we had to sync with reality. Calling a channel ā€˜Productionā€™ when it is actually worse than ā€˜Stableā€™ seemed harmful.

Now yeah, we could try to hide the production channel that delivers our customer build but given that I still get people who say ā€œwhat, there is a company behind Nextcloud that offers customer services? Oh, we decided on another product because we had no ideaā€* I think itā€™s fine to tell people that if theyā€™re at a bigger organization there is support available.

And I saw that @kesselb pretty much caught up on this in his comment, by the way :wink:

Feedback and ideas are, of course, always welcome.

  • Iā€™m not kidding. I know we try to say it everywhere we reasonably can, even annoying some users, yet others still donā€™t pick up on it :frowning:
11 Likes

hey @jospoortvliet,

thanks for clearing that up a bitā€¦ though i like to add one or two suggestions to it:

thatā€™s good. but now that the system got changed itā€™s about time to change the linked page as well.

note: early-adoptors are ALWAYS going to nag devs about newest releasesā€¦ but please donā€™t mistake them for being any majority here. though they complain a lot and very loud. most of them - speaking from my experience here on the forum - are not only early-adoptors but as well not really ready dealing with problems some new (un)stable version would bring.

i, personally, missed that out, i think. if youā€™re referring to major versions as in nc x.0.1, x.0.2 or x.0.3 - if yopuā€™re referring to RC1, RC2 or RC3 that might be the truth.

thatā€™s ok, again. but then again: why would you offer nc x.0.0 versions on stable when you had production channel offering versions >nc.0.2 ? now well it is like it is and you, of course, put thought into it. but maybe it would be nice offering ne major versions >x.0.2 at stable from now on? or come up with a checkbox where the user could enter a value hinself from which version on he wanted to be informed about ā€œstableā€ versions - setting itā€™s default to x.0.0ā€¦
on the other hand itā€™s no problem for me to wait until x.0.2 myself.

awwwā€¦ and have you taken into consideration that some app-devs would rely upon your release-schedule with former production-channel? like @mdw with his great password-app? or @nachoparker and his team with nextcloudpi?

anywaysā€¦ another thought: if you guys are withdrawing a ā€œstableā€ x.0.0-release it would help all users here if youā€™d make a note about it or we would have the more or less same discussion about ā€œwhere is my release?ā€ with every new major version. i think withdrawing releases is ok to protect users from possible harm theyā€™d do to their instancesā€¦ but please please please put word out that you did so. :wink:

anywaysā€¦ thanks for your time and understanding.
cheerio
jimmy

5 Likes

Note to self: Do not update on notification but wait at least for .0.2 and a few weeks and check the forum for issues before updating.

3 Likes

@jospoortvliet Thanks for the clarification. Please donā€™t forget in the future, there needs to be a front-page PSA when a change like this is coming. All I saw on my end was one day Iā€™m using the production channel, and the next day itā€™s been renamed enterprise and made into a paid feature. Thatā€™s not a good look.

It reminds me of what the Opsview people did years ago, and I ripped them up down and sideways over it and ultimately stopped using or recommending their product. Itā€™s a software based on Nagios with a fairly good web UI that supports configuration. Opsview Community was the FOSS version and was fully featured. One day they ā€œdiscontinuedā€ that version and replaced it with Opsview Core, which was basically the same thing, but with a few features such as clustering and SNMP traps now requiring a paid license. Then a couple years later they nixed Opsview Core and replaced it with Opsview Atom, again the same software but now imposing a 25 device limit without paying, rendering it largely useless.

I told them that isnā€™t a game you get to play with community-driven open-source software. Letā€™s not go there.

7 Likes

@jospoortvliet
I am sorry, but I am not convinced that you really appreciate feedback, since every time any of you Nextcloud devs read or hear something which is not to your liking you dismiss it.

The fact that you remove the production channel is a mockery to the community. Hereā€™s why:

I have complained for years that there wasnā€™t a way to lock in the release Iā€™m using and that you force people to upgrade when they want to use the updater.
Long story short, my suggestion resulted in the way how the production channel is currently handled. (You can read up on that on the issue tracker.) So as long as thereā€™s a minor update available within the major release that is installed, the updater will present you that minor update.

You do not allow people to choose the major release, nor do you allow to set a check mark to lock the current release. So the production channel was finally a useable workaround (after me urging for a solution for years). And now, you get rid of the only channel that makes any sense.
Thereā€™s not even a command line option to manually specify the update you want.

Your channel systen is a mess that you guys created, because you do not listen to people. I want to upgrade to a version I choose, not to something you force on people.
So I can either do what you want or I have to upgrade manually, which makes the updater useless.

So by telling people that the production channel (which was the only channel useful for people who did not want to upgrade on your timetable) you force yet again your idea of when to upgrade onto people. And we all know that even after .2 the releases are NOT stable.

You really must hate all people who do not want to shell out thousands of dollars/euros for an enterprise release and support contract.

11 Likes

@tessus they are hoping to be bought out by Microsoft so they are already applying itā€™s methods.

The version to update to should be configurable, a choice (with a recommendation). If you choose to update later you should still be able to pick that minor update (eg 16.0.6) if you have 16.0.5. It appears as if this isnā€™tā€™ available,if I am not mistaken.

1 Like

Let me try to reply to a few things here.

Yeap, Iā€™ll update that page to note that the production channel simply doesnā€™t exist anymore.

I meant major releases - what we did now was release 17.0.1 to stable, and that in an incremental roll-out. We did the same with 16, 15 and so on. That was actually what we described on the website as our plan for Production - which is why we remove production.

That was NEVER the plan. We only delayed Production EVEN MORE because we already delayed stable. But that was (and is) silly. If people want to wait until 0.2 or 0.3 or something else, thinking that somehow that number means it is some special kind of stable - sure, sure, go ahead. But weā€™re not recommending it, and via Production we seemed to do that, which was a mistake. The numbers donā€™t mean anything.

What means something is that a release has, say, 10.000 servers using it and no important problems being reported to us. THEN we say, ok, this can go to stable. Because the benefit of the new features is bigger than the risk of issues for most users. Of course, if you as a special user think you have a different balance between risk and benefit of new features, you can make your own schedule by updating manually.

It is entirely possible that some day, we release a 18.0.1, a 18.0.2 and a 18.0.3 and still we have major issues - and we would not put it on Stable. The numbers are meaningless. What matters is the actual level of reliability, and that is something the release team can decide.

We do, in every release announcement I tell people they can expect the updater to give them the release after some time, it can take a month or more. And you can switch to the Beta channel to get it right away. Also, we say that same thing in the text that explains the stable release.

If it makes you feel better, do it. But as I said above - numbers are meaningless. A forum check is indeed smart but also not 100% perfect, as typically the issues found after release are corner cases that hit, for example, ā€œNGINX users using SQLite on Ubuntu 16.04 with a broken imagick php moduleā€. The smarter thing to do would be to actually test our release candidate, and if it works for you, you can upgrade right after the release and get the new improvements and features 2-3 months earlier than by waiting. Plus you help our community and our users by helping the testing process. A win-win I would say, unless you of course donā€™t want to help out your fellow users.

Yeah, we could have done better there. Sorry!

I get it isnā€™t what you want. But it is actually exactly what WE wanted to provide: a recommendation. And make it easy to follow our recommendation. Doing anything other than what we recommend is extra work. That might not make it perfect for you, but we have ~300.000 system administrators out there and many are not as experienced and donā€™t have as much time to maintain Nextcloud as you do. Having them choose a ā€˜stable channelā€™ like 15 and stick with that means they get a less good experience than they could have. That is bad for them, and bad for Nextcloud. So if they want it, they have to do a bit of extra work.

That statement is ridiculous in multiple ways. First, we try to do whatā€™s best for the majority of our users, and sometimes that isnā€™t perfect for some individuals. Second, people canā€™t pay for what you want - we simply donā€™t offer a update channel that updates slower or stays on a stable release for customers either. So it has nothing to do with money. We push and heckle our customers to upgrade as much, if not more, as with other users, for the exact same reason: using an older Nextcloud version means you use software with not only less features, but also less users, which means it gets less testing and less scrutiny which makes it end up a worse product overall. We monitor what version our customers use, and reach out to them if it is old. We tell them in advance of releases and any security issues. We provide special builds when there are issues and so on. A customer pays for a better experience and access to our engineers, not for an older version of Nextcloud unless that is explicitly what they want and even then - they donā€™t have an updater that precisely gives them what they want as every customer wants and needs something else (plus, wants and actual needs can also diverge).

You have that - upgrade on your own choice and your own schedule. Yes, maybe it would nice if it was possible to tell the updater command line tool exactly what version to update too, eg to only update within the same current stable version. It might make some users, including you and tessus happier, I suppose. It would also make it easier to not follow our recommendation and get a less good Nextcloud experience so we are not motivated to do the work for that. We try to spend our time to make Nextcloud better for most users. Please remember that you are 1 or 2 out of 300.000 users, we have to choose what helps most of them. Not a few. But I would think that a pull request would get reviewed and, if of good quality, probably accepted, so if you really care about this, you can make it happen.

5 Likes

@tessus You can do this easily on the Docker version. If you set the image tag to i.e. ā€œ16-apacheā€ then it will remain on the latest image of version 16.

Iā€™m doing this now myself because Iā€™ve always considered it a good practice to wait for the first patch after a major release before I upgrade, and this was my rationale for using the production channel. I do this with everything else, Windows, iOS, etcā€¦ We all know there has to be some big controversy with every ā€œdot-ohā€ iOS releaseā€¦ so to me it makes sense to do the same with Nextcloud.

All it takes to upgrade then is to go into your Docker-compose file, change the image to 17-apache, docker-compose pull, docker-compose up -d, and itā€™s upgraded in seconds.

I find this method to be less risky during upgrades too because thereā€™s basically no risk of any sort of version incompatibility or dependency issue since everything is provided in a pre-tested image.

The problem is that your recommendations are really bad. Havenā€™t you read the issue tracker and the forum. There are a countless people who follow your recommendation and end up with a broken system. The idea of an upgrade system that only follows your notion of when and how to upgrade would only work, if the dev process, QA pipeline, and release management was flawless. In case you havenā€™t noticed, it isnā€™t.
You also dismissed the part that something was finally implemented after years of complaining. And now youā€™ve removed that again. Great job. Very considerate,

That is what you think. But it is clearly wrong. You do what you think is best for the users, but you are dead wrong. Otherwise you wouldnā€™t have left that option for enterprise users. My idea was finally understood by devs and now you made it a paid feature.

As mentioned earlier with the result that they break their system. But hey, they can pay you to solve their issues. Otherwise they can go back and use a backup. Seriously, donā€™t you read the upgrade issues on github and this forum?

The only thing that people want is to be in control. You take that away, unless they go the manual route, which is more error prone. The only thing you do is make it harder for people - thatā€™s all.

Less features is better than half-baked ones. You still ignore core issues and problems that have been bugging people for years. Example: Do you have a statistic on how many people are using your admin group feature? Except maybe when users donā€™t intersect. Everybody I know (or any sane admin) finds this ridiculous and dangerous. Anyway, I gave up explaining why. I just donā€™t use it, like everybody I know. But Iā€™m getting off-topic, but I had to reply to this ā€œmore features are betterā€ argument.

Wrong again. EVERYBODY who likes to be in control of their own environment, will be happier.

I think ending up with a broken system is probably worse.

1 Like

@KarlF12 thanks, this might work and seems like a good workaround. Unfortunately I currently canā€™t use Docker on that server for reasons I donā€™t want to go into right now.
Maybe some dayā€¦

Well, you still need to upgrade the db schema. Also, I would have to have a look at that image first. I donā€™t trust Nextcloudā€™s design decisions. As long as they donā€™t use a database in the image which requires to use persistent storage, it should be fine though.
But if I were to go that route, I probably would build my own image.

@tessus It doesnā€™t use an internal database. You run the MariaDB Docker image alongside it, and the schema upgrade is done automatically when you bring up a new version of the Nextcloud image. Very easy to work with. Redis is also run in a separate container.

Yep, I would have hoped that it was setup this way. However, running a database system within a container may or may not be the best idea. It always depends. But thatā€™s off-topic. I will keep Docker in mind next time I upgrade the server. Thanks again.

Compared to previous owncloud updates, the updates are much smoother in Nextcloud. When I compare the problems to back then, it is much less and the user base is probably larger. Your post leaves the impression that it is impossible to upgrade without problems. There are probably environments and configurations where it is more critical, and for these users it is not a nice experience.

I am not sure if this production channel solves the problem. Iā€™d first prefer an updater where you can chose to either update the current branch or upgrade to the next version.

More and earlier testing would help to spot and fix issues earlier. I have a second test setup with a similar configuration, where I test a new version early with all the apps I use. That way, I am sure that there are few problems when I move my productive system. Not everybody can and wants to do that. But on the other hand, you canā€™t expect Nextcloud to check the upgrade procedure on each webhosting provider and their specific configuration.

Great. There are a few people who claim to know what everybody wants but it seems there are too few that actually do what everybody wants :wink:

Well, I am not happy either with all decisions from Nextcloud, Iā€™d like them to focus more on my needsā€¦

2 Likes

Yes, it did.

At 16.0.5 the production channel showed 17.0.0 as update, but when 16.0.6 came out it showed 16.0.6, so I could have used the updater to update within the major release. With the channel gone, I will have to upgrade to 17 and canā€™t use the updater to update to 16.0.6.

Just came to say that this is yet another sad theater of users feeling entitled and aggressive and demanding towards an open source project that you should be thankful to be able to use for free when most likely (might be wrong) you donā€™t contribute in any meaningful way.

@tessus just fork it if you donā€™t like it. These people are running a company and have to pay salaries and make their decisions in good faith. Why do you think they have to listen to you above everything else? If you think you are helpingā€¦ you are not at all. We need to be thankful and use positive constructive criticism, not unfounded aggressive claims.

@jospoortvliet I admire you for keeping cool and putting up with this everyday. My advice is to ignore the troll. NC is not perfect but you are doing great.

9 Likes