How to deal with blocking lack of progress in Nextcloud repos

Hi,

First of all, let me be very clear that this post is not a rant. I am very well aware of the fact that sometimes maintainers simply donā€™t have time to look into all issues and PRs, so that is not what this is about.

This post is only about how app developers should deal with situations where there is a lack of progress on the relevant Nextcloud repositories, to the extent that it is more or less blocking further progress for the app developer (be it due to some bug, needed feature, dependency or what not).

One example is this PR in the user_saml app: Bump firebase/php-jwt from 4.0.0 to 6.3.1 in /3rdparty by dependabot[bot] Ā· Pull Request #667 Ā· nextcloud/user_saml Ā· GitHub - It raises the version of the php-jwt library from 4.0 to 6.3, which is a very good idea. Worth noting is that it supersedes a previous PR on the very same topic, namely Bump firebase/php-jwt from 4.0.0 to 6.3.0 in /3rdparty by dependabot[bot] Ā· Pull Request #635 Ā· nextcloud/user_saml Ā· GitHub .

A few things worth noting:

  • The current/old version of php-jwt in the app is 4.0. This version is from mid 2016, which is extremely old (7.5 years). Since then, there has been numerous important updates and fixes to the library.
  • The superseded PR was opened in july 2022, so even though the current PR is just about 1.5 months old, the matter of getting the php-jwt library up to date is been ongoing for six months.

The reasons why this (example) PR not being merged is a problem are mainly the following:

  • When this app (and also another one, in which the very same problem exists, use of a very very old php-jwt version) loads, it populates the namespace for php-jwt with this old 4.0 version.
  • This makes it so that when another app loads, which in it has its own newer version of php-jwt, it doesnā€™t load the newer version of php-jwt.
  • This results in the other app using the old and wrong version of php-jwt, which has a different API, which means that the other app just breaks.
  • As we all know, thereā€™s just one instance of the same namespace in PHP, so thereā€™s no way to work around this nicely. I already tried changing the order in which the apps are listed, but it didnā€™t matter (but even if it did make the newer php-jwt version load instead of the old one, that would instead break the apps wanting the old version of the library, so meh).

So in summary; Because this user_saml app (and at least one other, namely the onlyoffice one) uses a 7.5 years old library, other applications are unable to work. For this reason we either need to work around the namespace issue, or simply get these apps not use insanely old library versions. Of course the best solution would be that all apps stay up to date with current libraries.

Now, the question becomes; How should app developers deal with this in practice? Surely the main ingredient is to have some patience, we canā€™t expect the Nextcloud app maintainers to review and finally merge PRs straight away. But at some point we either have to do some ugly hacky workarounds, or keep nagging the maintainers to get their attention and reviews. However, the latter is not very fun for either part, and nor does it help very much (generally speaking it just increases maintainer fatigue).

What are other peopleā€™s views on this? What are the Nextcloud maintainers view on this (perhaps @Daphne knows)? How would they want us to act in situations like this?

Iā€™d like to point out that the above is just one single example. There are plenty of blocking issues and PRs. Some of them get attention after a while of bumping, some others donā€™t. Itā€™s just a general problem :slight_smile:

1 Like

Oefffff I have a lot of opinions and perspectives on this.

Let me start with that I recognise what you describe.

Iā€™ll block some time in my agenda for tomorrow to write an elaborate reply. :heart:

2 Likes

Just one idea (to have the link here: Christoph Wurst wrote once a few words on this topic: Escaping the dependency hell

Maybe you can use this to allow for individual app dependencies?

2 Likes

Thanks @christianlupus, I didnā€™t know about Mozart.

What I did in the other app that uses/needs a current version of php-jwt but that is broken because of the 7.5 year old php-jwt in the SAML and the OnlyOffice apps was exactly the same thing that Mozart does, I made a couple of sed substitutions that prefixed the other appā€™s own current copy of php-jwt with an additional namespace component, so it has itā€™s own namespace for that current version, thereby ignoring the other old versions of php-jwt.

I consider this a hacky workaround, itā€™s really not pretty and might also break if one isnā€™t careful.

Using Mozart is exactly what you need to do.
Iā€™m using PHP JWT 6.3 since October in one of my apps and there is no problem or conflict what so ever.
Donā€™t be scared by the line count, if you ignore the composer.lock files itā€™s an easy PR.

JWT is my only non-dev dependency, so you can copy my composer.* files to reuse them (but use the master versions to profit from updates and follow ups)

4 Likes

The important bit is this here:

It basically installs JWT into the namespace of MY APP.

And then Iā€™m using it as OCA\External\Vendor\Firebase\JWT\JWT

3 Likes

Thanks for the example, nice to see it in action :slight_smile: Since thereā€™s currently no other way to work around the example problem, I will probably change the two sed substitutions (which do the same thing as Mozart, just in a more manual way) with Mozart. I donā€™t want this to become a ā€œsolutionā€ or answer to the thread as a whole though, this was just an example.

1 Like

It will have to be the solution I fear. For each individual problem there is an individual solution. For PHP library conflicts this is it. There might always be a conflict with a dependency in one way or the other way. Could also be you releasing your app now with 6.3 and SAML at some point shipped 7.0 and it breaks your app again or SAML breaks because 6.3 is used. So using Mozart to solve this is the only way to fix it in all directions.

Of course we should also use Mozart for the dependencies in the SAML app and OnlyOffice, so they can also be updated without breaking each other, but until then you can solve it already for your app.

2 Likes

Sure, for the example issue, itā€™s a hack but the only viable workaround to the problem. I appreciate your input! I just meant that the main question In this thread is not about that, but rather:

This post is only about how app developers should deal with situations where there is a lack of progress on the relevant Nextcloud repositories, to the extent that it is more or less blocking further progress for the app developer (be it due to some bug, needed feature, dependency or what not).

In other words, where there isnā€™t a workaround available :slight_smile:

so to continue the conversation on your core question, ā€˜how should app developers deal with situations like these?ā€™ and ā€˜how would Nextcloud maintainers want app developers to act in situations like thisā€™, I think what @nickvergessen tried to propose from his perspective as a maintainer is that it could make sense to raise your blocking questions to this forum or the community developer chat. We can then talk about the possibilities. He wrote for each different problem there is an individual solution, I didnā€™t thought of it like this yet as I was more thinking of a generic way to approach this but what he wrote made me think, as for some problems we might be aware of workarounds, other problems we can maybe solve structurally, etc.

1 Like

I think it would be doable to ping about certain issues or PRs here in the forum, but it would feel like one is trying to get past others in a queue or similar. It would feel like it would annoy maintainers. So it would have to be done when really needed, if itā€™s an acceptable method of gaining traction on more important issues/PRs.

Hi,
I have an idea for dependencies not getting updated, but this would require some commitment from the company (or other people willing to put in some work):

We want to figure out what apps donā€™t get dependency updates, so we just search for them.
A basic query I made returns all PRs by dependabot created before December this year that are still open: Search Ā· org:nextcloud created:<2022-12-01 author:app/dependabot state:open Ā· GitHub

Of course monitoring this way isnā€™t fun, but if we set it to a greater time period like 3 months and automate it a little, then we could have some list/queue that shows which dependency updates are really needed right now. Then people could just look at that list and work of the PRs if possible.
It wonā€™t be as easy as I just described it, but I think itā€™s a good start to get an overview which dependency updates have been missed for a while.

It would be better to not base it on the date the PR was opened but one some ā€˜criticalnessā€™ score. That would probably get complex very fast, but for example monitoring major version bumps could already help this.

2 Likes

Hi all,

first of all thank you @rawtaz for raising this concern and triggering a discussion about how to generally deal with such scenarios to resolve a blocking situation for an app developer or generally speaking any contributor that gets stuck because they rely on the action of somebody else or more specifically a maintainer of an app or server-core, clients or common Nextcloud build/maintained libraries.

As you mentioned besides pinging people on Github in the respective PR or issue I would also say raising it in the forum or on the Nextcloud Talk room at https://cloud.nextcloud.com/call/xs25tz5y are an appropriate way to help with raising the attention on such an aspect. I usually have done so when contributing to the Android apps of/around Nextcloud since besides raising the attention it usually comes with the need of some discussion or exchange of ideas in a more real-time way than async comments on Github (while best case the result is then posted to Github). While this is just my point of view and others might have a different opinion on how they would like such things to be raised in the best possible way.

I agree this has the side-effect that it might feel like cutting the queue while I believe in reality there is no queue I am afraid. It is more about taking action by getting into a conversation that will turn then into actual action as-in implementing things or making decisions, clarifying missing steps where one could help out and hence unblocking things.

Iā€™d be very interested in further, other peopleā€™s opinions here.

Specifically looking at the SAML app case, this one is a bit special in a way. Due to its nature it canā€™t be properly tested with automated tests and has a lot of code paths offering various use cases / behaviors which leads to a larger effort needed to test the changes. From my point of view the best approach is to discuss about potential side-effects and needed testing and offer to help with that. So for SAML as well as any other app I would also ask the question ā€œWhat can I do to support getting the following change merged?ā€ sometimes this might mean ā€œWhat do you need me to test, so this is save to merge from a maintainerā€™s perspective?ā€
Also in this case I believe the proposed solution with Mozart is the way to go, trying to maintain a specific version of a library across apps leads to a dependency between apps. The goal in my opinion should always be to keep dependencies at a minimum and with respect to apps to enable developers to be able to implemented in an isolated way, not depending on other apps libs/implementation etc. because else it wonā€™t be manageable in the long-run.

Looking forward to further comments. :heart:

4 Likes