Boosting community app development

[te-online]

This is fantastic news, @Daphne Welcome!

I’ve been waiting for an initiative like this for a long time, so I just want to add a few words.

I maintain two apps, TimeManager and Link Editor. The biggest issue for me is lacking documentation on almost everything.

My tooling / setup and struggles it addresses

This is what has made life easier for me after a lot of trial & error, in no particular order:

1. Releases — I wrote a webhook server to publish releases to the appstore that I host on Uberspace to automate the process of publishing releases through a Github webhook. It has the added benefit that my apps are an exact copy of what can be found in my repositories, since no manual steps are involved in the publish process. Before I had this script, it took me a lot of time to publish releases properly, without missing any steps. I think it should be a no-brainer to add publishing tools to the Nextcloud developer ecosystem. Publishing manually is a recipe for disaster, especially for less experienced developers.
2. Development environment — I use the official nextcloud docker container for development after years of barely getting by with MAMP. How to set up a proper development environment on Windows, Mac and Linux should be in the developer documentation, too. Especially since it needs some hand-holding to get up-and-running properly. I’d be more than happy if the documentation and docker-compose file that I use for my apps could be re-used in Nextcloud’s docs.
3. End-to-end tests — The docker-compose setup I also use to run cypress end-to-end tests in Gitlab CI. It’s not at all perfect, but gives me more confidence when testing with new Nextcloud releases. There seems to be no documentation on end-to-end tests, even though – in my opinion – they are much more useful for UI-heavy Nextcloud apps than unit tests. We need documentation on how to write end-to-end tests and how to set up CI for them.
4. PHP stubs and editor Intellisense — I’ve started using Intelephense for Visual Studio Code and included my local Nextcloud git, as well as some stubs by @ChristophWurst, that can be found in composer, in its config. See this post. We urgently need documentation on how to set up most used editors for development. I’d never have guessed some deprecations or specific call signatures that I now get to see with this setup.
5. PHP API — This brings me to my last point: There appears to be no public roadmap for PHP APIs. I can never be sure what the latest and greatest way of doing things is (like loading assets, loading the current user object, routing, query building etc.), because things change all the time (with most of the legacy stuff staying around for a while, to be fair). The way I adapted to this, is to browse the core apps every once in a while to figure out what they changed. Also I subscribe to the Github issues on the server repository that mention breaking changes in each release. Finally, I’ve added a step to my CI tests that dump the nextcloud.log file to see if I missed any deprecation warnings or obscure errors during development. To fix this, we urgently need good documentation for all PHP APIs and some info on when they’re no longer recommended to be used and what to use instead and how to use it.
6. Missing basics — Part of my problem is that I don’t understand Symfony very well, which I think most Nextcloud APIs are based on? The developer documentation should link to the Symfony documentation or any other 3rd-party library used where necessary and explain common concepts of that library (like naming, mappers, available super classes etc).

Nextcloud, the company, and developer relations

I’d be happy to help with this effort, but I also see Nextcloud as a company in (deep) debt here, because only they have the resources to hire someone like you to take a first shot at this. And since the whole ecosystem profits from good apps, I think there should be incentive to invest much more into developer relations, but I can remember that we mentioned this avenue to Frank a few years ago and he didn’t seem impressed … so there’s that

Let’s look around for good examples

Lastly, I want to suggest to not re-invent the wheel. We should take a look at other projects that have a successful (and diverse in terms of levels, from beginner to expert) app developer community. I’m not saying it’s a prime example, but take a look at how good the WordPress documentation has gotten. It has examples on almost all methods, hooks, actions etc. It also has guides now to cover basic concepts and those guides are exhaustive.

I know the Nextcloud developer documentation has guides too, and kudos to the person who wrote them, but they’re really only scratching the surface if we’re being honest.

---

This got much longer than I intended – hope it’s useful regardless.

– Thomas

[Larry_Boyd]

Putting it through nextcloud’s mechanism for software distribution is just too much overhead to deal with. I’m happy to work on my projects and even share via public git servers (like gitlab), but that’s as far as I go.

As far as I’m concerned, if you’d like to publish my work on your system, then provide a form where I can submit a link to a project on GIT (and nothing else), and you can get to scan and parse the project and handle publishing it.

[SysKeeper]

One thing I noticed: Why do I have to use a web form (Generate app - App Store - Nextcloud) to generate a skeleton app to start developing? (You can’t even select NC24 as a target). I guess it would make sense (especially after reading the comments above) to offer some kind of CLI tools to: create a skeleton app, register an in the app store, publish an app, etc.
Then suddenly there’s GitHub - ChristophWurst/krankerl: A CLI helper to manage, package and publish Nextcloud apps, which looks like it would solve some of the publishing problems mentioned above, but (much like the docker dev container) it’s quite inofficial and not mentioned in the documentation. There are a few good enhancement requests in the repo there:
Add commands for certificate management
Have a GitHub action

I guess this would be really appreciated to have.

Btw, the web form for the skeleton app is not correctly translated …

[z4k]

@juliushaertl

It would definitely be interesting to learn what caused problems there and what the simplified approach is you’ve taken. I’d always be happy to improve the documentation on my setup or adjust if we can make things easier.

@juliushaertl : Yes, I have to take time to explain my problems to use your repo
But, yes, I imagine you would be happy if we give you feedbacks.

@all : What do you think if we create a new subject on docker, in general not only on Julius’, to improve the development process ?

@TessyPowder

This does exist. There are Nextcloud Talk Chatrooms on https://cloud.nextcloud.com. You need somebody with the needed privileges to invite you to be able to join. I remeber that i was invited by Jan-Christoph Borchardt quite a while ago. It should not be hard to get an invite, you probably just need to ask.

@TessyPowder : I am in this chatrooms and, I’m sorry, but the Talk app is not like a “Discord-like” or “Slack-like”. You cannot create several channels (you have to invite the person), you cannot write markdown, you cannot create threads (bonus) and so on.
One more, with Talk you joign one group and not a server with channels to exchange.

This app is like Skype, Messenger (Facebook) or MSN Messenger (for old ).

I think a Mattermost, RocketChat or a Discord-like it’s better than Talk app.
But, I like Talk app ! It’s just an app to exchange with one person or a group limited to 5 or 10 people.
By the way, I think it’s a good idea to have an app on desktop

@TessyPowder

The solution I found for me was https://www.gitpod.io . It lets you spin up dev environments in “the cloud” with full root access and Visual Studio Code in mere seconds. If you add a gitpod configuration file to your repository, you can make it automatically start a nextcloud server.

I’m sorry, I don’t understand why Gitpod it helped you
For me, Gitpod it’s just a VScode through the web, not more. No ?

@te-online : I agree with you on everything you said

@SysKeeper : Oooh ! Very interesting ! I don’t know, this project by Christoph Wurst
But, yes, I think it’s very cool to have a command line to generate a skeleton like Laravel or any other tool. In general, I agree that we need a command line to :

• Generate a project
• Generate migration files
• Generate seed files
• Publish our app
• etc.

Like artisan of Laravel or Symfony

[TessyPowder]

@z4k

I am in this chatrooms and, I’m sorry, but the Talk app is not like a “Discord-like” or “Slack-like”. You cannot create several channels (you have to invite the person), you cannot write markdown, you cannot create threads (bonus) and so on.
One more, with Talk you joign one group and not a server with channels to exchange.

Yes, Talk is not like Discord, but I didn’t write about them in my answer, because they are like …

@z4k

Discord or equivalent (Mattermost, etc.)

but because they are used for the …

@z4k

exchange between the Nextcloud developers and / or its community dev

like you asked for.
I did not know, that you were already in these Talk rooms.

I think that the Talk rooms fulfill every function, that a discord server would, in terms of actual chatting, not function in terms of chat software features.
I think it is only logical for the nextcloud gmbh to use their own chat tool for their company and community chats.

I would rather propose making their Talk chats easier to use by splitting them up more (there are already a few, but i don’t think there is a list, so I only of the ones I am in) and creating a Community Hub Channel with a list and a way to ask/automated way for joining the specific community channels.

@z4k

I’m sorry, I don’t understand why Gitpod it helped you
For me, Gitpod it’s just a VScode through the web, not more. No ?

Well kind of, but if you screw up your dev environment you can spawn a new one and if you configured it to automatically set up, you will basically never have to debug your dev environment.

Also if you want to work on multiple branches concurrently that require changes to the dev environment you can just spawn another workspace and use both concurrently.

I use this often to test an app on multiple nextcloud versions, I do not have to spin up multiple nextcloud servers on my local machine (annoying port conflicts, clutters your storage, takes a while to install multiple times, download times limited to your internet speed), but just spawn a throwaway workspace and have a quick script, that changes the nextcloud version. After that i can just close the mess and return to a clean workspace.

It was basically the solution to dependency hell for me, which I find the most annoying about dev environments, especially if you work on mutliple things concurrently.

If you don’t run into this problem you of course don’t need a solution for it

At least I never want to have a nginx nextcloud dev server on my laptop again.

You can of course recreate this on a local machine by using multiple dockers, but that exactly is what gitpod does, but more quickly and easier.

There would also be benefits to Gitpod, not for personal projects, but for the big community/official nextcloud projects: It makes it incredibly easy for first time contributers to get started. No need to set anything up, just change something in the code and see the result. I think that is quite powerful.

(I would never recommend gitpod if it was closed source btw)

[z4k]

@TessyPowder

like you asked for.

Hmmm… Yes and No ^^
That’s not what I meant. I meant, it’s better to have a platform, not only exchange message, but help each other with differents channels as I quoted here :

@z4k

In the case of Nextcloud, we could have for example, a channel on :

• #dev-frontend : To exchange/help each other on the front-end (JS, Reat, Vue, CSS and so on.)
• #dev-backend : Like for the front-end but on the PHP
• #dev-ci : To exchange on our different CIs to have a quality app
• #ux : To exchange on the UX-Design of Nextcloud and its apps

The main of which is to be able to share our code (with Markdown), share our ideas on UX or UI, help each other if we have a problem in our code and so on. All this, with a better flow than Talk app.

@TessyPowder

I think that the Talk rooms fulfill every function, that a discord server would, in terms of actual chatting, not function in terms of chat software features.

I don’t agree with you. It’s very complicated to share my code if the markdown it’s not present.
One more, exchange on UI, UX, Front-End, Back-End, Database, and so on. On one group… It’s a bad idea for fluidity.
Someone who is UX or UI Designer would not talk in the channel of Talk. Because, either they will be drowned out by the messages or because it’s a group for developers only.
One more, as a Front-End or Back-End developers, she/he would also feel drowned by the messages.

@TessyPowder

I think it is only logical for the nextcloud gmbh to use their own chat tool for their company and community chats.

Maybe… Or maybe not. Currently, Talk app doesn’t meet the requirements I mentioned.
After all, all depends on its orientation, either as a messaging tool like Skype, or like Slack where you can join different channels.
One more, in Talk you can invite one person on a group. If we want to stay on Talk, this means adding people one by one in each group.
If you add a UX or UI designer only in a Talk UI/UX group. You exclude them from other groups and therefore from other exchanges

But beware, this is just my opinion on the matter

@TessyPowder

I would rather propose making their Talk chats easier to use by splitting them up more (there are already a few, but i don’t think there is a list, so I only of the ones I am in) and creating a Community Hub Channel with a list and a way to ask/automated way for joining the specific community channels.

In this sense, I agree. But, it’s very time-consuming and not user friendly.
Again, it depends on the orientation of the app.

@TessyPowder

Well kind of, but if you screw up your dev environment you can spawn a new one and if you configured it to automatically set up, you will basically never have to debug your dev environment.

Also if you want to work on multiple branches concurrently that require changes to the dev environment you can just spawn another workspace and use both concurrently.

I use this often to test an app on multiple nextcloud versions, I do not have to spin up multiple nextcloud servers on my local machine (annoying port conflicts, clutters your storage, takes a while to install multiple times, download times limited to your internet speed), but just spawn a throwaway workspace and have a quick script, that changes the nextcloud version. After that i can just close the mess and return to a clean workspace.

It was basically the solution to dependency hell for me, which I find the most annoying about dev environments, especially if you work on mutliple things concurrently.

If you don’t run into this problem you of course don’t need a solution for it

At least I never want to have a nginx nextcloud dev server on my laptop again.

You can of course recreate this on a local machine by using multiple dockers, but that exactly is what gitpod does, but more quickly and easier.

There would also be benefits to Gitpod, not for personal projects, but for the big community/official nextcloud projects: It makes it incredibly easy for first time contributers to get started. No need to set anything up, just change something in the code and see the result. I think that is quite powerful.

(I would never recommend gitpod if it was closed source btw)

Oooh I see ! It’s very surprising !

The idea is not bad

But, I think it’s heavy to use to create an app quickly.
Today, I use 2 dockers : One on the release 22 and the other on version 23.

I think a small docker based solution is a good idea and if we want to go further, we can propose Gitpod ?

I read prerequisites to install Gitpod Self-Hosted, and I feel like it’s a lot to handle, no ?

I haven’t used Kubernetes yet ^^’

Self-Hosted - Gitpod Docs

[TessyPowder]

@z4k

On one group… It’s a bad idea for fluidity.

It is not one group, i think you are in just one and don’t know of the other rooms. I am for example also in the VueJS Chat, the News App Chat and Frontend/Design Chat, all in different rooms on the same nextcloud instance.
Which just further illustrates the need for a way to discover these groups. If you can only know of them if you are a member, that is quite impractical for new developers.

@z4k

I think a small docker based solution is a good idea and if we want to go further, we can propose Gitpod ?

Absolutely, Nextcloud development of course should be easy, regardless of if you want to use Gitpod.

@z4k

I read prerequisites to install Gitpod Self-Hosted, and I feel like it’s a lot to handle, no ?

I haven’t used Kubernetes yet ^^’

Until a few months ago, it was basically impossible to understand how to install it. Now they have good documentation.
I have not tried for myself yet, because it is not cost efficient for me to self host gitpod, as the limits of the free hosted version are pretty generous.

@z4k

But, I think it’s heavy to use to create an app quickly.

If you want self host: definitely, for beginners I meant the hosted version.

[PackElend]

@z4k

That’s not what I meant. I meant, it’s better to have a platform, not only exchange message

agree here. It has to be more than simple text.
Things have to be easy to use. A continuous surge of simple text makes conversation quite challenging. In particular, if you begin to share lings URLs, quotes not talking about code yet.
You should give people a chance to leave for a couple of days but to come back on an open conversation.
(that is how I would get attracted to use it )

[carlschwan]

This was something I want to implement for a while now so that third parties desktop/mobile apps can be listed. A bit like Clients | Matrix.org

Unfortunately I didn’t manage yet to find the time and there are more urgent things to update on the app store

[just]

Here are those issues filed for the appstore on github

• add “Mobile client” category"
• add “Desktop” app category

@scubajeff

How about a separate section dedicated to mobile apps in the app store?

@carlschwan

This was something I want to implement for a while now so that third parties desktop/mobile apps can be listed. A bit like Clients | Matrix.org

[z4k]

@TessyPowder

It is not one group, i think you are in just one and don’t know of the other rooms. I am for example also in the VueJS Chat, the News App Chat and Frontend/Design Chat, all in different rooms on the same nextcloud instance.
Which just further illustrates the need for a way to discover these groups. If you can only know of them if you are a member, that is quite impractical for new developers.

So, it confirms that Talk is not the right tool. Because, I was well added to the group “Community Chat” but not to the others. And yet, I need to be added to these groups when I have a problem or suggestion on Vue.js or Design.

@TessyPowder

Absolutely, Nextcloud development of course should be easy, regardless of if you want to use Gitpod.

Yes, I agree !!!

What do you think of docker-compose by Julius : GitHub - juliushaertl/nextcloud-docker-dev: Nextcloud development environment using docker-compose ?

@TessyPowder

Until a few months ago, it was basically impossible to understand how to install it. Now they have good documentation.
I have not tried for myself yet, because it is not cost efficient for me to self host gitpod, as the limits of the free hosted version are pretty generous.

But, isn’t Kubernetes resource intensive ?

[SysKeeper]

@z4k

And yet, I need to be added to these groups when I have a problem or suggestion on Vue.js or Design

You can just search for “public” and see all publicly available rooms and join them

[szaimen]

Regarding Talk and multiple topics per discussion or whatever your are calling it, I made a proposal two years ago how I imagine how it could work in Talk. See Grouping of Chats · Issue #2448 · nextcloud/spreed · GitHub and Grouping of Chats · Issue #2448 · nextcloud/spreed · GitHub but open chats and being able to search for them already solves many use cases, I guess.

[TessyPowder]

@z4k

But, isn’t Kubernetes resource intensive ?

It is just a container orchestration system, it itself does not have much resource usage, but the containers you run on it can.
And running a big Gitpod Instance with many users certainly takes a lot of resources.

@z4k

What do you think of docker-compose by Julius : GitHub - juliushaertl/nextcloud-docker-dev: Nextcloud development environment using docker-compose ?

It seems convenient, but I have not tried it yet.

I replied to your other points in the branched out thread.

[z4k]

Oooh !
I tried but I couldn’t find it in the search bar

[carlschwan]

Search for frontend and the public Files channel could also interest you

[christianlupus]

Let me share my opinion on some of the topics I have read here.

In general, I consent that the documentation needs improvement. However, I think we need a broader style. Currently, we have a basic API reference and a more-or-less walkthrough for a minimalistic app. What I am missing:

• A general overview of the dev process for complete newbies (get started with web development in 10 minutes)
• Detailed architectural descriptions on the building blocks of the NC backend (how are the gears mesh )
• Dedicated chapters related to security-relevant topics: What are the notions? What is the motivation behind certain structures?
• A complete API reference with all classes/interfaces in \OCP well-defined and documented.
• There should be different walkthroughs for different levels from beginner to advanced also with different reading speeds and/or cross-links. Keep in mind to have devs from foreign languages but with architecture experience
• Give the app developers a space to share common APIs to simplify common work. Also, best practices should be defined before.

Adding more documentation might make restructuring the docs important. Currently, there is a category Digging Deeper that is sort of catch-all. This might need a break-apart.

---

One could consider creating e.g. screencasts and publishing them. This will attend to another audience then just plain text. Example form joomla with youtube video.

I know this is a hell of work to carry out good videos but this might be a community approach at best. But then, it needs backup by the core team to for example not make significant errors or document something in detail that is due to change soon.

---

One thing, I want to oppose here on the thread a bit is the request for more abstractions. Yes, the NC cure team could implement a general way to exchange data between the backend and frontend. Up to a certain level, this is a good idea. However, this should be the second step.

I have been developing with Joomla and there is a lot more of this automagical sh… going on. Either you know this and learn it by heart or you are lost. You can read other people’s code but you have a hard time getting your things to work. This is one reason, I abandoned my recent project and decided to do it in the NC core. The documentation and code of the NC are way cleaner. (This is my personal opinion. You can argue but just have a look at their kernel.)
It is true, you can get results faster. But once you leave the basic paths you are mainly lost. You have to do everything manually. There is no Symfony framework having your back and helping with e.g. class loading. It’s on you, mainly.

Long story short: I think the current structure is a good one. You can learn it. You can use it. Yes, you need documentation for it.
Now comes the second step: You could add some abstractions on top of that as an addon to simplify life with typical problems. But this should not render the basic skills second-level citizens. In contrast, this should be marked as such. Devs are responsible for their decision on what interface might be better suited.

---

There have been requests for Mattermost, Discord, etc here. I do not see fit for these. What is the benefit over the forum here? All of them provide support for markdown, threads, etc. What is the significant difference?
If there is, I might change my mind. I personally prefer Mattermost over discourse but this is just personal taste.

Adding another channel might split the community even further. ATM we have a Matrix channel, this forum, the GitHub discussions/issues, and the various chat rooms on the official server. Did I miss something? The more options we have the more valuable experienced developers need to spread.

I am even considering if the NC talk rooms are a good option for the public community. These are only reachable using a public link. This is not ideal for public community building. No notifications, no last-read state, etc.

After the initial post, I decided to go through some of the topics here on the forum and help out if I can. I see a small part of the responsibility for this situation also with us community members. There are many seeking advice and do not get it. Then, if these devs accumulate their personal knowledge, they do not share it. Voila, you have a vicious circle. I can only pledge: All devs keep a look on the new topics and try to help newcomers as best as you can.

---

Some things can be done centrally but need documentation before being implemented. For example, it took me quite some time to set up a DevOps system for automated tests to run on the dev machine with an arbitrary PHP version/DB system involved with exactly the same configuration during CI. It is a lengthy python script that might easily be adopted for general use.

It took me quite some time to get it working and I’d like to share it with other authors. But, it will require some setup and documentation on how to integrate it. Here is also the question(probably @Daphne knows who to address), how such contributions can be handled.

---

What I am missing most is access to the more/most experienced developers. I had quite some time questions related to the core and its behavior. In the end, I debugged the server instead of writing my own app. This is not time-efficient.

In another project (https://octoprint.org/) there is a monthly (?) FAQ over a live video stream with the main developer. It is sort of a Ask Me Anything session (you can provide questions before and these are addressed one by one by her. The session is live for Patreon supporters and the recording is published afterward for all interested. Tina was awarded for a very strong community binding.

Could we do something similar? Maybe even from the community, a few experienced devs are there for questions. These do not need all to be core dev members. But it would be nice if there was at least access to the core team. You could combine this with some sort of badge/tier that must be affordable. These are in fact your future senior devs!

---

Some other point that the NC company could provide is infrastructure. For example, for devs it can be interesting to extract some sort of telemetry data. For example: is it safe to drop support for NC 21 in my app without losing too many users? What apps are installed alongside the app?
Maybe there are dedicated questions the app devs might ask more specifically (How many recipes are registered per user avg/max?)

This could be extended to other services as well. Example: We are discussing an AI-based detection of strings that needs training data. It would be useful to let the users voluntarily upload data to train/enhance the classifier. But this is rather advanced.

---

One big time-eater is requesting the required information for bug issues. Sometimes it is hard for the users to collect the required information. An app to open new issues might help them.

---

Sorry for the very long post.

[christianlupus]

This is on the cloud.nextcloud.com instance? WHo did you ask to get access? I am just curious how to get access.

[Daphne]

Any Nextcloud employee can give you access I can add you! Just send me your mail address in a 1:1 message