Crowdsourced QA tool

đź”® Meta
1

Hi everyone,

there is an idea floating in my head since some months, that I want to ask for feedback from you forum people here.

Introduction: On the development side of the project we have a lot of automated testing available, started with unit tests, integration tests, acceptance tests some tests for the APIs, some tests for the UI (especially files), some static code analysis and all of this. We have the basics covered quite good and have decent results there.

The problem: On the other side there are a lot of special cases that are not covered by the above automated testing cases and that are also quite some work to implement. That’s the reason why we call for testing and have alphas, betas, RCs and then final releases for this project and not just release final packages. Currently those testing efforts are quite intransparent - especially the positive side of it. If something is broken quite fast the github issue tracker reflects that, but the stuff that works is not shown immediately.

The solution: My idea was to bring a little web page to life, which makes it easy for random persons to participate in this process with the lowest possible barrier. It is basically a collection of user stories like:

As a user I want to be able to move a folder “Photos” (incl. it’s content) into a folder “Personal” via drag and drop in the web UI.

Then you could go to a page that lists a lot of these user stories and you could give feedback if it works for you or not including an optional comment and adding links (for images/videos or github/forum discussions) to the test case.

No login required: This also does not require a login or any authentication, because it is basically for collection the basic information and should be as easy as possible to participate in. To avoid vandalism the results are just the average over all results (if 10 times the feedback was “works” and 1 time “does not work” it is likely to work).

Help needed: It also allows to show which tests are already done by other persons and which aren’t. That makes it easy for people that want to test some stuff to pick the pending tasks.

Versioned: Obviously the results are stored per version of Nextcloud or app.

It then would allow a feedback loop to the developers when the time is right for further polishing or if all the basic tests are done.

Give me feedback about this idea!

This should just be a quick introduction into what is in my head for quite some time and I want to get basic feedback if this would make sense at all and people would help to gather this information or if there is nobody that actually want to test.

Disclaimer: I will add more details to this in the coming days - I have collected already a lot more thoughts about this but don’t want to start bikeshedding about details right from the begining. This should just be a rough draft to get an idea if this is wanted or not. If the idea makes sense or is complete nonsense.

@tflidd @JasonBayton @Andy @TobiasKaminsky @Soko @Schmu @anon99252149 @juliushaertl @MariusBluem @rullzer I just picked you as the more active people here in the forums and may already have a quite good overview if this is needed or not. Maybe you could mention other people that are active so that they see this. Or you have already read about this and can link to other relevant forum posts. Thanks :slight_smile:

2

Hi @MorrisJobke,

From how I picture this in my mind, it sounds like a good idea and a good way to go.
I would use that option. Now there is the pressure to finally get my test server up and running (where I struggled quite a while already) :wink:
I will put my focus on that and will contribute more then - with tests and with giving feedback on your planned site. Looking forward to see it go online.

And thanks for putting me on the list, appreciate it :slight_smile:

1 Like
3

hey morris,

for me that sounds like a good idea, basically.
and good boy that i am i’m not screwing that with a “but”.

all i can say is: i would at least like to test this idea. if i’m gonna use it on a regular basis? i don’t know, honestly.

good luck & cheers
jimmy

1 Like
4

I really like the idea to have a list of qa test cases available to the public and the possibility to also engage contributors to help testing those.

One thing that might be a bit out of scope for now but it would of course be pretty awesome if users would not need to setup their own instance for testing. Maybe something like https://probo.ci/, which can spin up a testing instance for every pull request after the tests succeeded.

1 Like
5

So, just to avoid spammers I would us some kind of login. Maybe just auth against the forum or whatever?

Having users create/extend/test stories makes a lot of sense. Especially if we find things that are wrong. Mutliple people can easily test and maybe help debug.

As a nice bonus this can be used to extend the current existing automated tests to focus integration and acceptance tests on where they are most needed.

4 Likes
6

This is a good idea but I am not sure how much the benefit it.
Easy user stories can be tested with automated testing. But what about the more complex one, which might even require some pre-setup?
For instance on android:

  • enable auto-upload, take a picture and verify it gets uploaded (easy)
  • switch storage to external path, enable auto-upload, set option to move file to nc folder, make a really big video
  • enable auto upload, set “upload only on wifi”, make lots of photos, leave wifi range while upload is still in progress

Please do not get me wrong. I like the idea. And I am sure that there are some people who try these complex scenario, but that will be so few, that I am unsure if we can rely on it (especially on android where there is this big fragmentation)

3 Likes
7

I also think it is a good idea especially for the webUI while I agree with Tobias that it might not bring a large benefit to Android where we could still use it for tracking tested use cases.

1 Like
8

I would say, that it is still fine to define complex user stories. I had it quite often that people approach me with: I want to help and have 30 to 60 minutes, what could I do? And then pointing them to such steps would still be helpful and could be used to track what is tested and if at all.

That is a quite good point. :+1:

Exactly - the idea is to have the complex stuff covered and also some regression prevention (like making a ticket to an user story and keep track of how it is used).

9

@Rello :stuck_out_tongue_winking_eye:

1 Like
10

I’m all for this, though agree SSO with the forums, GH or likewise would be the way forward. Inundated with spam otherwise.

11

It is a good idea to give some more indications what should be tested. For a normal user, it is difficult to figure out what has changed and might be worth testing.

However, I generally test the new version out of curiosity, I want to see and test the new features, see if the performance has improved, if some new apps might be interesting. People won’t play guinea pigs for Nextcloud, especially if these are no features and repetitive stuff. For this case you should hire a test team or invest more in automated testing.

For the more community driven apps, this could be a very nice thing and people are probably more willing to help out.

Test machines are great, e.g. @JasonBayton’s instances help people to take a first look on Nextcloud, compare the behaviour of the system with their owns, they can test their clients and much more. So this is very useful. But, this is a pretty default setup (as all test setups will somehow be “default”) but many problems are due to a specific environment. Take for example the theming issues in the snap images, which also showed up on many reverse-proxy systems but on “standard” setups they weren’t visible.

Is there a way to test different systems at the same time (somehow replicate my actions)? You would still need to provide a dozens different setups to test.

2 Likes
12

First of all I like that Idea.
But do you want to give a easy way to collect Issues and get feedback how many people get them. Or do you want to collect Feature Ideas too?

I think it would be usefull to be alble as a developer to place files to be run. For example if you have a funktion creating a systemcall. Providing the system on the platform to see if it works on all different kinds of platforms and setups.

Or to provide a spezial client build wich is maby not a full RC canidate but just a build of some branch.

And theire should be filters to select if the thing can be tested as a user or if you need to be admin of the instance. Futher more if it can be tested on a productiv System or if you need a testing System for it.

13

Thanks @tflidd for the feedback. Just to highlight the first thing: this is not about letting people play guinea pig. This is about organising ourselves in a collective way. It’s the same why we have a platform like transifex, that helps to coordinate people, that want to spend some minutes to contribute to a project in a useful way.

Also the points about the setup problems is correct. On the other side this should also reflect how Nextcloud was planned to be used and how people use it. It could help us developers as well to understand what critical user stories are and include them into the big picture and not loose track of them.

Thanks for this feedback, @tflidd

@tacruc This is not about feature ideas, but about if features that were implemented are still working like they were drafted in the beginning.

It is also not about the fact to let people create instances, but to give peoples an overview what steps (like uploading files) need to be checked and verified if they still work on a users instance (or demo instance or one of the instances of @JasonBayton).

This also will not provide special builds of the clients - it is more about if pre-releases and releases still provide the promised functionality.

This is a really good point. I will see if this could be integrated somehow, but will first focus on completely isolated test instances to not harm anything.

1 Like
14

When I read the idea I was instantly implying that we need to have our own test servers.
Actually I see it in everybody’s interest to test the new release on one’s own specific environment. Still - and I totally understand that - not everybody will or can setup his own test server.

However, I see it as most helpful for the devs when people, who want to contribute, have their own tests environments.

While tests on own servers will probably close out many community members and combination of both could be useful then. Whoever can, tests on their own servers and for all the rest we use @JasonBayton 's servers :wink: (in my opinion however there could just be some more apps enabled by default on your test servers :wink: )

1 Like
15

How long does it take you to enable them? :stuck_out_tongue:
Seriously, both you and @tflidd have asked - what should be enabled?

16

Do you want to give access to 3rd-party apps (at least those on the Store) and developers as well?

17

@JasonBayton It’s not an issue :slight_smile: They are enabled quickly, indeed. I was just thinking that a Nextcloud demo server with most apps enabled gives a better impression on Nextcloud’s possibilities and is nearer to the setups that users run. And yes, sure: it will always differ more or less from one’s own setup.

On my test server I just enabled every app that doesn’t require to install additional services (like SOLR for example).

The apps

  • notes
  • markdown editor
  • contacts
  • music/ audio player
  • polls
  • announcements
  • deck

are nice to have a look at. Since it’s a release I also tested dropit.

Unfortunately I discovered that most apps don’t enable on your server. They throw an error message, that DB tables already exist and cannot be created. Don’t know where this is coming from, your test environment or NC13 RC2.

I want to point out: I’m not complaining! I’m glad your servers exist! :heart_eyes:

18

In less than a minute third-party apps are installed. :stuck_out_tongue_winking_eye:
Thanks for the test servers! :+1:

19

Either Music or Audio Player. (There can be only one)

20

Obviously. That’s why I wrote

music/ audio player

/ means “or”.

1 Like