[Qgis-psc] Grant QEP 180 report - Quality Assurance methodology and infrastructure

[Alexandre Neto]

Hi Nyall,

On Tue, Dec 22, 2020 at 4:46 AM Nyall Dawson <nyall.dawson at gmail.com> wrote:

> Can you open issue tracking on
> https://github.com/qcooperative/qgis-core-tests please?
>
>
Good catch. Done! And also for the qgis-tester-plugin repository.


> I'm wondering now what your next planned steps are. E.g.
> -  How are you planning on growing the testing user base? Will there
> be a media push for users to get involved, or will that be delayed
> till after the test suite is expanded?
>

We were doing to do a media push right away (well, starting in January) to
the developers/contributors community. Our aim, for now, is to provide
enough examples of test cases and test plans. The test suite will never be
finished, so it would be great to have a broader group of people discuss
which tests make sense to do manually, and which don't. Also, which tests
make sense to run in every release, etc...

Normal users (or power users) would be asked to be involved in the testing
itself for the 3.20 release, where we expect to have some details ironed
out.


> - Similarly, what's the plan to get the community involved in writing the
> tests?
>

Ideally, we would recruit a small team of people (power users) that would
be willing to write tests based on requests from developers.
Tests make sense when it's hard or impossible to implement unit tests. So I
think we need to involve developers that will point the "testing team" to
flaws in (unit) tests. We have started to write some proposed workflows
that would lead to writing manual test cases. I still want to have more
workflows examples.


> - Where does the division sit between writing a test in the plugin vs
> just adding a test in Kiwi? I noticed that there's some logic and
> pyqgis scripting involved in writing the tests in the plugin, which
> would definitely be a barrier of entry for non-developers. Is the
> intention that EVERY test in kiwi has a corresponding test in the
> plugin? Or could the bulk of the tests be added solely in kiwi (by non
> developers) and the plugin left as an optional "nice to have"?
>

The idea is to have the tests written in kiwi, then, if we feel that a
particular test case will be very used and could benefit from some
automation, we will try to move it to the tester plugin, and remove or
disable the test case from kiwi. There will be a test case in kiwi to run
all the semi-automated tests in a specific category. We already have an
example like that for regression tests:

https://qgis.tenant.kiwitcms.org/case/8/


> -  Who will be managing this service and pushing for community to run
> the test suites prior to releases? Has anyone formally taken on this
> responsibility yet?
>

The grant proposal included running the example test suite for the next two
releases 3.18 and 3.20. This will also be a way of improving the
methodology. After that, I will gladly assume this responsibility out of my
personal time.

Tomorrow morning I will be talking about what we did until now for QGIS
Open Day.

Thanks for your questions and interest

Alexandre Neto


>
> Thanks for your hard work!
> Nyall
>
>
>
>
>
> >
> > Alexandre Neto
> >
> >
> >>
> >> Nyall
> >>
> >> >
> >> >
> >> >>
> >> >>> It covers all the steps needed to participate in the testing
> process, from setting up a clean environment to running tests and creating
> new test cases in the KIWI TCMS.
> >> >>>
> >> >>>
> >> >>> Thanks a lot for this opportunity, we hope it's a huge step forward
> for QGIS QA!
> >> >>
> >> >> thanks to you!
> >> >> looking forward to see the first results
> >> >
> >> >
> >> > And we are curious to see what the community thinks of our proposal.
> All feedback is welcome!
> >> >
> >> > Thanks
> >> >
> >> > Alexandre Neto
> >> >
> >> >>>
> >> >> cheers Marco
> >> >>>
> >> >>>
> >> >>> Best regards,
> >> >>>
> >> >>>
> >> >>> Alexandre Neto
> >> >>> Alexander Bruy
> >> >>> Giovanni Manghi
> >> >>> _______________________________________________
> >> >>> Qgis-psc mailing list
> >> >>> Qgis-psc at lists.osgeo.org
> >> >>> https://lists.osgeo.org/mailman/listinfo/qgis-psc
> >> >
> >> > _______________________________________________
> >> > Qgis-psc mailing list
> >> > Qgis-psc at lists.osgeo.org
> >> > https://lists.osgeo.org/mailman/listinfo/qgis-psc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-psc/attachments/20201223/23adba6a/attachment.html>