[Qgis-psc] [QGIS-Developer] Unscheduled 3.18.1 release?

[Alexandre Neto]

Hi,

On Sun, Feb 28, 2021 at 11:09 PM Nyall Dawson <nyall.dawson at gmail.com>
wrote:

> - Isn't the Kiwi user test framework supposed to be directed at
> recognising these regressions? Could we get a status update on that
> and what needs to be done on pushing that effort forward? (The
> https://github.com/qcooperative/qgis-core-tests repo seems to be all
> but abandoned...?)
>
> *<TL;DR>*
1. Yes, indeed, the goal of the QA testing framework [0] is to catch this
kind of anomalies before normal users do.
2. The mentioned repository represents only a sub-part of the QA
infrastructure, the work has continued on the kiwi platform (check a more
detailed status below).
3. Putting it simply, to move forward, we now need, community
feedback\buy-in, more tests, more testers, and ofc time.
4. Giovanni's freeze/Release candidate proposal is absolutely necessary if
we want to be able to catch issues related to packaging, which includes
dependencies bumps or changes. It's hard, if not impossible, to test a
moving target. Besides, manual testing is a time-consuming activity, one
week would be better than nothing but, very short to run all the (expected)
test cases, have time to fix the issues, and re-test the fixes.
*</TL;DR>*

*QEP 180 Status*
Regarding the status of the work, we have:
- established/proposed a base methodology for how to run tests [0]
<https://github.com/qcooperative/qgis-testing/blob/main/README.md>;
- implemented a TCMS (Test Case Management System) infrastructure [1]
<https://qgis.tenant.kiwitcms.org/> using https://kiwitcms.org/ free
instance (available for open source projects);
- created a limited set of test cases (manual tests) on kiwi [2]
<https://qgis.tenant.kiwitcms.org/cases/search/>, to serve as examples and
for internal testing (so far);
- revived Boundless's the tester plugin [3]
<https://github.com/qcooperative/qgis-tester-plugin>;
- created a limited set of semi-automated test for the tester plugin [4]
<https://github.com/qcooperative/qgis-core-tests> (the repo mentioned
above), to serve as examples and for internal testing (so far);
- organized and executed test plans for the 3.16.4 and 3.18.0 releases, for
Ubuntu 20.04 and Windows 10 [5]
<https://qgis.tenant.kiwitcms.org/plan/search/>, using several nightlies,
in an iterative procedure, to polish the process as much as possible.

*Next steps (outside QEP 180 - grant 2020 scope)*
Until the next release (3.20), my plan is to:
- Create more test cases. This can't be done without the community buy-in,
including the developers, that, better than anyone else, know what has been
changed during the release cycle, and what functionalities were not
possible to fully cover using unit tests. Alessandro already requested test
cases for geopackages and spatialite usage in the browser panel, which is
WIP.
- Attract users to become testers and train them.
- Organize the test plans for 3.20 release, in as many platforms as
possible (depending on the number of available testers, ofc)
- Execute the test plans and report the issues. This should be done during
the feature freeze period, ideally on a full freeze including from
packaging or Release candidate.

More stuff to do:
- Move testing documentation [0] to official QGIS documentation.
- Create a testing data set for use in tests (or improve QGIS data sample).

*Blockers\annoyances*
The fact that we are running kiwi on a free instance of
https://kiwitcms.org/ has a few limitations:
1. A very annoying two-step registry process. People need to register on
kiwi.org, and then request access to the QGIS tenant to someone already
accepted.
2. No users and group permissions settings. Meaning that all users that
have access to QGIS kiwi instance have read and write access to all test
cases, test plans, executions, etc... Anyone can make a mistake and erase a
lot of work from others.
3. No public (unregistered read-only) access to test cases and test plans,
to use in GitHub and mailing list links. This is not related to the free
instance, it's how kiwi works.

I think annoyances 1 and 2 could be solved by hosting our own kiwi instance
or paying for one of their subscriptions plan (32€/mo).

For annoyance 3, a self-hosted instance would allow for a workaround. I
have opened a Feature Request [6]
<https://github.com/kiwitcms/Kiwi/issues/2229> on kiwi github, but
obviously, that does not mean it will ever be implemented. I already asked
for a cost estimate.

Best regards,

Alexandre Neto

[0] - https://github.com/qcooperative/qgis-testing/blob/main/README.md
[1] - https://qgis.tenant.kiwitcms.org
[2] - https://qgis.tenant.kiwitcms.org/cases/search/
[3] - https://github.com/qcooperative/qgis-tester-plugin
[4] - https://github.com/qcooperative/qgis-core-tests
[5] - https://qgis.tenant.kiwitcms.org/plan/search/
[6] - https://github.com/kiwitcms/Kiwi/issues/2229


Nyall
>
>
>
>
>
> > I like this idea.
> >
> > Would you see that happen during feature freeze or post release?
> > How would this work branding wise, i.e. what will be the name of this,
> something like beta or release candidate?
> >
> > Matthias
> > _______________________________________________
> > Qgis-psc mailing list
> > Qgis-psc at lists.osgeo.org
> > https://lists.osgeo.org/mailman/listinfo/qgis-psc
> _______________________________________________
> Qgis-psc mailing list
> Qgis-psc at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/qgis-psc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-psc/attachments/20210301/3411d3ee/attachment.html>