[Qgis-psc] [QGIS-Developer] Unscheduled 3.18.1 release?

Mon Mar 1 17:33:55 PST 2021

hi All,
thanks for all the inputs. It seems to me that the general consensus is
that we should have a testing phase at the beginning of a new release cycle.

I think that communication is key and that our user need to be clearly
instructed on what this releases are all about. I see more of a
communicative issue here than a technical one.
We are probably not doing a great job in letting people know that the they
should be trying our weekly packages before the official release and
formalizing this with something like a "release candidate" naming would
make this kind of communication easier and clearer.

3.18.0 to us is a new package from a new series but, for regular users or
even for some power users, it is just a number whereas "release candidate"
is something that automatically signals what the package is about.

I would suggest renaming .0 packages into release candidates and allow 2
weeks testing. After that we can release the next package and start the
regular monthly bugfixing packaging intervals.

Additionally we could allow (If during the two weeks of testings we
recognize bad issues like it happened with the last release) creating new
release candidate packages and postponing the release announcement.

I really think this kind of approach would benefit the project a lot since
we would not be shipping code that has not been tested at large and we
would have a simple way to tell people that we're getting a new version
ready and it is now time to test it heavily.

We've had a fantastic feedback loop and according rating increase of 0.4
stars in QField since we started releasing using a similar approach.

I'm convinced that having this together with all automated and
semi-automated testing that we do now have will allow QGIS to become even
more stable.

Later today I'll rise this at the PSC meeting.

For what is worth, this might also be worth restarting separately the
discussion about having only 2 releases per year. In which case I'd suggest
allowing 1 month of RC testing.

Cheers Marco

On Mon, 1 Mar 2021, 18:58 Alexandre Neto, <senhor.neto at gmail.com> wrote:

>
> Hi,
>
> On Sun, Feb 28, 2021 at 11:09 PM Nyall Dawson <nyall.dawson at gmail.com>
> wrote:
>
>> - Isn't the Kiwi user test framework supposed to be directed at
>> recognising these regressions? Could we get a status update on that
>> and what needs to be done on pushing that effort forward? (The
>> https://github.com/qcooperative/qgis-core-tests repo seems to be all
>> but abandoned...?)
>>
>> *<TL;DR>*
> 1. Yes, indeed, the goal of the QA testing framework [0] is to catch this
> kind of anomalies before normal users do.
> 2. The mentioned repository represents only a sub-part of the QA
> infrastructure, the work has continued on the kiwi platform (check a more
> detailed status below).
> 3. Putting it simply, to move forward, we now need, community
> feedback\buy-in, more tests, more testers, and ofc time.
> 4. Giovanni's freeze/Release candidate proposal is absolutely necessary if
> we want to be able to catch issues related to packaging, which includes
> dependencies bumps or changes. It's hard, if not impossible, to test a
> moving target. Besides, manual testing is a time-consuming activity, one
> week would be better than nothing but, very short to run all the (expected)
> test cases, have time to fix the issues, and re-test the fixes.
> *</TL;DR>*
>
> *QEP 180 Status*
> Regarding the status of the work, we have:
> - established/proposed a base methodology for how to run tests [0]
> <https://github.com/qcooperative/qgis-testing/blob/main/README.md>;
> - implemented a TCMS (Test Case Management System) infrastructure [1]
> <https://qgis.tenant.kiwitcms.org/> using https://kiwitcms.org/ free
> instance (available for open source projects);
> - created a limited set of test cases (manual tests) on kiwi [2]
> <https://qgis.tenant.kiwitcms.org/cases/search/>, to serve as examples
> and for internal testing (so far);
> - revived Boundless's the tester plugin [3]
> <https://github.com/qcooperative/qgis-tester-plugin>;
> - created a limited set of semi-automated test for the tester plugin [4]
> <https://github.com/qcooperative/qgis-core-tests> (the repo mentioned
> above), to serve as examples and for internal testing (so far);
> - organized and executed test plans for the 3.16.4 and 3.18.0 releases,
> for Ubuntu 20.04 and Windows 10 [5]
> <https://qgis.tenant.kiwitcms.org/plan/search/>, using several nightlies,
> in an iterative procedure, to polish the process as much as possible.
>
> *Next steps (outside QEP 180 - grant 2020 scope)*
> Until the next release (3.20), my plan is to:
> - Create more test cases. This can't be done without the community buy-in,
> including the developers, that, better than anyone else, know what has been
> changed during the release cycle, and what functionalities were not
> possible to fully cover using unit tests. Alessandro already requested test
> cases for geopackages and spatialite usage in the browser panel, which is
> WIP.
> - Attract users to become testers and train them.
> - Organize the test plans for 3.20 release, in as many platforms as
> possible (depending on the number of available testers, ofc)
> - Execute the test plans and report the issues. This should be done during
> the feature freeze period, ideally on a full freeze including from
> packaging or Release candidate.
>
> More stuff to do:
> - Move testing documentation [0] to official QGIS documentation.
> - Create a testing data set for use in tests (or improve QGIS data sample).
>
> *Blockers\annoyances*
> The fact that we are running kiwi on a free instance of
> https://kiwitcms.org/ has a few limitations:
> 1. A very annoying two-step registry process. People need to register on
> kiwi.org, and then request access to the QGIS tenant to someone already
> accepted.
> 2. No users and group permissions settings. Meaning that all users that
> have access to QGIS kiwi instance have read and write access to all test
> cases, test plans, executions, etc... Anyone can make a mistake and erase a
> lot of work from others.
> 3. No public (unregistered read-only) access to test cases and test plans,
> to use in GitHub and mailing list links. This is not related to the free
> instance, it's how kiwi works.
>
> I think annoyances 1 and 2 could be solved by hosting our own kiwi
> instance or paying for one of their subscriptions plan (32€/mo).
>
> For annoyance 3, a self-hosted instance would allow for a workaround. I
> have opened a Feature Request [6]
> <https://github.com/kiwitcms/Kiwi/issues/2229> on kiwi github, but
> obviously, that does not mean it will ever be implemented. I already asked
> for a cost estimate.
>
> Best regards,
>
> Alexandre Neto
>
> [0] - https://github.com/qcooperative/qgis-testing/blob/main/README.md
> [1] - https://qgis.tenant.kiwitcms.org
> [2] - https://qgis.tenant.kiwitcms.org/cases/search/
> [3] - https://github.com/qcooperative/qgis-tester-plugin
> [4] - https://github.com/qcooperative/qgis-core-tests
> [5] - https://qgis.tenant.kiwitcms.org/plan/search/
> [6] - https://github.com/kiwitcms/Kiwi/issues/2229
>
>
> Nyall
>>
>>
>>
>>
>>
>> > I like this idea.
>> >
>> > Would you see that happen during feature freeze or post release?
>> > How would this work branding wise, i.e. what will be the name of this,
>> something like beta or release candidate?
>> >
>> > Matthias
>> > _______________________________________________
>> > Qgis-psc mailing list
>> > Qgis-psc at lists.osgeo.org
>> > https://lists.osgeo.org/mailman/listinfo/qgis-psc
>> _______________________________________________
>> Qgis-psc mailing list
>> Qgis-psc at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/qgis-psc
>>
> _______________________________________________
> Qgis-psc mailing list
> Qgis-psc at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/qgis-psc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-psc/attachments/20210302/5b392829/attachment.html>