[Qgis-developer] Rethinking the testing and release procedure of QGIS

Thu Jul 7 05:32:26 EDT 2011

Hi

On Thu, Jul 7, 2011 at 10:54 AM, Andreas Neumann <a.neumann at carto.net> wrote:
> yes - I think it should be the goal that developers can work full time on
> QGIS. And a policy should be introduced that all newly implemented features
> should have tests.
>
> I wouldn't only blame the paying customers for not wanting the unit tests,
> it is just as much the duty of the developer to educate the customer about
> the unit tests/testing infrastructure. We do some substantial funding of
> QGIS, but we were not aware about the problem of the lacking tests until
> just recently. I don't see a problem to pay an extra amount dedicated to
> testing the new features.
>

Oh sorry my tone came across wrong - I wasnt trying to 'blame' anyone :-)

> Sandro Santilli (strk) is a Postgis developer who did some work for us. He
> is always doing the testing and documentation as part of the contract. I
> think this should also be the case for QGIS development.
>

Yes this would be great.

> But for the existing functionality of QGIS we need to find a different
> solution. Strk said that he would be available to help in building a test
> infrastructure if we can find funding. We from our part would be willing to
> help funding such an effort (of course there would have to be a couple other
> organizations to help as well).
>

We already have such an infrastructure - under tests in the source
tree -  but as Sandro pointed out it is in need of revision and the
coverage of the code base is very poor. Let me try to explain what is
already in place:

We (mainly myself and Martin I think) implemented some basic tests for
core and gui under tests/src/
I also implemented a tool called renderchecker that does an image
comparison between a control image and an image rendered off the code
base
I have documented in quite some detail in CODING the procedure for
writing a test
Martin has written the test_builder.pl to generate test stubs
I have a test suite builder script (which will try to run test_builder
for every source file in core)
I put some example regression tests too e.g. tests/src/core/regression992.cpp
We have scripts that will run the tests tests/src/runtests.sh
There is a test template tests/src/core/test_template.cpp

So there is already a lot of groundwork in place for testing - though
it has not been maintained for some time for reasons stated in my
previous email. As mentioned above, Martin once wrote some perl script
(tests/src/core/test_builder.pl) to automate the generation of test
stubs. I think that some meta programming of this nature could be
useful to create initial test cases covering the whole of core and
gui. Then someone could be funded to populate all the stubs with
tests. Ideally if he was going to revise it, we should accompany the
revision with setting up automated cmake testing on the server so that
it runs the tests nightly. CMake already has all the tools for doing
this, it just needs someone to spend the time to do it.

I think we should focus on testing the libraries - in the future Qt
also provides an infrastructure for creating GUI automation tests.
What I am trying to say really is that we did already put quite some
thought into testing but we are spread too thin to carry it through.
So having someone funded to get the tests really working nicely would
be an awesome addition to the project. From a social side, mandating
tests hasnt really worked till now - I guess we are always a bit
scared of losing peoples interest (which we depend on so much) by
setting too many difficult barriers in place for them to contribute.
At some point I guess we will have to bite the bullet and start
refusing contributions unless accompanied by tests (and that they dont
break existing tests).

Regards

Tim

> Thank you for your thoughts,
> Andreas
>
> On Thu, 7 Jul 2011 10:35:54 +0200, Tim Sutton wrote:
>>
>> Hi Andreas and others
>>
>>
>> I wrote a bit of an essay on this a week or two back:
>> http://linfiniti.com/2011/06/some-thoughts-on-the-future-of-qgis/
>>
>> Believe me we have tried just about every suggestion including having
>> a policy of unit tests for new features and bug fixes. The truth is
>> that until we have undirected funding sufficient for the project to
>> employ some of us developers to work full time on QGIS, things will
>> probably never improve - developers need to earn a living which means
>> they do work on QGIS based on client requirements or do other work and
>> work on QGIS in their spare time. Most clients don't mandate tests be
>> written to accompany the work they commission. With so many things
>> that need working on, testing etc just doesnt attract the attention it
>> should. So the idea of creating some kind of fund that is aimed at
>> having a group of developers working full time on stabilising and
>> professionalising QGIS is where we need to head for. I have already
>> been having some behind the scenes discussions with potential funding
>> sources on this, but so far nothing has materialised. I believe in the
>> future it will come to pass that we can do this. From your side you
>> can help too - when you contract developers to create new features,
>> mandate unit tests to be part of the deliverables. If every
>> organisation who was paying to add functionality to QGIS did this we
>> would already be in a lot better  shape!
>>
>> Regards
>>
>> Tim
>>
>> On Sat, Jul 2, 2011 at 10:51 AM, Sandro Santilli <strk at keybit.net> wrote:
>>>
>>> On Sat, Jul 02, 2011 at 09:32:14AM +0200, Paolo Cavallini wrote:
>>>>
>>>> Il 30/06/2011 21:55, Andreas Neumann ha scritto:
>>>>
>>>> > Maybe the establishment of a testing infrastructure, as proposed by
>>>> > Strk
>>>> > and others, would also help to maintain/raise the quality.
>>>>
>>>> Agreed: AFAICT this is a major task: any idea of how much work would be
>>>> necessary for
>>>> reasonable results? An order of magnitude would be enough for now.
>>>
>>> I belive a 30k figure might get things going far enough that developers
>>> could have a policy of no functional change and no bugfix w/out a test
>>> accompanying would be allowed.
>>>
>>> --strk;
>>>
>>>  ()   Free GIS & Flash consultant/developer
>>>  /\   http://strk.keybit.net/services.html
>>> _______________________________________________
>>> Qgis-developer mailing list
>>> Qgis-developer at lists.osgeo.org
>>> http://lists.osgeo.org/mailman/listinfo/qgis-developer
>>>
>>
>>
>>
>> --
>> Tim Sutton - QGIS Project Steering Committee Member (Release  Manager)
>> ==============================================
>> Please do not email me off-list with technical
>> support questions. Using the lists will gain
>> more exposure for your issues and the knowledge
>> surrounding your issue will be shared with all.
>>
>> Visit http://linfiniti.com to find out about:
>>  * QGIS programming and support services
>>  * Mapserver and PostGIS based hosting plans
>>  * FOSS Consulting Services
>> Skype: timlinux
>> Irc: timlinux on #qgis at freenode.net
>> ==============================================
>> _______________________________________________
>> Qgis-developer mailing list
>> Qgis-developer at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/qgis-developer
>
> --
> --
> Andreas Neumann
> Böschacherstrasse 10A
> 8624 Grüt (Gossau ZH)
> Switzerland
> _______________________________________________
> Qgis-developer mailing list
> Qgis-developer at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/qgis-developer
>

-- 
Tim Sutton - QGIS Project Steering Committee Member (Release  Manager)
==============================================
Please do not email me off-list with technical
support questions. Using the lists will gain
more exposure for your issues and the knowledge
surrounding your issue will be shared with all.

Visit http://linfiniti.com to find out about:
 * QGIS programming and support services
 * Mapserver and PostGIS based hosting plans
 * FOSS Consulting Services
Skype: timlinux
Irc: timlinux on #qgis at freenode.net
==============================================