[Qgis-user] Crowdfunding Project: Automated tests for QGIS

[Matthias Kuhn]

Hi Jürgen

On 10/10/2014 03:33 PM, Jürgen E. Fischer wrote:
> Hi Matthias,
>
> On Fri, 10. Oct 2014 at 14:35:39 +0200, Matthias Kuhn wrote:
>> I'm pretty sure it is. Do you ever looks at this page? I doubt that the
>> majority of devs even knows it exists. They do however know that github
>> exists.
> I do and I mention it whenever this comes up again - so I figure everyone who
> doesn't know is probably quite new or doesn't care enough.

Good thing that you do. I must admit, I don't do it much - but one 
should not judge others by one's own standard.
A switch from "pull" information (by visiting the cdash) to "push" 
information (by mail/github pull requests) should be nice for all.

>
> But I must admit that I only fixed a few.  I think most of the tests are not
> actually pointing at bugs - except in the test itself.  Most of the failing
> tests are merily testing stuff that intentionally was changed, is even gone or
> replaced with something better.  E.g. instead of testing the (old) rendering
> the test now essentially outlines that the migration from old to new rendering
> settings is flawed, but that's no rendering issue.

My point is, that it should not be you fixing the bugs, but the one's 
that broke it. And it should not be once in a while, but just at the 
time of breaking a test because that's when a developer still knows the 
code best and therefore can adapt it much easier than later on.

In the rendering example that means, that when these changes are merged 
(best via pull request) the developer notices that there are failing 
tests and has to decide if the tests are obsolete (disable them, but 
consciously), if he needs to write compatibility code (flawed migration 
from old to new settings) and that the same things could also be tested 
with the new rendering engine (writing some new tests)

>
>> Main points to nightly build vs. per-commit build is that responsibility  is
>> visible. And therefore it's possible to revert the responsible commit. And
>> there's mail notification that makes it pretty hard to  ignore failing tests.
>> Additional plus point for Travis CI -> github is that pull requests are  also
>> tested.
> Of course. Using travis would be nice and I don't see any possible harm.  And
> having that from the start would have not let the tests become that messy and
> hard to clean up.  No so sure that it is hard to ignore ;)

If it is too easy to ignore it we could raise the ignorance bar with a 
"git revert" ;)

>
> But I'm hesitant about introducing a hard requirement for new stuff to have
> (meaningful and significant enough) tests, because that might keep out nice new
> stuff - even if its working fine or would be easy enough to fix once exposed to
> a broader base of brave users.

That is not part of my initiative and should be discussed separately in 
Nyall's QEP/RFC. I think that my initiative will already encourage 
people to write tests, regardless if it's a hard requirement or not.

>
> But that also implies that we'll probably end up with low test coverage.  Are
> there actually people that find tests fun to write?

I enjoy to see them pass. And I sometimes find myself having fun writing 
tests. But in general, writing new features is more "rewarding" than 
writing a test.

Matthias