[Qgis-user] Crowdfunding Project: Automated tests for QGIS

[Jürgen E. Fischer]

Hi Matthias,

On Sat, 11. Oct 2014 at 14:29:18 +0200, Matthias Kuhn wrote:
>> But I must admit that I only fixed a few.  I think most of the tests are not
>> actually pointing at bugs - except in the test itself.  Most of the failing
>> tests are merily testing stuff that intentionally was changed, is even gone
>> or replaced with something better.  E.g. instead of testing the (old)
>> rendering the test now essentially outlines that the migration from old to
>> new rendering settings is flawed, but that's no rendering issue.

> My point is, that it should not be you fixing the bugs, but the one's that
> broke it.

I usually also only fix my own bugs.  But there are also a lot of bugs that
stay unaddressed for a while.  And for those it's often not that hard to figure
out how they were introduced.


> And it should not be once in a while, but just at the time of breaking a test
> because that's when a developer still knows the code best and therefore can
> adapt it much easier than later on.

It's just not like that couldn't have happened before and still didn't.  Having
travis in place will probably raise the bar.

> In the rendering example that means, that when these changes are merged (best
> via pull request) the developer notices that there are failing  tests and has
> to decide if the tests are obsolete (disable them, but  consciously), if he
> needs to write compatibility code (flawed migration  from old to new
> settings) and that the same things could also be tested  with the new
> rendering engine (writing some new tests)

Or just decides that there are more important things to do than tests.  Making
the visibility of the tests will raise that bar, but still...


>>> Main points to nightly build vs. per-commit build is that responsibility is
>>> visible. And therefore it's possible to revert the responsible commit.  And
>>> there's mail notification that makes it pretty hard to ignore failing
>>> tests.  Additional plus point for Travis CI -> github is that pull requests
>>> are  also tested.

>> Of course. Using travis would be nice and I don't see any possible harm.
>> And having that from the start would have not let the tests become that
>> messy and hard to clean up.  No so sure that it is hard to ignore ;)

> If it is too easy to ignore it we could raise the ignorance bar with a  "git
> revert" ;)

We wouldn't have done that with pallabeling, symbology-ng nor MTR - even if
there had been notifications about broken tests ;)

But I'm not arguing against travis, more tests, better visibility of the
results, clearer assignment of problems.  Why would I?  All that can help.

I'm just voicing doubts that it's a game changer - that's all.


>> But I'm hesitant about introducing a hard requirement for new stuff to have
>> (meaningful and significant enough) tests, because that might keep out nice
>> new stuff - even if its working fine or would be easy enough to fix once
>> exposed to a broader base of brave users.

> That is not part of my initiative and should be discussed separately in
> Nyall's QEP/RFC.

Sure, but it's related and could be used to backup it up.


> I think that my initiative will already encourage people to write tests,
> regardless if it's a hard requirement or not.

Me too.



Jürgen

-- 
Jürgen E. Fischer           norBIT GmbH             Tel. +49-4931-918175-31
Dipl.-Inf. (FH)             Rheinstraße 13          Fax. +49-4931-918175-50
Software Engineer           D-26506 Norden             http://www.norbit.de
QGIS release manager (PSC)  Germany                    IRC: jef on FreeNode                         
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 827 bytes
Desc: Digital signature
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20141011/c1aca611/attachment.sig>