[Qgis-developer] Failing tests consider blockers

[Jürgen E. Fischer]

Hi Larry,

On Fri, 14. Feb 2014 at 10:04:25 -0700, Larry Shaffer wrote:
> I'm 100% for failing unit tests being a blocker for any release.

If all the tests were significant.  That's not currently the case.

All the render tests seem quite fragile.   There are small differences, font
mismatches across platforms and version and even tests that still require old
renderers that we already removed.  I don't see why those should block
releases.  That's my point.

Currently there are a lot of constantly failing tests that make it harder to
see when meaningful tests start to fail.  That kind of ruins the point of
running the tests in the first place.

Don't get me wrong - of course these should either be fixed or removed - and
once that happens, we can define failing tests as blockers.  But this also has
been brought up again and again, but not much action has been taken.


> Another methodology would be to invest in a bunch of different CI servers on
> VMs on OSGeo servers somewhere. Then, a dev could just review all failures
> independent of contacting anyone else. That's a large investment in time and
> resources, however.

We already have that - didn't I mention that already? ;)

The windows and debian/ubuntu/ubuntugis builds feed their test results into
cdash[1] - since about April 2012.  And your mac builds do to.  So there should
be enough coverage to find tests to fix.  I don't think that that is the
problem.  Lack of time is probably the main reason for the situation and I
doubt that pressure will help with that.


Jürgen

[1] http://dash.orfeo-toolbox.org/index.php?project=QGIS

-- 
Jürgen E. Fischer         norBIT GmbH               Tel. +49-4931-918175-31
Dipl.-Inf. (FH)           Rheinstraße 13            Fax. +49-4931-918175-50
Software Engineer         D-26506 Norden               http://www.norbit.de
QGIS PSC member (RM)      Germany                      IRC: jef on FreeNode                         

-- 
norBIT Gesellschaft fuer Unternehmensberatung und Informationssysteme mbH
Rheinstrasse 13, 26506 Norden
GF: Jelto Buurman, HR: Amtsgericht Emden, HRB 5502