[Qgis-developer] Failing tests consider blockers

[Larry Shaffer]

Hi Jürgen,

On Fri, Feb 14, 2014 at 6:16 PM, Jürgen E. <jef at norbit.de> wrote:

> Hi Larry,
>
> On Fri, 14. Feb 2014 at 10:04:25 -0700, Larry Shaffer wrote:
> > I'm 100% for failing unit tests being a blocker for any release.
>
> If all the tests were significant.  That's not currently the case.
>
> All the render tests seem quite fragile.   There are small differences,
> font
> mismatches across platforms and version and even tests that still require
> old
> renderers that we already removed.  I don't see why those should block
> releases.  That's my point.
>

> Currently there are a lot of constantly failing tests that make it harder
> to
> see when meaningful tests start to fail.  That kind of ruins the point of
> running the tests in the first place.
>


> Don't get me wrong - of course these should either be fixed or removed -
> and
> once that happens, we can define failing tests as blockers.  But this also
> has
> been brought up again and again, but not much action has been taken.
>

Agreed. Although, I'm not sure how any valid test that is failing could be
classified as un-meaningful. It either tests something that needs tested or
not, yes? Or, are you saying there are too many corner cases included?

Maybe there could be a 'main' test suite (that always has to pass 100% for
major platforms before a release) and an 'extended' suite for testing less
common use cases? Should be doable for indivudal Python tests using skip
decorators. Not sure how to do it for individual C++ tests.

> Another methodology would be to invest in a bunch of different CI servers
> on
> > VMs on OSGeo servers somewhere. Then, a dev could just review all
> failures
> > independent of contacting anyone else. That's a large investment in time
> and
> > resources, however.
>
> We already have that - didn't I mention that already? ;)
>
> The windows and debian/ubuntu/ubuntugis builds feed their test results into
> cdash[1] - since about April 2012.  And your mac builds do to.  So there
> should
> be enough coverage to find tests to fix.  I don't think that that is the
> problem.  Lack of time is probably the main reason for the situation and I
> doubt that pressure will help with that.
>

As far as I know, only Tim has (or did have) a continuous integration
server setup, using Jenkins. Otherwise, aren't they all daily/nightly
builds with CDash reports? That's what mine are. I'm not saying that
resource is not enough for reviewing the current master branch. It is very
good and I use it often.

I am suggesting to have official or sponsored CI servers as a preventative
measure. For example, many github accounts have a CI server hooked into
their pull requests. This way, the repo maintainers have a constant check
as to whether the proposed change builds and passes tests *before* the
request is merged.

The same could be done for QGIS's github account (via one of the github
webhooks [0]), building and testing across 3 or more platforms, and
possibly OS versions as well. This way, for example, I could do a pull
request with unit tests and check back later to see if any unit tests have
failed across platforms, including ones I have no access to.

Currently the QGIS github account only offers info on whether the pull
request is merge-able by git.

I'm guessing that maybe about 3 - 6 CI servers would be enough: the 3 major
OSes, with current OS and maybe one older version (e.g. current LTS
version). Anything more than that can fallback on the CDash information
after code is committed.

I would be willing to set up a CI and maintain it for Mac OS X 10.7+, if
there were OSGeo or QGIS resources available for it. That would allow its
continuation, regardless of individual developer(s) financial resources.
Mac OS can only be legally virtualized on Mac hardware, which is a bit of a
financial roadblock.

[0] https://help.github.com/articles/creating-webhooks ,
     Also third-party Webhooks and Services under repo Settings, e.g.
Jenkins (GitHub plugin)

Regards,

Larry


> Jürgen
>
> [1] http://dash.orfeo-toolbox.org/index.php?project=QGIS
>
> --
> Jürgen E. Fischer         norBIT GmbH               Tel. +49-4931-918175-31
> Dipl.-Inf. (FH)           Rheinstraße 13            Fax. +49-4931-918175-50
> Software Engineer         D-26506 Norden
> http://www.norbit.de
> QGIS PSC member (RM)      Germany                      IRC: jef on FreeNode
>
> --
> norBIT Gesellschaft fuer Unternehmensberatung und Informationssysteme mbH
> Rheinstrasse 13, 26506 Norden
> GF: Jelto Buurman, HR: Amtsgericht Emden, HRB 5502
>
> _______________________________________________
> Qgis-developer mailing list
> Qgis-developer at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/qgis-developer
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20140215/7be469e1/attachment.html>