[Qgis-developer] Failing tests consider blockers

[Jürgen E. Fischer]

Hi Larry,

On Sat, 15. Feb 2014 at 13:11:50 -0700, Larry Shaffer wrote:
> Agreed. Although, I'm not sure how any valid test that is failing could be
> classified as un-meaningful. It either tests something that needs tested
> or not, yes? Or, are you saying there are too many corner cases included?

What I wanted to say is that there are a bunch of tests that fail because they
are not up to date or fail for other reasons that are not related to actual
bugs in qgis itself. For example render tests, which fail because of fonts or
other differences (eg. removed renderers).  I think that are the majority of
tests that currently fail.

So the tests need work. But that's a known fact and it's not at all new.  Still
not much has been done about it - except talk.

Another thing is that we don't have the concept of actual blockers anymore.  We
release every four months what we have at that point.  Blockers are just
highest priority for bugs.


> Maybe there could be a 'main' test suite (that always has to pass 100% for
> major platforms before a release) and an 'extended' suite for testing less
> common use cases? Should be doable for indivudal Python tests using skip
> decorators. Not sure how to do it for individual C++ tests.

> > > Another methodology would be to invest in a bunch of different CI servers
> > > on VMs on OSGeo servers somewhere. Then, a dev could just review all
> > > failures independent of contacting anyone else. That's a large investment
> > > in time and resources, however.
 
> > We already have that - didn't I mention that already? ;)
 
> > The windows and debian/ubuntu/ubuntugis builds feed their test results into
> > cdash[1] - since about April 2012.  And your mac builds do to.  So there
> > should be enough coverage to find tests to fix.  I don't think that that is
> > the problem.  Lack of time is probably the main reason for the situation
> > and I doubt that pressure will help with that.

> As far as I know, only Tim has (or did have) a continuous integration server
> setup, using Jenkins. Otherwise, aren't they all daily/nightly builds with
> CDash reports? That's what mine are. I'm not saying that resource is not
> enough for reviewing the current master branch. It is very good and I use it
> often.

Not sure if CI would have made a big difference.  We could have fixed the
failing tests with the nightly build reports.

But I think Tim planned to add a jenkins instance to our new server.


> I am suggesting to have official or sponsored CI servers as a preventative
> measure. For example, many github accounts have a CI server hooked into their
> pull requests. This way, the repo maintainers have a constant check as to
> whether the proposed change builds and passes tests *before* the request is
> merged.

We could use travis[1] for that (other OSGeo projects already do). But I think
there is no coverage for OSX and Windows there.

Not sure where to get OSX servers and how much that would cost, but at Hetzner
Windows 2012 R2 Standard Edition costs 25 EUR extra a month (ie. 84 EUR a month
for Windows Server).  But both could probably be integrated with jenkins on the
new servers and also take over the nightly builds.
 
But that also only makes real sense once the tests are fixed (for what we don't
need any of the above).


Jürgen


[1] http://docs.travis-ci.com/user/getting-started/

-- 
Jürgen E. Fischer         norBIT GmbH               Tel. +49-4931-918175-31
Dipl.-Inf. (FH)           Rheinstraße 13            Fax. +49-4931-918175-50
Software Engineer         D-26506 Norden               http://www.norbit.de


-- 
norBIT Gesellschaft fuer Unternehmensberatung und Informationssysteme mbH
Rheinstrasse 13, 26506 Norden
GF: Jelto Buurman, HR: Amtsgericht Emden, HRB 5502