<div dir="ltr">Hi Jürgen,<br><div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 14, 2014 at 6:16 PM, Jürgen E. <span dir="ltr"><<a href="mailto:jef@norbit.de" target="_blank">jef@norbit.de</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Larry,<br>
<br>
On Fri, 14. Feb 2014 at 10:04:25 -0700, Larry Shaffer wrote:<br>
> I'm 100% for failing unit tests being a blocker for any release.<br>
<br>
If all the tests were significant. That's not currently the case.<br>
<br>
All the render tests seem quite fragile. There are small differences, font<br>
mismatches across platforms and version and even tests that still require old<br>
renderers that we already removed. I don't see why those should block<br>
releases. That's my point.<br></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Currently there are a lot of constantly failing tests that make it harder to<br>
see when meaningful tests start to fail. That kind of ruins the point of<br>
running the tests in the first place.<br></blockquote><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Don't get me wrong - of course these should either be fixed or removed - and<br>
once that happens, we can define failing tests as blockers. But this also has<br>
been brought up again and again, but not much action has been taken.<br></blockquote><div><br>Agreed. Although, I'm not sure how any valid test that is failing could be classified as un-meaningful. It either tests something that needs tested or not, yes? Or, are you saying there are too many corner cases included?<br>
<br>Maybe there could be a 'main' test suite (that always has to pass 100% for major platforms before a release) and an 'extended' suite for testing less common use cases? Should be doable for indivudal Python tests using skip decorators. Not sure how to do it for individual C++ tests.<br>
<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> Another methodology would be to invest in a bunch of different CI servers on<br>
> VMs on OSGeo servers somewhere. Then, a dev could just review all failures<br>
> independent of contacting anyone else. That's a large investment in time and<br>
> resources, however.<br>
<br>
We already have that - didn't I mention that already? ;)<br>
<br>
The windows and debian/ubuntu/ubuntugis builds feed their test results into<br>
cdash[1] - since about April 2012. And your mac builds do to. So there should<br>
be enough coverage to find tests to fix. I don't think that that is the<br>
problem. Lack of time is probably the main reason for the situation and I<br>
doubt that pressure will help with that.<br></blockquote><div><br></div><div>As far as I know, only Tim has (or did have) a continuous integration server setup, using Jenkins. Otherwise, aren't they all daily/nightly builds with CDash reports? That's what mine are. I'm not saying that resource is not enough for reviewing the current master branch. It is very good and I use it often.<br>
<br></div><div>I am suggesting to have official or sponsored CI servers as a preventative measure. For example, many github accounts have a CI server hooked into their pull requests. This way, the repo maintainers have a constant check as to whether the proposed change builds and passes tests *before* the request is merged.<br>
<br></div><div>The same could be done for QGIS's github account (via one of the github webhooks [0]), building and testing across 3 or more platforms, and possibly OS versions as well. This way, for example, I could do a pull request with unit tests and check back later to see if any unit tests have failed across platforms, including ones I have no access to.<br>
<br>Currently the QGIS github account only offers info on whether the pull request is merge-able by git.<br><br></div><div>I'm guessing that maybe about 3 - 6 CI servers would be enough: the 3 major OSes, with current OS and maybe one older version (e.g. current LTS version). Anything more than that can fallback on the CDash information after code is committed.<br>
</div><div><br></div><div>I would be willing to set up a CI and maintain it for Mac OS X 10.7+, if there were OSGeo or QGIS resources available for it. That would allow its continuation, regardless of individual developer(s) financial resources. Mac OS can only be legally virtualized on Mac hardware, which is a bit of a financial roadblock.<br>
</div><div><br>[0] <a href="https://help.github.com/articles/creating-webhooks">https://help.github.com/articles/creating-webhooks</a> ,<br></div><div> Also third-party Webhooks and Services under repo Settings, e.g. Jenkins (GitHub plugin)<br>
</div><div><br></div><div>Regards,<br><br></div><div>Larry<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Jürgen<br>
<br>
[1] <a href="http://dash.orfeo-toolbox.org/index.php?project=QGIS" target="_blank">http://dash.orfeo-toolbox.org/index.php?project=QGIS</a><br>
<span class=""><font color="#888888"><br>
--<br>
Jürgen E. Fischer norBIT GmbH Tel. +49-4931-918175-31<br>
Dipl.-Inf. (FH) Rheinstraße 13 Fax. +49-4931-918175-50<br>
Software Engineer D-26506 Norden <a href="http://www.norbit.de" target="_blank">http://www.norbit.de</a><br>
QGIS PSC member (RM) Germany IRC: jef on FreeNode<br>
<br>
--<br>
norBIT Gesellschaft fuer Unternehmensberatung und Informationssysteme mbH<br>
Rheinstrasse 13, 26506 Norden<br>
GF: Jelto Buurman, HR: Amtsgericht Emden, HRB 5502<br>
<br>
_______________________________________________<br>
Qgis-developer mailing list<br>
<a href="mailto:Qgis-developer@lists.osgeo.org">Qgis-developer@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/mailman/listinfo/qgis-developer" target="_blank">http://lists.osgeo.org/mailman/listinfo/qgis-developer</a><br>
</font></span></blockquote></div><br></div></div></div>