[Qgis-developer] Continuous Integration / Testing with TravisCI

[Even Rouault]

Matthias,

Great move from yours !

I can point you to an email I sent to the mapguide folks recently, sharing my 
experience with C.I. tools for GDAL. See the end of 
http://lists.osgeo.org/pipermail/mapguide-internals/2014-August/007518.html

I'm skeptical you'll manage to build QGIS with (the free program of) AppVeyor, 
since GDAL reaches their build time limit.

Regarding comparing images, MapServer used to use the 'perceptualdiff' utility 
that can accept some, invisible to the eye, differences between the reference 
image and the image produced. But this may be sometimes still be a bit 
oversensitive. Definitely a tricky subject to have rendering tests running 
consistently accross platforms.
You mention comparing WKT, but that can be non obvious too since there might 
be different rounding accross platforms. Better to have a dedicated geometry 
comparison method that accept an epsilon when doing vertex comparisons.

Even

> Hi all,
> 
> You may have noticed that there is a new symbol on the qgis github page
> that (hopefully) says build passing. [1]
> 
> The last week I have been busy with fixing tests for integration with
> Travis CI.
> 
> Right now, whenever somebody pushes a change to
>  * a branch in the qgis repository (that affects "master" and
> release-x_y branches)
>  * a pull request for the qgis repository
> a service called Travis CI [2] will be activated and compile the
> sourcecode and run the test suite there. That has the advantage that we
> have a reference platform (ubuntu precise for the moment) where all
> tests pass. This defined environment is a big step forward as it makes
> test runs comparable and if something goes wrong we know that code is
> responsible and not some other variable.
> Currently all tests (with a few exceptions which are disabled) are
> passing. And it would be excellent if it stays like that!
> 
> What does that mean for developers?
> 
> **Use pull requests**
> Whenever you open a pull request first you get the chance to test your
> changes without making the nice green symbol on the front page go red.
> You can push to these pull requests until the tests go green. For
> reviewers this is also nice: you don't need to spend time on pull
> requests that don't build or don't pass tests.
> 
> **Write tests**
> If you implement a new feature or fix a bug, write a test for it. If
> somebody breaks the test, the pull request or the symbol on the github
> project page will turn red. This means that it's clear which commit
> broke the test. But the test is your feature or your bugfix!
> 
> **Try to avoid rendering tests**
> A big portion of fixing has been spent on rendering tests. The problem
> with rendering tests is, that a lot of them have slight inconsistencies
> across systems. Gradients, anti aliasing, fonts... This all may render
> differently without being really a problem. But it produces a (false)
> alarm on another system and one needs to take measures. There are such
> measures: defining color intolerance, allowed pixel mismatches, known
> anomalies, since last week also different reference pictures... But in
> the end this means extra work and false alarms once in a while. We do
> create a software that has visual products. Therefore rendering tests
> are required. But think very about different possibilities before
> writing such a test. Maybe you can extract the WKT from a geometry and
> compare that against a known WKT string instead? That is less brittle,
> faster to test and much easier to maintain than a rendering test!
> If you really need to write a rendering test it is likely that it will
> fail on travis in the beginning. To make it pass on travis, create a
> pull request and wait for it to be tested. The results will be uploaded
> to the OTB CDash [3] where you get information what exactly you need to
> adjust. How many pixels failed? Visual comparison if adjusting the color
> tolerance a bit may help. Is an anomaly ok? Is there a need to add
> another reference picture?
> 
> **C++ vs. Python tests**
> I made an earlier call for having only either one or the other. That was
> based on misinterpretation of test results from my side. Sorry.
> My current opinion is: do what you feel comfortable with. If you prefer
> to write C++ tests, do so. If you prefer to write Python tests, do so.
> If you want both, do so. The important thing is: DO IT!
> If you really need advice: go for python tests, they test the same
> functionality like C++ functions but also test the python API and
> therefore cover a slightly bigger area.
> 
> Which tests are not fixed (on the reference platform)?
> 
>  * The atlas tests had interdependencies, that made some tests pass just
> because the environment was ok from previous ones. These tests are now
> changed to be independent one of another. And there is a PR that should
> fix the issues and re-enable the test. [4]
>  * The symbology tests fail due to problems with sld loading. There is a
> pull request that partly reverts a commit that broke things and
> re-enables the test. [5]
>  * The server pal labeling tests produced very different rendering
> results. I could imagine that they would be better off if the would also
> use the QGIS test font? New reference images could also be introduced.
> Dakota cartography's call.
>  * The server canvas labeling tests crash on exit because of a
> not-yet-ended thread. Multithreading issue? Something else?
>  * Recently the WCS test started to fail. I guess the server it uses to
> test does not always respond? We should either fix the server, disable
> the test, increase the timeout...
> 
> Where to go from here?
> 
> There are two things I would like to see:
> 
>   Many more tests :)
>   More platforms: Travis can also handle mac. We could ask them to
> enable mac testing for us. But we'd first need to fix the tests there.
> There is a service called appveyor that runs tests for windows. But we'd
> first need to fix the tests there.
> 
> I am very happy that there was such a great interest in the crowdfunding
> campaign that made it possible to do that work. THANK YOU to everybody
> who helped to get the testing integration to the next level.
> 
> Matthias
> 
> [1] https://github.com/qgis/QGIS
> [2] http://travis-ci.org/
> [3] http://dash.orfeo-toolbox.org/index.php?project=QGIS
> 
> _______________________________________________
> Qgis-developer mailing list
> Qgis-developer at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/qgis-developer

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com