[Qgis-developer] Continuous Integration / Testing with TravisCI

[Tim Sutton]

Hi

On Tue, Nov 18, 2014 at 4:52 PM, Matthias Kuhn <matthias.kuhn at gmx.ch> wrote:

> Hi all,
>
> You may have noticed that there is a new symbol on the qgis github page
> that (hopefully) says build passing. [1]
>
> The last week I have been busy with fixing tests for integration with
> Travis CI.
>
> Right now, whenever somebody pushes a change to
>  * a branch in the qgis repository (that affects "master" and
> release-x_y branches)
>  * a pull request for the qgis repository
> a service called Travis CI [2] will be activated and compile the
> sourcecode and run the test suite there. That has the advantage that we
> have a reference platform (ubuntu precise for the moment) where all
> tests pass. This defined environment is a big step forward as it makes
> test runs comparable and if something goes wrong we know that code is
> responsible and not some other variable.
> Currently all tests (with a few exceptions which are disabled) are
> passing. And it would be excellent if it stays like that!
>
> What does that mean for developers?
>
> **Use pull requests**
> Whenever you open a pull request first you get the chance to test your
> changes without making the nice green symbol on the front page go red.
> You can push to these pull requests until the tests go green. For
> reviewers this is also nice: you don't need to spend time on pull
> requests that don't build or don't pass tests.
>
> **Write tests**
> If you implement a new feature or fix a bug, write a test for it. If
> somebody breaks the test, the pull request or the symbol on the github
> project page will turn red. This means that it's clear which commit
> broke the test. But the test is your feature or your bugfix!
>
> **Try to avoid rendering tests**
> A big portion of fixing has been spent on rendering tests. The problem
> with rendering tests is, that a lot of them have slight inconsistencies
> across systems. Gradients, anti aliasing, fonts... This all may render
> differently without being really a problem. But it produces a (false)
> alarm on another system and one needs to take measures. There are such
> measures: defining color intolerance, allowed pixel mismatches, known
> anomalies, since last week also different reference pictures... But in
> the end this means extra work and false alarms once in a while. We do
> create a software that has visual products. Therefore rendering tests
> are required. But think very about different possibilities before
> writing such a test. Maybe you can extract the WKT from a geometry and
> compare that against a known WKT string instead? That is less brittle,
> faster to test and much easier to maintain than a rendering test!
> If you really need to write a rendering test it is likely that it will
> fail on travis in the beginning. To make it pass on travis, create a
> pull request and wait for it to be tested. The results will be uploaded
> to the OTB CDash [3] where you get information what exactly you need to
> adjust. How many pixels failed? Visual comparison if adjusting the color
> tolerance a bit may help. Is an anomaly ok? Is there a need to add
> another reference picture?
>
> **C++ vs. Python tests**
> I made an earlier call for having only either one or the other. That was
> based on misinterpretation of test results from my side. Sorry.
> My current opinion is: do what you feel comfortable with. If you prefer
> to write C++ tests, do so. If you prefer to write Python tests, do so.
> If you want both, do so. The important thing is: DO IT!
> If you really need advice: go for python tests, they test the same
> functionality like C++ functions but also test the python API and
> therefore cover a slightly bigger area.
>
> Which tests are not fixed (on the reference platform)?
>
>  * The atlas tests had interdependencies, that made some tests pass just
> because the environment was ok from previous ones. These tests are now
> changed to be independent one of another. And there is a PR that should
> fix the issues and re-enable the test. [4]
>  * The symbology tests fail due to problems with sld loading. There is a
> pull request that partly reverts a commit that broke things and
> re-enables the test. [5]
>  * The server pal labeling tests produced very different rendering
> results. I could imagine that they would be better off if the would also
> use the QGIS test font? New reference images could also be introduced.
> Dakota cartography's call.
>  * The server canvas labeling tests crash on exit because of a
> not-yet-ended thread. Multithreading issue? Something else?
>  * Recently the WCS test started to fail. I guess the server it uses to
> test does not always respond? We should either fix the server, disable
> the test, increase the timeout...
>
> Where to go from here?
>
> There are two things I would like to see:
>
>   Many more tests :)
>   More platforms: Travis can also handle mac. We could ask them to
> enable mac testing for us. But we'd first need to fix the tests there.
> There is a service called appveyor that runs tests for windows. But we'd
> first need to fix the tests there.
>
> I am very happy that there was such a great interest in the crowdfunding
> campaign that made it possible to do that work. THANK YOU to everybody
> who helped to get the testing integration to the next level.
>
>
​Brilliant stuff Matthias! Now we need to agree to 'no new code without
tests' as a formal requirement and we will be a long way down the road
towards a more stable QGIS.

Regards

Tim​



> Matthias
>
> [1] https://github.com/qgis/QGIS
> [2] http://travis-ci.org/
> [3] http://dash.orfeo-toolbox.org/index.php?project=QGIS
>
> _______________________________________________
> Qgis-developer mailing list
> Qgis-developer at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/qgis-developer
>



-- 
------------------------------------------------------------------------------------------
Tim Sutton
Visit http://kartoza.com to find out about open source:
 * Desktop GIS programming services
 * Geospatial web development
* GIS Training
* Consulting Services
Skype: timlinux Irc: timlinux on #qgis at freenode.net
Tim is a member of the QGIS Project Steering Committee
-------------------------------------------------------------------------------------------
Kartoza is a merger between Linfiniti and Afrispatial
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20141118/12cbc9a6/attachment-0001.html>