[Qgis-developer] Failing tests

[Matthias Kuhn]

Hi,

For more than a week already we have failing tests on master
https://travis-ci.org/qgis/QGIS/builds

This basically renders the whole test suite a lot less usable.

Some examples/thoughts:

 * Every pull request fails tests [1] I often look at the test results
to check if it's even worth looking at a pull request. Handling pull
requests is an issue on its own as there is no direct benefit for the
reviewer most of the time. Don't make the reviewer's time even harder!

 * We don't see when a commit trashes our compiler. Today we had a
commit which broke the compiler. The author was not informed by mail.

 * We don't see which commit broke a certain test. E.g. At the moment
the PyQgsAppStartup test is failing. And to find out which commit broke
it one has to dig deep because failing of this test is masked by the
already failing tests.

 * It discourages people to write tests if they can be broken easily.

The tests started to fail with the merge of some big work, just previous
to feature freeze. I can understand that people want to get commits into
master before feature freeze. But quality should not suffer from this.
And if there's something broken, please fix it fast.

Finally Nyall was starting to fix things - crucial GIS functionality
like splitting features - although I don't think he received anything in
return, thank you very much, Nyall. The one thing left to do is to add
doxygen API documentation - and to fix the newly broken PyQgsAppStartup
test. I really hope we will finally see the green light again.

I am a hippy. I don't like rules and policies. I am not in favor of a
very strong policy concerning the tests. Common sense should be valued
more than rules. Policies like "every new feature has to come with a
test" are no good solution as they encourage people to write silly
tests. And I am much too lazy to write a QEP for a policy to revert
commits which break the tests.

So... How do we fix it?
If tomorrow everybody on this list writes some API documentation we
should have fixed that test by noon.
And then I'll promise I'll fix the failing PyQgsAppStartup test to get
the green light going again.

Thank you very much
Matthias

[1] https://github.com/qgis/QGIS/pulls ;
https://github.com/qgis/QGIS/pulls?q=is%3Apr+is%3Aclosed
[2]
https://github.com/qgis/QGIS/commit/ded11b32562c100aeaae95bf20e21bcfc38d0777

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20150601/0d48e023/attachment.pgp>