[Qgis-developer] Python tests and C++ tests for the same functionality

[Matthias Kuhn]

Hi,

While working on the test suite and trying to get all tests to pass I
realized that there are a lot of tests that exist in two versions: C++
and python (e.g. composerhtml, atlas composition). The problem with that
is, that they work on the same set of reference images and either one or
the other fails because of slight inconsistencies in the result (nothing
we should worry about, see e.g. [1]). I don't know exactly, what the
source of these is, I assume that either there are some
rounding/floating point differences in the python API or that meanwhile
one or the other test was modified but not both synchroneously.

What can we do?

(1) We could accept these deviations with anomalies or multiple
reference images.

(2) Or we could remove the C++ test in favor of the python test which
will test the same functionality/codepath and the python binding in
addition. That would have the advantage that maintainability will be
easier, because not every of these tests needs to be written and
adjusted twice. The only disatvantage I can see is that you then don't
know if it's the python binding or the logic behind it that made a test
fail. But that shouldn't make a big difference in the time one needs to
debug a test.

My favorite is (2).

Are there any thoughts from other devs/authors of these tests?

Best
Matthias

[1] http://dash.orfeo-toolbox.org/testDetails.php?test=26809858&build=162985

-- 
Help getting QGIS to the next level of quality before November 15!
http://blog.vitu.ch/10102014-1046/crowdfunding-initiative-automated-testing