<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">All,</div>
<div class=""><br class="">
</div>
<div class="">As hinted in previous mails I have been working on a proposal to improve</div>
<div class="">testing in general. Here are my thougts on the test setup in PROJ and how we</div>
<div class="">can improve it. The testing situation in PROJ is not particularly good at the</div>
<div class="">moment. It is a lot better than a couple of years ago but there is still a lot</div>
<div class="">that can be improved. Even has already settled on a testing framework for the</div>
<div class="">new features related to WKT/WKT2- support and later on also the active use of</div>
<div class="">the EPSG database. That still leaves the rest, which could use a bit of spring</div>
<div class="">cleaning. I'll try to address that below.</div>
<div class=""><br class="">
</div>
<div class="">Today we have:</div>
<div class=""><br class="">
</div>
<div class=""> 1. nad/</div>
<div class=""> Shell scripts that call cs2cs with various input and checks the</div>
<div class=""> output against a distributed "answer book". These tests does not work</div>
<div class=""> on Windows (unless using cygwin or something similar).</div>
<div class=""><br class="">
</div>
<div class=""> 2. src/</div>
<div class=""> Individual programs testing the threading capabilities of PROJ</div>
<div class=""> (multistresstest.c & test228.c) and the geodesic library (geodtest.c).</div>
<div class=""><br class="">
</div>
<div class=""> 3. test/</div>
<div class=""> The gie test-suite [0]. Gie was created to make it simpler to write</div>
<div class=""> test cases for PROJ. A number of test files for gie are present</div>
<div class=""> in test/gie/ and test/gigs. These are mostly comprised of the auto-</div>
<div class=""> generated tests from the 'proj -C' selftest that was introduced with</div>
<div class=""> PROJ 4.9.3 (and removed again in 5.0.0) placed in builtins.gie. A</div>
<div class=""> handful of other test files have been written adding more "sane" test</div>
<div class=""> data. As part of the gie test-suite the GIGS [1] test data is included</div>
<div class=""> as well. Gie is a good environment for testing coordinate conversions</div>
<div class=""> given a setup string that can go into the proj_create() function.</div>
<div class=""><br class="">
</div>
<div class=""> 4. gie.c</div>
<div class=""> gie incoorporates some remnants of the selftest of PROJ 4.9.3 that can</div>
<div class=""> not be translated to the gie test file format. They are mostly simple</div>
<div class=""> API function tests.</div>
<div class=""><br class="">
</div>
<div class="">The above tests do a fairly good job of testing coordinate input and output of</div>
<div class="">transformations. They also directly test a smaller subset of the functions</div>
<div class="">exposed in proj_api.h and proj.h (mostly the latter). The rest of the functions</div>
<div class="">are to some extent covered indirectly by the tests in nad/ and gie/. This is</div>
<div class="">not a particularly good way of testing the functionality of these functions</div>
<div class="">since edge cases are likely to be missed.</div>
<div class=""><br class="">
</div>
<div class="">With the adoption of Catch2, as suggested by Even, we should be able to move</div>
<div class="">the tests from src/ and gie.c to this dedicated framework, preferably somewhere</div>
<div class="">in the test/ directory to keep things in a logical place separated from the</div>
<div class="">source code.</div>
<div class=""><br class="">
</div>
<div class="">I would also like to move the tests in nad/ to test/. Initially they can just</div>
<div class="">be moved more or less as is but eventually I would like to see them migrated to</div>
<div class="">the gie format. It is not totally straight-forward but with a bit of scripting</div>
<div class="">it should be possible to automate. Doing that has the unfortunate side-effect</div>
<div class="">of not testing the cs2cs application any more since gie works with API</div>
<div class="">functions. Therefore a dedicated test setup for that is needed. I regard that</div>
<div class="">as a good thing, since that can also be used with proj, cct and geod.</div>
<div class=""><br class="">
</div>
<div class="">So in summation my proposed TODO list goes something like:</div>
<div class=""><br class="">
</div>
<div class=""> 1. Move tests from src/ to the Catch2 framework</div>
<div class=""> 2. Move selftest remnants in gie.c to the Catch2 framework</div>
<div class=""> 3. Move shell-script tests in nad/ to test/</div>
<div class=""> 4. Add a test framework for the command line applications, also in test/</div>
<div class=""><br class="">
</div>
<div class="">Thoughts? What have I missed? Any suggestions on how to test the command line</div>
<div class="">applications?</div>
<div class=""><br class="">
</div>
<div class="">[0] <a href="https://proj4.org/apps/gie.html" class="">https://proj4.org/apps/gie.html</a></div>
<div class="">[1] <a href="https://www.iogp.org/bookstore/product/geospatial-integrity-of-geoscience-software-part-1-gigs-guidelines/" class="">
https://www.iogp.org/bookstore/product/geospatial-integrity-of-geoscience-software-part-1-gigs-guidelines/</a></div>
<div class=""><br class="">
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 29 May 2018, at 20:24, Kristian Evers <<a href="mailto:kreve@sdfe.dk" class="">kreve@sdfe.dk</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Mateusz,
<div class=""><br class="">
</div>
<div class="">That was a good assumption at the time :-) Things are a little bit different now.</div>
<div class="">It would seem that Catch2 is also good for C code, yes? Obviously compiled</div>
<div class="">with a C++ compiler. The documentation is a bit vague on this (it just says that</div>
<div class="">maybe it works).</div>
<div class=""><br class="">
</div>
<div class="">I am working on a proposal to improve testing of the existing code. So far I have</div>
<div class="">been working under the assumption that Catch2 would work for testing the</div>
<div class="">C code we have now. If not I will change that to cunit but it would definitely be</div>
<div class="">nice to just use one framework.</div>
<div class=""><br class="">
</div>
<div class="">/Kristian</div>
<div class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 29 May 2018, at 20:02, Mateusz Loskot <<a href="mailto:mateusz@loskot.net" class="">mateusz@loskot.net</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="auto" class="">
<div class="">Kristian, as you know, I've been considering PROJ API tests. </div>
<div dir="auto" class="">I assumed it would must be in C, so an obvious choice was cunit (as used in PostGIS for instance). I haven't arrived with anything though. </div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">It's just a comment, not recommendation to stick with C and cunit. </div>
<div dir="auto" class=""><br class="">
<div data-smartmail="gmail_signature" dir="auto" class="">Mateusz Loskot, <a href="mailto:mateusz@loskot.net" class="">
mateusz@loskot.net</a><br class="">
(Sent from mobile) </div>
<br class="">
<div class="gmail_quote" dir="auto">
<div dir="ltr" class="">On Tue, 29 May 2018, 19:26 Kristian Evers, <<a href="mailto:kreve@sdfe.dk" class="">kreve@sdfe.dk</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Even,<br class="">
<br class="">
Your timing is perfect. I was literally just typing up a long email about improving<br class="">
the test situation in PROJ right now! I was going to propose a larger restructuring<br class="">
of the tests we already have as well as adding a dedicated testing framework for<br class="">
testing API functions etc. You’ve taken care of the latter and most pressing issue.<br class="">
I’ll address the rest of my proposal at a later time I think.<br class="">
<br class="">
I have only looked briefly at catch2 but it seems good to me. I am happy that you’ve<br class="">
put your tests in test/.<br class="">
<br class="">
/Kristian<br class="">
<br class="">
> On 29 May 2018, at 19:15, Even Rouault <<a href="mailto:even.rouault@spatialys.com" target="_blank" rel="noreferrer" class="">even.rouault@spatialys.com</a>> wrote:<br class="">
> <br class="">
> Hi,<br class="">
> <br class="">
> I was researching a framework to test my new code (and that could also be used <br class="">
> to test the existing C API if needed). Currently src/gie.c has ad-hoc testing, <br class="">
> but it is really limited feature-wise: no nice error message (returns error <br class="">
> code), no way to make a testcase go on even if a test check fails, etc...<br class="">
> <br class="">
> So a dedicated framework seems a better idea, and I've found catch2 :<br class="">
> <a href="https://github.com/catchorg/Catch2/blob/master/docs/Readme.md" rel="noreferrer noreferrer" target="_blank" class="">https://github.com/catchorg/Catch2/blob/master/docs/Readme.md</a><br class="">
> <br class="">
> One of its main feature it is a header only testing framework, which means we <br class="">
> can embed it easily in proj source tree, which is practical compared to other <br class="">
> frameworks I've considered ( googletest, cppunit, etc... ).<br class="">
> The tut framework used by GDAL and GEOS<br class="">
> ( <a href="http://mrzechonek.github.io/tut-framework" rel="noreferrer noreferrer" target="_blank" class="">
http://mrzechonek.github.io/tut-framework</a> ) would also enter this category <br class="">
> of header(s) only, but it has not as much as activity than catch2.<br class="">
> There's also Boost.Test, but I was a bit afraid with the boost name in it <br class="">
> (although apparently it has a standalone mode).<br class="">
> <br class="">
> Example of tests I've just written with catch2 (just a minimalistic use of the <br class="">
> testing framework)<br class="">
> <a href="https://github.com/rouault/proj.4/blob/iso19111_ptr/test/cpp/test_crs.cpp" rel="noreferrer noreferrer" target="_blank" class="">
https://github.com/rouault/proj.4/blob/iso19111_ptr/test/cpp/test_crs.cpp</a><br class="">
> <br class="">
> I'm not particularly calling for a flame debate on the suject, just wanting to <br class="">
> mention this finding.<br class="">
> <br class="">
> Even<br class="">
> <br class="">
> -- <br class="">
> Spatialys - Geospatial professional services<br class="">
> <a href="http://www.spatialys.com/" rel="noreferrer noreferrer" target="_blank" class="">
http://www.spatialys.com</a><br class="">
> _______________________________________________<br class="">
> Proj mailing list<br class="">
> <a href="mailto:Proj@lists.maptools.org" target="_blank" rel="noreferrer" class="">
Proj@lists.maptools.org</a><br class="">
> <a href="http://lists.maptools.org/mailman/listinfo/proj" rel="noreferrer noreferrer" target="_blank" class="">
http://lists.maptools.org/mailman/listinfo/proj</a><br class="">
<br class="">
_______________________________________________<br class="">
Proj mailing list<br class="">
<a href="mailto:Proj@lists.maptools.org" target="_blank" rel="noreferrer" class="">Proj@lists.maptools.org</a><br class="">
<a href="http://lists.maptools.org/mailman/listinfo/proj" rel="noreferrer noreferrer" target="_blank" class="">http://lists.maptools.org/mailman/listinfo/proj</a></blockquote>
</div>
</div>
</div>
_______________________________________________<br class="">
Proj mailing list<br class="">
<a href="mailto:Proj@lists.maptools.org" class="">Proj@lists.maptools.org</a><br class="">
<a href="http://lists.maptools.org/mailman/listinfo/proj" class="">http://lists.maptools.org/mailman/listinfo/proj</a></div>
</blockquote>
</div>
<br class="">
</div>
</div>
_______________________________________________<br class="">
Proj mailing list<br class="">
<a href="mailto:Proj@lists.maptools.org" class="">Proj@lists.maptools.org</a><br class="">
http://lists.maptools.org/mailman/listinfo/proj</div>
</blockquote>
</div>
<br class="">
</body>
</html>