[Proj] Testing framework

[Kristian Evers]

On 29 May 2018, at 21:14, Mateusz Loskot <mateusz at loskot.net<mailto:mateusz at loskot.net>> wrote:

On Tue, 29 May 2018, 20:37 Kristian Evers, <kreve at sdfe.dk<mailto:kreve at sdfe.dk>> wrote:
So in summation my proposed TODO list goes something like:

    1. Move tests from src/ to the Catch2 framework
    2. Move selftest remnants in gie.c to the Catch2 framework
    3. Move shell-script tests in nad/ to test/
    4. Add a test framework for the command line applications, also in test/

Thoughts? What have I missed?

Although might be to early, but perhaps an outline of tests organization into suites and cases.

For example:

- test suite per API function - a sort of unit kind of tests focused on exercising single function (similar tests also could verify definitions of data structures). Eg. Call foo(a, b) with valid, invalid, boundary, random, etc. values for parameters.

- test suites of more complex test cases, more complex Arrangements preparing for actual test as well as more elaborate assertions following a testing act - I like to think of those as functional tests (function is a black box) or inter-function integration tests :) Eg.  Verifying conversion of X from CRS A to B gives expected results, depending on input valid, invalid, boundary…

These considerations need to be made at some point or another. That might as well be now.
I don’t have any particular preference. I think I’ll have to study the docs of Catch2 some more
to see what is recommended and how that translates to PROJ.

Your first point sounds like something we should do at least. We definitely need tests for each
function in the library. The second point, if I understand correctly, is more or less covered by
gie tests.

To me personally, names and definitions of particular testing do not really matter as long as there is agreement, at test case level, about what behaviour is actually being tested and asserted (sticking with single assertion per test case is a good method to reveal smelly places where it is hard to tell what is being tested)

Any suggestions on how to test the command line
applications?


Some options:
- extract gist of program logic into utility libraries (technique used in OSRMand GDAL too, AFAIR)
- run programs directly (either via OS-specific C/C++ API or Python or...) and parse/check it's output.

I think the first option is the preferred way to go. Unfortunately it also involves the most work.
It would allow us to stay within the same framework as the API tests. I think cct and gie are
fairly easy to rewire this way but the same can’t be said for proj and cs2cs. geod is somewhere
in between in difficulty.

The second option is similar to the nad/ tests and those only really work on *nix.
I fear that it would be too much work to keep it working on both Windows and *nix. There are
frameworks that does this sort of thing but I’d rather keep it simple and not introduce more
test frameworks than necessary.


Mateusz Loskot, mateusz at loskot.net<mailto:mateusz at loskot.net>
(Sent from mobile)

_______________________________________________
Proj mailing list
Proj at lists.maptools.org<mailto:Proj at lists.maptools.org>
http://lists.maptools.org/mailman/listinfo/proj

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/proj/attachments/20180529/4f3030d6/attachment.html>