[GRASS-dev] [GRASS GIS] #2105: Missing guidelines for testing

[GRASS GIS]

#2105: Missing guidelines for testing
--------------------------------------------------------+-------------------
 Reporter:  wenzeslaus                                  |       Owner:  grass-dev@…              
     Type:  task                                        |      Status:  new                      
 Priority:  normal                                      |   Milestone:  7.0.0                    
Component:  Tests                                       |     Version:  svn-trunk                
 Keywords:  testing, tests, PyUnit, doctest, testsuite  |    Platform:  All                      
      Cpu:  Unspecified                                 |  
--------------------------------------------------------+-------------------

Comment(by wenzeslaus):

 I agree with [comment:4 huhabla (comment 4)] that the tests should be
 primarily in Python. We actually already did the decision: GRASS modules
 which are scripts are in Python. The reasons for this were probably stated
 at that point but let's recall them:
  * it's easier than C/C++ to write and run and more flexible if necessary
 when doing special things (might apply to tests)
  * once you need to write an if statement, Python is simpler to write and
 read than Bash (and same applies to strings and `sed`, `awk`, ...)
  * although most of the GRASS developers and power users writes in Bash a
 lot, the common language for most of the developers is Python (today and
 probably in the future too)
  * Python is a general purpose language (both scripts and complicated
 applications are possible)

 And as for the additional complexity of Python comparing to Bash
 (comment:5), the module test is not only a call of the module which would
 be easier to write in Bash. If the module returned 0 or not, is not
 enough. It is usually needed to set region, prepare data, call the module
 and test the output. Which is actually the same as it is said about
 writing unit tests in general: The test for the function can be longer
 than the function itself. So, I don't expect tests to be simple, although
 I wish them to be.

 However, I would like to point out that we have quite different ''units''
 to test, basically modules, C function and Python functions. For some of
 them the choices are unclear but for Python libraries, Python testing is
 just the best choice. There are even `unittest` and `doctest` in the
 standard distribution (`doctest` is compatible with `unittest`), so we
 don't need to use something external. No need to write some new things.

 C/C++, on the other hand, requires some more thinking and unfortunately,
 it is hard to follow KISS principle, because it is not simple with them.
 There is no standard or build-in way of testing, so writing some special
 code is inevitable. You can write some testing programs but sooner or
 later you need some special code to compare results, manage tests, etc.

 Besides some classic ''unit test'' frameworks for C/C++ there could be
 something which uses Doxygen to extract tests from doc (similarly to
 Python doctest) and also something which uses abilities of LLVM/clang (JIT
 compiler, C++ interpreter) but I have never searched for it. However, we
 would have probably a lot of problems with using some 3rd party solution
 or, so all these options are not usable. See also
 [http://hub.qgis.org/projects/quantum-gis/wiki/Developers_Manual#Unit-
 Testing QGIS Unit Testing] which uses Qt and CTest testing frameworks.

 Especially for the C/C++ modules, instead of writing testing programs to
 test their internal functions, we can also implement something like a
 `--test` option which would be available only when certain `#define` would
 be active (using make/configure) and testing functions would be directly
 in the module and dependent on the `#define` value. Common testing
 functions can be in some library.

 Here is the summary of options I see as doable, probable and acceptable.

 Python libraries Python:
  * `unittest` and `doctest`

 Functions in Python modules:
  * `unittest` and `doctest`

 Interface of C/C++ libraries:
  * interface by `unittest` and `doctest` through `ctypes`

 Internal (`static`, `private`, ...) functions in C/C++ libraries:
  * testing programs
  * generate `ctypes` interface and then `unittest` and `doctest`

 Functions in C/C++ modules:
  * testing programs
  * generate `ctypes` interface and then `unittest` and `doctest`
  * `--test` parameter for module

 C/C++ modules and Python modules:
  * `unittest` and `doctest`
  * Bash/shell scripts

 I don't know if the `ctypes` generation for testing purposes would be
 possible but if yes, it seems to be a good option. However, you still
 cannot test ``static`` functions in that way.

 There are five other main areas of the testing topic:
  * recursive running of tests (`make test`)
  * reports/outputs
   * output after `make test` is necessary
   * HTML or parsable text or XML would be really useful for online
 presentation
  * automatic/online running
   * automatic running is not necessary but important (run of the test
 after commit would be even better)
   * there are some online testing services which we might use or be
 compatible with
  * supporting functionality especially that for comparing (maps, numbers)
  * data for testing
   * from my experience random are not enough
   * minimal NC sounds good
   * some artificial data would be good
   * no idea about LL locations, sometimes important but this might apply
 to other projections, too
 I cannot tell about any of these topics anything more now but they should
 be discussed as well as the way of testing itself.

 But back to the ticket original question. ''What is the way to write tests
 now?''
  * For Python, it seems that you should use `doctest` or `unittest` and
 invoke it in whatever way you want.
  * For modules, you can write test Python or Bash scripts both are
 currently legal to use but Python might be the only option in the future
 (have a look to the comment:5 for some possible ways of testing
 computational results).
  * For C/C++ library functions and module functions, the current way is to
 write special program which will test the functions, set up a build by
 copy and paste from existing tests (good luck).

 However, question actually was ''Where do I find the information about
 tests?'' We must say that the best (and only) place is now this ticket
 (#2105).

-- 
Ticket URL: <http://trac.osgeo.org/grass/ticket/2105#comment:7>
GRASS GIS <http://grass.osgeo.org>