[GRASS-dev] [GRASS GIS] #2105: Missing guidelines for testing

[GRASS GIS]

#2105: Missing guidelines for testing
-------------------------+--------------------------------------------------
  Reporter:  wenzeslaus  |       Owner:  grass-dev@…                               
      Type:  task        |      Status:  closed                                    
  Priority:  normal      |   Milestone:  7.1.0                                     
 Component:  Tests       |     Version:  svn-trunk                                 
Resolution:  fixed       |    Keywords:  testing, tests, PyUnit, doctest, testsuite
  Platform:  All         |         Cpu:  Unspecified                               
-------------------------+--------------------------------------------------
Changes (by wenzeslaus):

  * status:  new => closed
  * resolution:  => fixed
  * milestone:  7.0.0 => 7.1.0


Comment:

 There was a GSoC:
  * wiki:GSoC/2014/TestingFrameworkForGRASS

 And documentation is now available:
  * http://grass.osgeo.org/grass71/manuals/libpython/gunittest_testing.html
  * source:grass/trunk/lib/python/docs/src/gunittest_testing.rst?rev=61572

 This ticket was asking for documentation but new possible tool for testing
 was discussed too, so here are the points from comment:7 applied to
 current state of testing framework.

 Python libraries Python:

  * `gunittest`
  * `unittest` use its assert methods through `gunittest`
  * `doctest` use only for documentation purposes and then test this
 documentation (i.e. do complex tests using `gunittest`), there is a
 special way how to import tests

 Functions in Python modules:

  * doctest should at least partially work, currently best to invoke
 directly (without testing framework)
  * to standardize/promote `--test` parameter for GRASS modules might be
 useful here
  * others under investigation

 Interface of C/C++ libraries:

  * test using special (test) GRASS modules, invoke these separately for
 benchmark and as standard GRASS modules through `gunittest`
  * test as Python libraries through ctypes

 Internal (static, private, ...) functions in C/C++ libraries:

  * N/A

 Functions in C/C++ modules:

  * use special (test) GRASS modules as for C/C++ libraries
  * to standardize/promote `--test` parameter for GRASS modules might be
 useful here
  * static and private cannot be tested

 C/C++ modules and Python modules:

  * `gunittest`

 There are five other main areas of the testing topic:

  * recursive running of tests
   * e.g., `python -m grass.gunittest.main --location nc_spm_grass7
 --location-type nc`
   * `make test` not implemented
  * reports/outputs
   * textual output on stdout
   * HTML
   * parsable key-value files with summaries
   * XML or database not implemented
   * automatic and online running supported using custom scripts and
 documented for custom server
    * tests currently runs every day
    * no online testing services explicitly supported or tested
  * supporting functionality especially that for comparing (maps, numbers)
 implemented or using `unittest`
  * data for testing is a complex issue but most important are random data,
 data included in tests and NC SPM sample location, other can be used too
 but this was not much tested yet

 Closing this ticket as more then fixed, create new tickets for further
 issues with testing (use component Tests).

-- 
Ticket URL: <http://trac.osgeo.org/grass/ticket/2105#comment:9>
GRASS GIS <http://grass.osgeo.org>