[GRASS-dev] [GRASS GIS] #2105: Missing guidelines for testing

[GRASS GIS]

#2105: Missing guidelines for testing
----------------------------+-----------------------------------------------
 Reporter:  wenzeslaus      |       Owner:  grass-dev@…              
     Type:  task            |      Status:  new                      
 Priority:  normal          |   Milestone:  7.0.0                    
Component:  Tests           |     Version:  svn-trunk                
 Keywords:  testing, tests  |    Platform:  Unspecified              
      Cpu:  Unspecified     |  
----------------------------+-----------------------------------------------

Comment(by wenzeslaus):

 Replying to [comment:1 huhabla]:
 > Replying to [ticket:2105 wenzeslaus]:
 > >
 > > There is a page about the test framework we wish to have:
 > >  * http://grasswiki.osgeo.org/wiki/Test_Suite
 > > But since it does not exists the page creates just confusion.
 >
 > This page describes in very detail how the testsuite can be implemented
 for modules and libraries. Python doctests and how to implement automated
 testing in the make system are not covered. It includes a guideline howto
 implement tests for modules based on shell scripts. Hence it is unclear to
 me how this leads to confusion?
 >
 > However, i am all for a complete new test suite design. I would suggest
 to implement all module and Python library tests in Python, so that these
 tests can be executed independently from the OS or the used command line
 interpreter, with or without the make system.
 >

 For me it seems that it is not worth to build our own tests system and
 bash-like language interpreter (as proposed on the wiki page). I vote for
 writing module tests directly in Python. From my experience I was writing
 module test in bash and it is easier at the beginning but if you want to
 do something more special than calling a module and check the return code
 bash-like language is not sufficient. Moreover, we want cross platform
 tests and I think we don't want to implement bash-like syntax parser and
 interpreter for variables, if-statements, for loops in Python, although it
 sounds interesting. When writing directly in Python, the tests dependents
 on GRASS or Python process calling capabilities but they would depend
 anyway if the interpreter would be build. The only advantage in writing in
 bash-like syntax is that it can run as bash/sh... script, but Python
 script is great too, isn't it?

 For the test of the library code it is clear that it must be done in
 Python as you say. I suppose that it would be tested using ctypes, am I
 right? The problem I can see is that I would like to test also `static`
 functions and this is not possible I guess.

 > The next big issue is how to automatically test the GUI?
 >
 There are currently several ways we are using. You should split the GUI to
 some logic/representation (MVC etc.), so for some functions you can just
 use standard tests, we are using `doctest`.

 Sometimes, logic requires you to write some pseudo-GUI (e.g. text
 outputting classes, or generally test code) to be able to test it. This is
 usually combined with the approach of tests which run as standard script,
 produce some output and this is compared manually or by some diff. So far,
 no testing framework such as `unitest` was used for this.

 Than there are completely non-automatic tests which finally contain the
 GUI components/classes. You just run a script/module with some parameters
 and you try how the window behaves without the burden of running the whole
 g.gui and setting things manually. The tandalone GUI modules (`g.gui.*`)
 are in this category and then there are some Python files which enables
 you to run them and they show some GUI which is not useful for users but
 just for testing. The problem of the last ones is that they are not well
 maintained because they are not listed or documented anywhere and are
 harder to use comparing to `doctest`.

 For what I heard the automated GUI tests are not worth to do. But I'm open
 to any ideas about is as well as about improvements of currently used
 techniques.

 > I would like to mentor this as a GSoC project in 2014. I have some
 experience with test suites ...
 > And i have the experience that nobody cares to implement module or
 library tests ... .

 GSoC project for tests of any kind would be great.

 It seems to me that the server-side automatic test are important. There
 are even some services for this. But this goes really far from the
 original ticket problem.

 ''And by the way, Trac wiki synatax does not support our tradition of
 references done through `[1]`. The trac syntax is becoming an issue for
 us, I would say.''

-- 
Ticket URL: <https://trac.osgeo.org/grass/ticket/2105#comment:3>
GRASS GIS <http://grass.osgeo.org>