[SoC] Weekly report 13: Testing framework for GRASS GIS

[Vaclav Petras]

1. What did you get done this week?

I've finally closed #2105 which partially stands behind this GSoC. I used
my last comment there to evaluate what I've done in GSoC using the
requirements which resulted from the ticket.

I've fixed report generation which was broken for the edge case when no
test is executed because of error during set up phase.

There were two days without tests when report generation failed due to the
fact that number of executed tests from one file was zero. This is fixed
and the commit causing no tests to be executed (as well as the current
number of failing tests) was reverted.

As I expected, I was not able to polish code or documentation but
documentation is pretty much ready to use and the code is almost fully
documented and has tests.

Finally, during writing this email, I decided to implement a new feature,
shell scripts can be now used as test files.

Before using shell scripts, there was currently 11 successfully running
test files and 18 failing test files, 4 successfully running testsuites and
12 failing testsuites, and 709 successfully running tests and 61 failing
tests (510 of the tests are from PyGRASS module test).

2. What do you plan on doing next week?

Next week no coding should be done for GSoC. Instead, the final evaluation
should be submitted. I also plan to prepare code for submission to Google.

If time allows, I will also write down a TODO list or Final thoughts and
create a new page for testing (and mark the GSoC one as read only).

3. Are you blocked on anything?

Not for me, but for GRASS the blocker is still the lack of tests. There is
a lot of shell scripts which can be used as test.

To make creating new tests from old ones simpler, I decided to add a
possibility to use shell scripts in place of python scripts.* in other
words, a test file can by now .py or .sh. Shell script will be executed
using `sh -e -x`. To implemented fast I had to drop some unused features
and I made MS Windows compatibility worse but this can be fixed in the
future and now we can have more tests which is more useful.

So, feel free to move shell scripts to testsuite directories (of things
they are testing) and you can even create simple tests scripts from
documentation.

However, note that writing a new tests using other things that gunittest is
strongly discouraged because only by using gunittest we are able to obtain
detailed test results and (optionally) extent testing by analyses such as
valgrind.



Wiki:
http://trac.osgeo.org/grass/wiki/GSoC/2014/TestingFrameworkForGRASS#Week12

Daily test results online:  http://fatra.cnr.ncsu.edu/grassgistests/

Ticket #2105: http://trac.osgeo.org/grass/ticket/2105

Shell scripts becoming tests:
http://trac.osgeo.org/grass/changeset/61658
http://trac.osgeo.org/grass/changeset/61653

* Alternative to that calling a shell script from a Python test file. The
only condition is that shell script must be in data directory of the
testsuite and Python file must return a return non-zero if tests failed
(e.g. using set -e).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/soc/attachments/20140815/a17d1608/attachment.html>