[SoC] Weekly report 10: Testing framework for GRASS GIS

[Vaclav Petras]

1. What did you get done this week?

I significantly improved the test file page which is the central page of
the report in the sense that there you can get all the details. I also
added details about successful and failed (individual) tests to main index
page (the one with links to all test files). Additionally, I created
different set of pages which sorts the test files from the point of view of
testsuite directories which might be the primary way of browsing tests
since it is aligned with how tests are included in the source code. The
page for a given testsuite also contains approximate information about
authors of tests and authors of tested code.

Each test file has now a key-value file which contains information about
success, number of tests, and other information. These files can and are
collected to obtain summaries. The key-value format was selected mainly
because of its usage in GRASS GIS (otherwise, for example Python
documentation for inputs and outputs recommends JSON [1]).

I believe that I designed report classes in the way that they can be
extended but already I can see some room for refactoring or at least for
better organization. This might be a topic for week 13.

I did not created any XML output. The situation seems unclear and a lot of
transformations are done to get XML working with different tools. On the
other hand, with some tools even trivial XML with numbers (test statistics)
should work and this should be easy to create from key-value files or
directly. Also there is a unittest-xml-reporting package [2] for Python
`unittest` which should create XML in the style of Ant/JUnit. In theory
this should work directly with `gunittest`. I was not considering database
output much but I think that it is not needed now and it can be created
from key-value files if necessary.

A also did not focus on CSS of the HTML reports. They are readable but
that's all. Some CSS might be added in the future.

There is currently 20 successfully running test files and 5 failing test
files, 9 successfully running testsuites and 4 failing testsuites, and 686
successfully running tests and 7 failing tests (533 of the tests are from
PyGRASS module test).

I also added some tests for `r.mapcalc` `rand()` function and new behavior
with seed in r61355.

See the project wiki page for screenshots and more detailed description of
the test report or obtain one yourself running:

# in the grass souce code directory, in grass session
python -m grass.gunittest.main gisdbasepath nc_smp_location_name nc
# open file "testreport/index.html" or "testreport/testsuites.html"

2. What do you plan on doing next week?

I plan to rewrite some of the existing tests using `gunittest` or write new
ones. I already started to write something for `r.slope.aspect` and
`r.mapcalc`.

If time allows, I would like to add some key-value summary files to
reports, include some test metadata such as GRASS location or architecture,
and also introduce more options to report generation (currently, it cannot
be configured).

Additionally, It would be nice to have all path in files shortened so that
they don't include `dist...` or full path to the file and also it would be
nice to have an example of running the tests automatically in some trivial
form.

3. Are you blocked on anything?

I think that some CSS and JavaScript can improve the reports significantly
but I did not included any nice design to HTML pages because it would cost
me too much time (and it was not part of the GSoC plan anyway).

Wiki with screenshots:
http://trac.osgeo.org/grass/wiki/GSoC/2014/TestingFrameworkForGRASS#Week10

[1]
https://docs.python.org/2/tutorial/inputoutput.html#saving-structured-data-with-json
[2] https://github.com/xmlrunner/unittest-xml-reporting
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/soc/attachments/20140726/06a349bb/attachment.html>