[GRASS-dev] G7: turning manual page examples into test cases

[Vaclav Petras]

Hi,

I've created a Python script which extracts code from HTML file and puts it
to a Bash script. It works well for our course websites but even with some
fixes it does not work for GRASS HTML manual pages. It might work well for
some modules, so you can try (both attached). Now, I have the following
ideas in case you want to dive into it on your own.

The script should be turned into library function and generalized to work
with different HTML-styles or even different markups (I think that
something like template method design pattern should work).

However, it is necessary to distinguish code from other things in the HTML.
We should look at HTML5 at that point to make the change right once we are
doing it. Basically we need to make the HTML more semantic. Perhaps some
additional classes or attributes will be also needed to mark code which
should not be executed.

Taking this idea further, we might have different languages at the page
such as Bash, Python and some platform-neutral command line. Pretty much
(visually) what we have here:

http://fatra.cnr.ncsu.edu/temporal-grass-workshop/

This would also help with solving the issue of running Bash/Shell on MS
Windows where just the Python and platform-neutral code would be executed
(which is better than nothing). Soeren's shell interpreter (or
shell-to-python translator, http://grasswiki.osgeo.org/wiki/Test_Suite) is
of course always an option if Bash will have to by used everywhere for lack
of other tests and examples. But even this requires keeping some rules for
writing examples, consider the following diff for example ("make bash
command copy-paste-able to command line"):

--- vector/v.net/v.net.html    (revision 63739)
+++ vector/v.net/v.net.html    (working copy)
@@ -170,10 +170,10 @@
 <div class="code"><pre>
 v.net points=geodetic_swwake_pts output=geodetic_swwake_pts_net \
       operation=arcs file=- << EOF
-> 1 28000 28005
-> 2 27945 27958
-> 3 27886 27897
-> EOF
+1 28000 28005
+2 27945 27958
+3 27886 27897
+EOF
 </pre></div>

Currently it seems to me that the way to go is running the tests generated
from documentation only when requested. Requested in this case means
writing a test file which would use some special API which would generate
and execute the documentation-based test. This is the way how it works for
Python doctest, too. The implementation would be on-the-fly genered bash
script executed from the Python test. Alternative is each example turned
into one test method but this wouldn't work because the order of method
execution is not preserved (by Python unittest). So, one HTML file must be
one bash script.

The approach above allows module-specific filtering of examples to test and
clear integration of this tests into module's testsuite. The obvious
disadvantaged of course is that one has to provide all the test files for
each module but the alternative would actually be large number of failing
tests due to strange (not necessarily wrong) documentation and a lot of
failing tests by default is always wrong.

The approach with manual addition of tests has actually one great feature.
Test can specify which maps should be checked against which references
which can be provided (and checked) in the same way as for all the other
scripts.

As this would not work for all modules unless explicitly requested, there
is still some place for some kind of monkey testing which would be based
just on the interface of the module. However, I would prefer not to have
these tests which might be failing by default together with the standard
hand-written tests. Anyway, there sometimes the number of errors might be
lowered by just requesting the modules to fail properly with an error
message (rather than segfaulting). Another interesting approach is to
define the interface of modules so well that the test data can be generated
in the way which in fact fulfills all the advanced requirements the module
can have (e.g. raster values in range 0-255. This of course goes far into
data provenance and software metadata topics.

Vaclav
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20141228/80763975/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: script_to_extract_tests_from_course.py
Type: text/x-python
Size: 5071 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20141228/80763975/attachment.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: script_to_extract_tests_from_documentation.py
Type: text/x-python
Size: 5071 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20141228/80763975/attachment-0001.py>