[GRASS-dev] Shell scripts for tests

Panagiotis Mavrogiorgos pmav99 at gmail.com
Fri Mar 29 05:27:15 PDT 2019


Hello all,

The GRASS testsuite contains several tests that have been written as shell
scripts. If I am not mistaken, those can be identified by the "unknown"
percentages in the testreport (example
<http://fatra.cnr.ncsu.edu/grassgistests/reports_for_date-2019-03-26-07-00/report_for_nc_spm_full_v2alpha_nc/testfiles.html>
).

IMHV this practise is causing several issues:

1. The testrunner seems to be skipping/omitting the majority of the
available tests. More specifically, the testrunner currently runs 27 shell
scripts, while it seems that there are 95+ tests
<https://gist.github.com/pmav99/8e016e7c044555cef4e37d685a6622b6> written
as shell scripts. I suspect that the testrunner is only matching
"testsuite" directories while skipping directories named "tests",
"test_suite" etc.

1a. There should be uniformity in the naming conventions of test
directories and test files.
1b. Needless to say, we need to make sure that the testrunner runs all the
available tests. I guess that if 1a is taken care off then 1b will have
been resolved too.

2. I only checked a bunch of the shell scripts but I am under the
impression that majority of them (if not all!) are not real tests but just
shell scripts that prove that the code runs while not really asserting that
the code is actually correct. Of course, this is better than nothing, but
it is not good enough.

2a. Ideally and as a mid-term goal these tests should be converted to
proper python tests
2b. In the meantime, and if it is not already being used, we should try to
make sure that the shells scripts are using "strict mode" so that errors do
propagate:
http://redsymbol.net/articles/unofficial-bash-strict-mode/
https://disconnected.systems/blog/another-bash-strict-mode/
2c. Also in the meantime, and in order to make testreports more useful, we
could also add a single python test file next to each shell script which
will contain something similar to this:

subprocess.check_out(shlex.split("bash test_foo.sh"))

This way, we would have reports with proper count of testsuites, failures
etc.

3. Finally, IMHV there should also be an official policy/requirement, that
new code/modules cannot be accepted unless they have proper and
comprehensive tests written in python.

with kind regards,
Panos
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20190329/7d94b745/attachment.html>


More information about the grass-dev mailing list