<div dir="ltr">Hi,<div class="gmail_extra"><br><div class="gmail_quote">2016-10-05 15:20 GMT+02:00 Moritz Lennert <span dir="ltr"><<a href="mailto:mlennert@club.worldonline.be" target="_blank">mlennert@club.worldonline.be</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-">On 05/10/16 14:24, Sören Gebbert wrote:<br>
</span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-">
Hi,<br>
<br>
2016-10-05 10:20 GMT+02:00 Moritz Lennert <<a href="mailto:mlennert@club.worldonline.be" target="_blank">mlennert@club.worldonline.be</a><br></span>
<mailto:<a href="mailto:mlennert@club.worldonline.be" target="_blank">mlennert@club.worldonl<wbr>ine.be</a>>>:<span class="gmail-"><br>
<br>
[sent this from the wrong address, so it didn't get through to the list]<br>
<br>
<br>
-------- Message d'origine --------<br>
Envoyé : 5 octobre 2016 00:41:20 GMT+02:00<br>
<br>
<br>
<br>
Le 4 octobre 2016 22:55:35 GMT+02:00, "Anna Petrášová"<br></span>
<<a href="mailto:kratochanna@gmail.com" target="_blank">kratochanna@gmail.com</a> <mailto:<a href="mailto:kratochanna@gmail.com" target="_blank">kratochanna@gmail.com</a>><wbr>> a écrit :<span class="gmail-"><br>
>On Tue, Oct 4, 2016 at 4:22 PM, Markus Metz<br>
><<a href="mailto:markus.metz.giswork@gmail.com" target="_blank">markus.metz.giswork@gmail.co<wbr>m</a><br></span><span class="gmail-">
<mailto:<a href="mailto:markus.metz.giswork@gmail.com" target="_blank">markus.metz.giswork@gm<wbr>ail.com</a>>> wrote:<br>
>> On Tue, Oct 4, 2016 at 5:42 PM, Sören Gebbert<br>
>> <<a href="mailto:soerengebbert@googlemail.com" target="_blank">soerengebbert@googlemail.com</a><br></span><div><div class="gmail-h5">
<mailto:<a href="mailto:soerengebbert@googlemail.com" target="_blank">soerengebbert@googlema<wbr>il.com</a>>> wrote:<br>
>>> Hi,<br>
>>>><br>
>>>><br>
>>>> ><br>
>>>> > You are very welcome to write the missing tests for core modules.<br>
>>>> ><br>
>>>> > However, i don't understand the argument that because many core<br>
>modules<br>
>>>> > have<br>
>>>> > no tests, therefore new modules don't need them. If developers of<br>
>addon<br>
>>>> > module are serious about the attempt to make their modules usable<br>
>and<br>
>>>> > maintainable for others, then they have to implement tests. Its<br>
>and<br>
>>>> > integral<br>
>>>> > part of the development process and GRASS has a beautiful test<br>
>>>> > environment<br>
>>>> > hat makes writing tests easy. Tests and documentation are part of<br>
>coding<br>
>>>> > and<br>
>>>> > not something special. I don't think this is a hard requirement.<br>
>>>> ><br>
>>>> > There is a nice statement that is not far from the truth:<br>
>Untested code<br>
>>>> > is<br>
>>>> > broken code.<br>
>>>><br>
>>>> these gunittests only test if a module output stays the same. This<br>
>>><br>
>>><br>
>>> This is simply wrong, please read the gunittest documentation.<br>
>><br>
>> but then why does<br>
>>><br>
>>> The gunittest for the v.stream.order addon is an example how its<br>
>done:<br>
>>><br>
><a href="https://trac.osgeo.org/grass/browser/grass-addons/grass7/vector/v.stream.order/testsuite/test_stream_order.py" rel="noreferrer" target="_blank">https://trac.osgeo.org/grass/<wbr>browser/grass-addons/grass7/ve<wbr>ctor/v.stream.order/testsuite/<wbr>test_stream_order.py</a><br>
<<a href="https://trac.osgeo.org/grass/browser/grass-addons/grass7/vector/v.stream.order/testsuite/test_stream_order.py" rel="noreferrer" target="_blank">https://trac.osgeo.org/grass/<wbr>browser/grass-addons/grass7/ve<wbr>ctor/v.stream.order/testsuite/<wbr>test_stream_order.py</a>><br>
>><br>
>> assume certain order numbers for features 4 and 7? What if these<br>
>order<br>
>> numbers are wrong?<br>
>><br>
>> Recently I fixed bugs in r.stream.order, related to stream length<br>
>> calculations which are in turn used to determine stream orders. The<br>
>> gunittest did not pick up 1) the bugs, 2) the bug fixes.<br>
>><br>
>>><br>
>>> You can write gunittests that will test every flag, every option,<br>
>their<br>
>>> combination and any output of a module. I have implemented plenty of<br>
>tests,<br>
>>> that check for correct error handling. Writing tests is effort, but<br>
>you have<br>
>>> to do it anyway. Why not implementing a gunittest for every feature<br>
>while<br>
>>> developing the module?<br>
>>>><br>
>>>><br>
>>>> My guess for the r.stream.* modules is at least 40 man hours of<br>
>>>> testing to make sure they work correctly. That includes evaluation<br>
>of<br>
>>>> float usage, handling of NULL data, comparison of results with and<br>
>>>> without the -m flag. Testing should be done with both high-res<br>
>(LIDAR)<br>
>>>> and low-res (e.g. SRTM) DEMs.<br>
>>><br>
>>><br>
>>> Tests can be performed on artificial data that tests all aspects of<br>
>the<br>
>>> algorithm. Tests that show the correctness of the algorithm for<br>
>specific<br>
>>> small cases should be preferred. However, large data should not be<br>
>an<br>
>>> obstacle to write a test.<br>
>><br>
>> I agree, for tests during development, not for gunittests.<br>
>><br>
>> From the examples I read, gunittests expect a specific output. If the<br>
>> expected output (obtained with an assumed correct version of the<br>
>> module) is wrong, the gunittest is bogus. gunittests are ok to make<br>
>> sure the output does not change, but not ok to make sure the output<br>
>is<br>
>> correct. Two random examples are r.stream.order and r.univar.<br>
><br>
><br>
>I am not sure why are we discussing this, it's pretty obvious that<br>
>gunittests can serve to a) test inputs/outputs b) catch changes in<br>
>results (whether correct or incorrect) c) test correctness of results.<br>
>It just depends how you write them, and yes, for some modules c) is<br>
>more difficult to implement than for others.<br>
<br>
<br>
Well, I agree with Markus that unittests are not a panacea and that<br>
we should not fall into the trap of thinking that these tests will<br>
guarantee that the results of our modules are correct.<br>
<br>
<br>
Then i live in a parallel universe. Simple question: How do you test<br>
your software? How do you assure the correct functionality of your<br>
software? Why is it impossible to implement your approach of testing in<br>
a dedicated gunittest? How do you assure software quality, if you don't<br>
provide tools so that other developers are able to test your software<br>
for correctness? Regression tests are not possible then, because the<br>
effect of changes in the core libraries can not be easily detected in<br>
modules without tests.<br>
</div></div></blockquote>
<br>
<br>
Please note that I was speaking about unit tests, here. I don't know how efficient our testing framework is for integration testing ? Maybe we also need to be clearer about what we understand by tests during such discussions ?<br>
<br>
Good discussion, though ! :-)</blockquote><div><br></div><div>I would like to put the GRASS test framework into perspective, since i think that its capabilities are not well known. <br></div><div>The gunittest framework is not about unit tests. It was designed to test all aspects of the GRASS development. This framework allows you to:</div><div><br></div><div>* Implement unit tests for the Python libraries, their mthods and classes</div><div>* Implement and run doctests in the source code of the Python libraries</div><div>* Run integration tests for all modules, checking correct output for almost all datatypes in GRASS (raster, vector, 3D raster, space-time datasets, categories, color definitions, stdout, ...). Module tests are IMHO integration tests, since module make use of different library methods and classes and combine them.</div><div>* Run C-library tests as unit and integration tests. C-library unit and integration tests can either be implemented in C or via ctypes in Python</div><div>* Run tests on library level, module level or for all libraries and modules in the whole GRASS source tree using a single command, ... </div><div>* Perform regression tests in dedicated test locations, autimatically triggered by a cronjob or a commit</div><div>* The framework allows you to run all library unit tests, before module integration tests are performed</div><div>* It creates temporary mapsest to run without problems in production locations</div><div>* It logs all tests in detail and generates easy to inspect HTML output at runtime, so you can check the progress of the tests and its gradually available results</div><div>* It allows on the fly mapset creation and deletion</div><div>* It supports temporary region environments</div><div>* It support user defined test data for input generation and output validation</div><div><br></div><div>These capabilities allow a wide range of tests to be created, covering most aspects of the GRASS development, with the exception of the GUI. </div><div>So please no excuses that the gunittest framework is not capable of implementing a test that is required to assure module correctness.</div><div><br></div><div>Best regards</div><div>Soeren</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-"><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Can you explain to me why the developers of the sophisticated software<br>
system VTK [1] implement unit and integration tests for all software<br>
components to assure the correct functionality of the framework? They<br>
didn't saw the trap? They are delusional to think that tests assure<br>
software quality?<br>
<br>
Why is test driven development [2] an integral part of agile software<br>
development approaches like scrum or extreme programming? They didn't<br>
saw the trap? They are delusional to think that tests assure software<br>
quality?<br>
<br>
[1] <a href="http://www.vtk.org/overview/" rel="noreferrer" target="_blank">http://www.vtk.org/overview/</a><br>
[2] <a href="https://en.wikipedia.org/wiki/Test-driven_development" rel="noreferrer" target="_blank">https://en.wikipedia.org/wiki/<wbr>Test-driven_development</a><br>
<br>
<br>
However, I do agree that these tests are useful in detecting if any<br>
changes to the code change the output, thus raising a flag that the<br>
developer has to at least take into account.<br>
<br>
I'll try to write some tests for the OBIA tools when I find the<br>
time, although I do agree with Markus that it wouldn't be useful to<br>
try to write tests that would cover each and every possible corner<br>
case...<br>
<br>
<br>
Why is it "not useful" to write tests for all cases the software is<br>
dedicated to solve? It is indeed a lot of effort, but it is useful.<br>
</blockquote>
<br></span>
I would say the question is rather, first, whether it is at all possible, and, second, that maybe by thinking that it is, we are too confident in our tests providing information that they really aren't trying to provide.<br>
<br>
But I'm no expert whatsoever, on the topic (I am not a computer scientist, just a scientist programming some tools with my very limited capabilities), so I don't want to stretch this discussion out. I do recommend reading this, though:<br>
<a href="http://www.rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf" rel="noreferrer" target="_blank">http://www.rbcs-us.com/documen<wbr>ts/Why-Most-Unit-Testing-is-<wbr>Waste.pdf</a><br>
<br>
I also like the table close to the top of<br>
<br>
<a href="http://blog.stevensanderson.com/2009/08/24/writing-great-unit-tests-best-and-worst-practises/" rel="noreferrer" target="_blank">http://blog.stevensanderson.co<wbr>m/2009/08/24/writing-great-uni<wbr>t-tests-best-and-worst-practis<wbr>es/</a><br>
<br>
(attached as image)<br>
<br>
And let's remember that this all started as the question of what should be required for a module to move from addons to core. The question, therefore, is to find the right balance between necessary effort and our desire to offer functionality to users. This also raises the question of why it would be better for a given module to be in core, rather than in extensions. We could also imagine the opposite direction, i.e. move modules from core to extensions to lighten the work load of maintaining core, while still offering the same functionalities.<br>
<br>
IMHO, the largest advantage of having a module in core is that when someone changes internal library APIs, then generally they check all of core and modify what needs to, but this is not necessarily the case for extensions...<br>
<br>
Maybe we should ask the users of whether this distinction between modules and core and extensions is really relevant for them, or whether most are perfectly happy to just install extensions.<span class="gmail-HOEnZb"><font color="#888888"><br>
<br>
Moritz<br>
</font></span></blockquote></div><br></div></div>