[gdal-dev] GDAL testing

[Mateusz Loskot]

On 17 October 2015 at 06:14, Kurt Schwehr <schwehr at gmail.com> wrote:
> What we have with autotest is super important.

No doubt.

> I've been trying to figure out how we take the first 17
> years of knowledge built up in GDAL and make keep going forward over the
> next couple decades.

A very interesting question.

> What will help people contribute to GDAL in the future?

I assume, you mean to identify and remove objective barriers at levels of
source code, build configuration, test suite, documentation.

> However, above everything else, we can't give up the coverage that we
> currently have.

No doubt.

>  It's a lot and it's taken a huge effort to get to this
> point, but I think it is not nearly enough.  I think that means that
> autotest in it's current form will be with us for the foreseeable future.
> Nor is it enough to just get the Coverity issue count to 0.

Right.
I think it is important to clarify the term coverage here.
It is meant in terms of coverity, static analysis, but not necessarily as
code coverage vs our test suite. I hope I get it right myself too.

> On Fri, Oct 16, 2015 at 6:01 AM, Mateusz Loskot <mateusz at loskot.net> wrote:
>> Kurt Schwehr-2 wrote:
>> > For my production work, I'm not able to use the autotest python code
>> > because of its non-unittest architecture.
>>
>> Could you elaborate more on what properties exactly are missing
>> from the current autotest suite?
>> IOW, what kind of gap autotest2 fills.
>
> Things that are missing from autotest python code:
>
> - By not being python unittest/unittest2/mock based (...)
> - It has a syntax that is unfamiliar to the average python coder.  The junit
> style is super common, so there are more folks available who really know
> unittest based tools.

I agree, it is a barrier for potential contributions who know Python
native testing package(s).
Since the autotest package is a complex piece of software, but not based on
standard Python tools, it does not allow knowledge transfer/re-use.

> - There is a lot of well developed utility in the the unittest/unittest2
> infrastructure that is very robust

I second that.

> - Isolated tests.  There are a lot of tests that cascade.  That makes
> running tests in parallel much more difficult

I wasn't aware of cascading requirement or inter-dependencies in current tests.
It would be good to solve such issues, indeed.

> - The python mocks library is pretty great for making tests of things like
> databases. (...)

I second that too.
So, you are considering to use fakes for infrastructure.
Apart from simplifying testing environment, speeding things up,
it would also enable development of proper unit tests.

> - unittest and other related tools have nice ways to allow running subsets
> of tests

Yes. I'm huge fan of nose and its tests tagging feature.

> It's worth taking a look at some examples.  e.g. Fiona's tests

I have and I do.

I'd add couple of points to your list:
- test single thing per test method, it means one assertion per test method
- make test methods 5 lines long max
- use long descriptive names for test methods, not Python comments!
(ogr_csv_5 and alike is not a good name)

(Note, I don't say test case, as test case is a class in unittest, so better
speak of test methods.)

Added value is that such tests become great set of examples and tutorials.


>> Kurt Schwehr-2 wrote
>> > So... I started creating python
>> > unittest and C++ gunit based tests.
>>
>> Since you mention unittest, then gmock too?
>
> Yes, gunit and gmock are usually used together.

Good idea.

>> Kurt Schwehr-2 wrote
>> > The tests are more focused on test isolation than autotest.  This allows
>> > for a lot more parallelism in testing.
>>
>> Isolation at what level?
>> I suppose, you mean test case isolation, but not isolation of code/unit
>> under test.
>
>
> Both types of isolation would be nice.  GDAL isn't designed for isolated
> component testing of the drivers.

You're right.

> I've mostly been able to do isolated testing on ports/cpl_*.

Yes, CPL should be nice for unit testing with fakes.

> The biggest isolation issue is between each test
> function/method. Using the same filenames or using global variables and
> having one test create a file and then follow on tests use that file.

Yes, that is an issue indeed.

>> Kurt Schwehr-2 wrote
>> > Here are some samples:
>> >
>> > C++ tests use  C++11, gunit, google logging, gflags:  (Hoping for C++14
>> > soon.. e.g. make_unique)
>> > - autotest2/cpp/port/cpl_conv_test.cc
>> > <https://gist.github.com/schwehr/13137d826763763fb031> (Yes, this
>> > is massively boring code)
>>
>> AFAIS, architecture of tests won't change much, typically, test cases will
>> target functional modules (drivers).
>> Or, do you plan for proper unit tests targeting individual
>> interfaces/classes in internal implementation of drivers, etc.?
>
>
> Looking at drivers, I think the level and type of testing that is
> appropriate varies quite a bit.  Some drivers are super simple and some are
> very complex.  In general, extensive testing of internals isn't super
> important.  What we do need a ton more of is testing the failure paths.
> e.g. I took AFL and ran it against ogr2ogr and a custom little loader
> program to read geojson.  GDAL code should never crash or hang, but it most
> definitely does.  We need to turn those input files into test cases.

I see.
I doubt it is practical to try to mock anything in drivers.
The gmock could be used to test some gcore interfaces though.

>> Kurt Schwehr-2 wrote
>> > I'm (mostly) following Google's style guides.
>> > All C++ should be formatted with "clang-format --style=Google"
>>
>> Unless it is not planned to include autotest2 into GDAL source code,
>> shouldn't it follow GDAL style?
>
> Not necessarily.   I'm working for Google and starting from scratch with
> Google engineers doing the reviews, so I'm using the Google Style Guides.

Ok, I understand.

> The style for GDAL isn't what users of the library are likely to do.

I can also imagine, Python users may expect Pythonic way of
writing Python tests for GDAL.

> A part of what I'm trying to do with autotest2 is also give examples of what I
> think are good ways to use GDAL.  People are likely going to use C++11/14
> and Hungarian-ish notation is not something I've seen in any other project
> I've worked on.

It is hard to disagree.

> The Google style guide is pretty well flushed out, where as
> RFC 8 for GDAL is very minimalistic.  I prefer to work in a space where I
> can just follow.

Right. Even if autotest2 makes it into GDAL codebase, it would be developed as
separate module, so not following GDAL style legacy shouldn't be an issue.
On the contrary, it could promote some more C++11/14 idiomatic code leaks into
GDAL core base...while GDAL is gradually dropping support for
non-C++11/14-compliant compilers :P

>> Kurt Schwehr-2 wrote
>> > Would like to eventually do (unsorted):
>> > - Fuzz testing, ASAN/MSAN/TSAN/Valgrind/Heap checks  (I've done some
>> > MSAN
>> > &
>> > heap checkers by hand)
>> > - Performance testing - time and memory usage
>>
>> IMHO, those belong to separate suites.
>>
>
> Why separate the memory and thread tests?

I meant, separate from unit or integration testing.

> If you write really solid unittests in C++, you can flip on or off those modes.
> As long as your unittests properly cleanup, then they are great code to use for
> ASAN/MSAN/etc.

My point was this:
You don't write unit or integration tests with performance as a goal.
Technically, you can, but that imposes too many goals for your unit tests -
while a good  unit test should tests single thing, assert single aspect, without
killing too may ducks with one shot.

Chances are, we understand the term "unit test" differently.

>> - Test the C API at the C level
>>
>> If plain unittests are planned, mocking might be possible.
>
> What do you mean by this?

C API is a public interface easily accessible.
You can unit test single C API function (or aspect of it)
by mocking related/dependency calls (i.e. https://cmocka.org).


I like your ideas and approach which are aligned with my own "wishes"
I've had collected about GDAL tests.
Once you properly public with your project, I may be able to help.

Best regards,
--
Mateusz  Loskot, http://mateusz.loskot.net