[gdal-dev] Refactoring the test suite with pytest

[Craig de Stigter]

Hi folks

I've been looking at a potential large change to the GDAL test suite,
replacing some of the worst parts. I thought I'd bring it up for discussion
here, in case anyone has any wisdom or strong feelings about it.

I've made a ticket at https://github.com/OSGeo/gdal/issues/949 .

Essentially my plan is to use pytest <https://docs.pytest.org/en/latest/>
as both a test runner and a helper library. We would remove or replace
large parts of the `gdaltest` utilities with pytest equivalents.

To give an idea of the main changes, this code:

if ds.GetLayer(0).GetSpatialRef() is not None:
>     gdaltest.post_reason('did not get expected spatial ref')
>     return 'fail'
>
would change to:

> assert ds.GetLayer(0).GetSpatialRef() is None, 'did not get expected
> spatial ref'
>

The pytest runner gives contextual tracebacks from assert statements, as
well as relevant variables and line numbers.

Other changes:
 * The `gdaltest_list` and the `__main__` entry point at the bottom of each
file will be removed. Some tests will be renamed if they don't already
start with `test_`.
 * The `<name>_cleanup` fixture will be turned into a module-scoped fixture
<https://docs.pytest.org/en/latest/fixture.html#scope-sharing-a-fixture-instance-across-tests-in-a-class-module-or-session>.
This allows users to run any number of tests in the module, and the cleanup
fixture will be invoked once after all of them are finished. If I find
corresponding setup functions I will make them into fixtures too, but I
haven't seen those so far in my digging.

Other, more intrusive changes are possible but this is what I'm considering
targeting as a first step. It is fairly straightforward to automate this
change; I have a script in progress already to do that.

I intend to make a compatibility shim for the old-style tests that can't
easily be automatically updated. The shim would be a decorator, perhaps
called `@old_test`, which would take 'skip' or 'fail' return values from
tests and turn them into the appropriate `assert False` or `pytest.skip()`
calls.

I don't intend to make any changes to the C++ tests.

A couple of complicating factors to be aware of:

   - pytest only supports python 2.7+. The tests already only run in 2.7+
   in Travis, so no big deal there. I think it's perfectly reasonable to only
   support 2.7+ these days, given that 2.6 has been unsupported since 2013,
   but I thought I'd mention it.
   - pytest has no builtin way of *ordering* tests. It is assumed that
   tests are independent of each other. In practice, all tests are run in
   alphabetical name order within a given module. It appears to me that some
   of GDAL's tests are *not* independent of each other within the same
   module. However, most are already in alphabetical name order, so I don't
   believe this is a problem; we are not worse off with pytest than previously.


Please let me know your thoughts.

-- 
Regards,
Craig de Stigter

Developer
Koordinates

+64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates
<https://twitter.com/koordinates>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20181002/c6dfe270/attachment.html>