[gdal-dev] Refactoring the test suite with pytest
Craig de Stigter
craig.destigter at koordinates.com
Mon Oct 1 15:32:24 PDT 2018
I've been looking at a potential large change to the GDAL test suite,
replacing some of the worst parts. I thought I'd bring it up for discussion
here, in case anyone has any wisdom or strong feelings about it.
I've made a ticket at https://github.com/OSGeo/gdal/issues/949 .
Essentially my plan is to use pytest <https://docs.pytest.org/en/latest/>
as both a test runner and a helper library. We would remove or replace
large parts of the `gdaltest` utilities with pytest equivalents.
To give an idea of the main changes, this code:
if ds.GetLayer(0).GetSpatialRef() is not None:
> gdaltest.post_reason('did not get expected spatial ref')
> return 'fail'
would change to:
> assert ds.GetLayer(0).GetSpatialRef() is None, 'did not get expected
> spatial ref'
The pytest runner gives contextual tracebacks from assert statements, as
well as relevant variables and line numbers.
* The `gdaltest_list` and the `__main__` entry point at the bottom of each
file will be removed. Some tests will be renamed if they don't already
start with `test_`.
* The `<name>_cleanup` fixture will be turned into a module-scoped fixture
This allows users to run any number of tests in the module, and the cleanup
fixture will be invoked once after all of them are finished. If I find
corresponding setup functions I will make them into fixtures too, but I
haven't seen those so far in my digging.
Other, more intrusive changes are possible but this is what I'm considering
targeting as a first step. It is fairly straightforward to automate this
change; I have a script in progress already to do that.
I intend to make a compatibility shim for the old-style tests that can't
easily be automatically updated. The shim would be a decorator, perhaps
called `@old_test`, which would take 'skip' or 'fail' return values from
tests and turn them into the appropriate `assert False` or `pytest.skip()`
I don't intend to make any changes to the C++ tests.
A couple of complicating factors to be aware of:
- pytest only supports python 2.7+. The tests already only run in 2.7+
in Travis, so no big deal there. I think it's perfectly reasonable to only
support 2.7+ these days, given that 2.6 has been unsupported since 2013,
but I thought I'd mention it.
- pytest has no builtin way of *ordering* tests. It is assumed that
tests are independent of each other. In practice, all tests are run in
alphabetical name order within a given module. It appears to me that some
of GDAL's tests are *not* independent of each other within the same
module. However, most are already in alphabetical name order, so I don't
believe this is a problem; we are not worse off with pytest than previously.
Please let me know your thoughts.
Craig de Stigter
+64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the gdal-dev