[gdal-dev] Refactoring the test suite with pytest

Kurt Schwehr schwehr at gmail.com
Mon Oct 1 22:05:49 PDT 2018


I am definitely interested in improvements to the test infrastructure, but
always fear slowing down Even or getting bogged down in a massive effort.

In addition to working in the normal osgeo github world...

My current state: I started "autotest2
<https://github.com/schwehr/gdal-autotest2/tree/master/python>" back in
2013-14 after being unable to make autotest work for me and released much
of it as opensource.  It's unittest based for python and GoogleTest/gmock
for c++.  The Java isn't very far along, but it's Truth + JUnit4.  It's in
use inside google typically running every 10 to 60 minutes depending on
what's going on and a couple times a day in ASAN, MSAN, and TSAN modes.  On
the python side, it doesn't cover that much compared to the full python
test suite in autotest.  But in doing rewrites, I did have to make each
test independent.  I also have to allow control of where tests write
temporary data as I work in a system where the source tree is read only.  I
am probably close to making the tests python 3.7+ only (but likely to work
on 3.5+).  Whenever I start using a driver for the first time, I try to
rewrite the python test and create at least basic C++ tests, but I never
have enough time to do as complete of a job as I would like (err... like
the partially done pull request I have in to Proj).

You are welcome to use anything in autotest2 for the core gdal tree if it
fits.  I (meaning Google) are happy to contribute any or all of the code to
the GDAL tree under the GDAL license.

On Mon, Oct 1, 2018 at 3:32 PM Craig de Stigter <
craig.destigter at koordinates.com> wrote:

> Hi folks
>
> I've been looking at a potential large change to the GDAL test suite,
> replacing some of the worst parts. I thought I'd bring it up for discussion
> here, in case anyone has any wisdom or strong feelings about it.
>
> I've made a ticket at https://github.com/OSGeo/gdal/issues/949 .
>
> Essentially my plan is to use pytest <https://docs.pytest.org/en/latest/>
> as both a test runner and a helper library. We would remove or replace
> large parts of the `gdaltest` utilities with pytest equivalents.
>
> To give an idea of the main changes, this code:
>
> if ds.GetLayer(0).GetSpatialRef() is not None:
>>     gdaltest.post_reason('did not get expected spatial ref')
>>     return 'fail'
>>
> would change to:
>
>> assert ds.GetLayer(0).GetSpatialRef() is None, 'did not get expected
>> spatial ref'
>>
>
> The pytest runner gives contextual tracebacks from assert statements, as
> well as relevant variables and line numbers.
>
> Other changes:
>  * The `gdaltest_list` and the `__main__` entry point at the bottom of
> each file will be removed. Some tests will be renamed if they don't already
> start with `test_`.
>  * The `<name>_cleanup` fixture will be turned into a module-scoped
> fixture
> <https://docs.pytest.org/en/latest/fixture.html#scope-sharing-a-fixture-instance-across-tests-in-a-class-module-or-session>.
> This allows users to run any number of tests in the module, and the cleanup
> fixture will be invoked once after all of them are finished. If I find
> corresponding setup functions I will make them into fixtures too, but I
> haven't seen those so far in my digging.
>
> Other, more intrusive changes are possible but this is what I'm
> considering targeting as a first step. It is fairly straightforward to
> automate this change; I have a script in progress already to do that.
>
> I intend to make a compatibility shim for the old-style tests that can't
> easily be automatically updated. The shim would be a decorator, perhaps
> called `@old_test`, which would take 'skip' or 'fail' return values from
> tests and turn them into the appropriate `assert False` or `pytest.skip()`
> calls.
>
> I don't intend to make any changes to the C++ tests.
>
> A couple of complicating factors to be aware of:
>
>    - pytest only supports python 2.7+. The tests already only run in 2.7+
>    in Travis, so no big deal there. I think it's perfectly reasonable to only
>    support 2.7+ these days, given that 2.6 has been unsupported since 2013,
>    but I thought I'd mention it.
>    - pytest has no builtin way of *ordering* tests. It is assumed that
>    tests are independent of each other. In practice, all tests are run in
>    alphabetical name order within a given module. It appears to me that some
>    of GDAL's tests are *not* independent of each other within the same
>    module. However, most are already in alphabetical name order, so I don't
>    believe this is a problem; we are not worse off with pytest than previously.
>
>
> Please let me know your thoughts.
>
> --
> Regards,
> Craig de Stigter
>
> Developer
> Koordinates
>
> +64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates
> <https://twitter.com/koordinates>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev



-- 
--
http://schwehr.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20181001/e16e2b73/attachment.html>


More information about the gdal-dev mailing list