[gdal-dev] Refactoring the test suite with pytest

Tue Oct 2 02:04:49 PDT 2018

Thanks. Are those tests entirely based on the osgeo git tests, or are there
some extra ones? It might be worth getting the extra ones in at least.

On Tue, 2 Oct 2018 at 18:06, Kurt Schwehr <schwehr at gmail.com> wrote:

> I am definitely interested in improvements to the test infrastructure, but
> always fear slowing down Even or getting bogged down in a massive effort.
>
> In addition to working in the normal osgeo github world...
>
> My current state: I started "autotest2
> <https://github.com/schwehr/gdal-autotest2/tree/master/python>" back in
> 2013-14 after being unable to make autotest work for me and released much
> of it as opensource.  It's unittest based for python and GoogleTest/gmock
> for c++.  The Java isn't very far along, but it's Truth + JUnit4.  It's in
> use inside google typically running every 10 to 60 minutes depending on
> what's going on and a couple times a day in ASAN, MSAN, and TSAN modes.  On
> the python side, it doesn't cover that much compared to the full python
> test suite in autotest.  But in doing rewrites, I did have to make each
> test independent.  I also have to allow control of where tests write
> temporary data as I work in a system where the source tree is read only.  I
> am probably close to making the tests python 3.7+ only (but likely to work
> on 3.5+).  Whenever I start using a driver for the first time, I try to
> rewrite the python test and create at least basic C++ tests, but I never
> have enough time to do as complete of a job as I would like (err... like
> the partially done pull request I have in to Proj).
>
> You are welcome to use anything in autotest2 for the core gdal tree if it
> fits.  I (meaning Google) are happy to contribute any or all of the code to
> the GDAL tree under the GDAL license.
>
> On Mon, Oct 1, 2018 at 3:32 PM Craig de Stigter <
> craig.destigter at koordinates.com> wrote:
>
>> Hi folks
>>
>> I've been looking at a potential large change to the GDAL test suite,
>> replacing some of the worst parts. I thought I'd bring it up for discussion
>> here, in case anyone has any wisdom or strong feelings about it.
>>
>> I've made a ticket at https://github.com/OSGeo/gdal/issues/949 .
>>
>> Essentially my plan is to use pytest <https://docs.pytest.org/en/latest/>
>> as both a test runner and a helper library. We would remove or replace
>> large parts of the `gdaltest` utilities with pytest equivalents.
>>
>> To give an idea of the main changes, this code:
>>
>> if ds.GetLayer(0).GetSpatialRef() is not None:
>>>     gdaltest.post_reason('did not get expected spatial ref')
>>>     return 'fail'
>>>
>> would change to:
>>
>>> assert ds.GetLayer(0).GetSpatialRef() is None, 'did not get expected
>>> spatial ref'
>>>
>>
>> The pytest runner gives contextual tracebacks from assert statements, as
>> well as relevant variables and line numbers.
>>
>> Other changes:
>>  * The `gdaltest_list` and the `__main__` entry point at the bottom of
>> each file will be removed. Some tests will be renamed if they don't already
>> start with `test_`.
>>  * The `<name>_cleanup` fixture will be turned into a module-scoped
>> fixture
>> <https://docs.pytest.org/en/latest/fixture.html#scope-sharing-a-fixture-instance-across-tests-in-a-class-module-or-session>.
>> This allows users to run any number of tests in the module, and the cleanup
>> fixture will be invoked once after all of them are finished. If I find
>> corresponding setup functions I will make them into fixtures too, but I
>> haven't seen those so far in my digging.
>>
>> Other, more intrusive changes are possible but this is what I'm
>> considering targeting as a first step. It is fairly straightforward to
>> automate this change; I have a script in progress already to do that.
>>
>> I intend to make a compatibility shim for the old-style tests that can't
>> easily be automatically updated. The shim would be a decorator, perhaps
>> called `@old_test`, which would take 'skip' or 'fail' return values from
>> tests and turn them into the appropriate `assert False` or `pytest.skip()`
>> calls.
>>
>> I don't intend to make any changes to the C++ tests.
>>
>> A couple of complicating factors to be aware of:
>>
>>    - pytest only supports python 2.7+. The tests already only run in
>>    2.7+ in Travis, so no big deal there. I think it's perfectly reasonable to
>>    only support 2.7+ these days, given that 2.6 has been unsupported since
>>    2013, but I thought I'd mention it.
>>    - pytest has no builtin way of *ordering* tests. It is assumed that
>>    tests are independent of each other. In practice, all tests are run in
>>    alphabetical name order within a given module. It appears to me that some
>>    of GDAL's tests are *not* independent of each other within the same
>>    module. However, most are already in alphabetical name order, so I don't
>>    believe this is a problem; we are not worse off with pytest than previously.
>>
>>
>> Please let me know your thoughts.
>>
>> --
>> Regards,
>> Craig de Stigter
>>
>> Developer
>> Koordinates
>>
>> +64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates
>> <https://twitter.com/koordinates>
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
>
> --
> --
> http://schwehr.org
>

-- 
Regards,
Craig

Developer
Koordinates

+64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates
<https://twitter.com/koordinates>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20181002/e894f914/attachment-0001.html>