[gdal-dev] Refactoring the test suite with pytest

[Kurt Schwehr]

The python tests in autotest2 are pretty much a subset of GDAL's autotest.
The C++ and Java tests are mostly outside of what autotest has.

On Tue, Oct 2, 2018 at 2:05 AM Craig de Stigter <
craig.destigter at koordinates.com> wrote:

> Thanks. Are those tests entirely based on the osgeo git tests, or are
> there some extra ones? It might be worth getting the extra ones in at least.
>
>
> On Tue, 2 Oct 2018 at 18:06, Kurt Schwehr <schwehr at gmail.com> wrote:
>
>> I am definitely interested in improvements to the test infrastructure,
>> but always fear slowing down Even or getting bogged down in a massive
>> effort.
>>
>> In addition to working in the normal osgeo github world...
>>
>> My current state: I started "autotest2
>> <https://github.com/schwehr/gdal-autotest2/tree/master/python>" back in
>> 2013-14 after being unable to make autotest work for me and released much
>> of it as opensource.  It's unittest based for python and GoogleTest/gmock
>> for c++.  The Java isn't very far along, but it's Truth + JUnit4.  It's in
>> use inside google typically running every 10 to 60 minutes depending on
>> what's going on and a couple times a day in ASAN, MSAN, and TSAN modes.  On
>> the python side, it doesn't cover that much compared to the full python
>> test suite in autotest.  But in doing rewrites, I did have to make each
>> test independent.  I also have to allow control of where tests write
>> temporary data as I work in a system where the source tree is read only.  I
>> am probably close to making the tests python 3.7+ only (but likely to work
>> on 3.5+).  Whenever I start using a driver for the first time, I try to
>> rewrite the python test and create at least basic C++ tests, but I never
>> have enough time to do as complete of a job as I would like (err... like
>> the partially done pull request I have in to Proj).
>>
>> You are welcome to use anything in autotest2 for the core gdal tree if it
>> fits.  I (meaning Google) are happy to contribute any or all of the code to
>> the GDAL tree under the GDAL license.
>>
>> On Mon, Oct 1, 2018 at 3:32 PM Craig de Stigter <
>> craig.destigter at koordinates.com> wrote:
>>
>>> Hi folks
>>>
>>> I've been looking at a potential large change to the GDAL test suite,
>>> replacing some of the worst parts. I thought I'd bring it up for discussion
>>> here, in case anyone has any wisdom or strong feelings about it.
>>>
>>> I've made a ticket at https://github.com/OSGeo/gdal/issues/949 .
>>>
>>> Essentially my plan is to use pytest
>>> <https://docs.pytest.org/en/latest/> as both a test runner and a helper
>>> library. We would remove or replace large parts of the `gdaltest` utilities
>>> with pytest equivalents.
>>>
>>> To give an idea of the main changes, this code:
>>>
>>> if ds.GetLayer(0).GetSpatialRef() is not None:
>>>>     gdaltest.post_reason('did not get expected spatial ref')
>>>>     return 'fail'
>>>>
>>> would change to:
>>>
>>>> assert ds.GetLayer(0).GetSpatialRef() is None, 'did not get expected
>>>> spatial ref'
>>>>
>>>
>>> The pytest runner gives contextual tracebacks from assert statements, as
>>> well as relevant variables and line numbers.
>>>
>>> Other changes:
>>>  * The `gdaltest_list` and the `__main__` entry point at the bottom of
>>> each file will be removed. Some tests will be renamed if they don't already
>>> start with `test_`.
>>>  * The `<name>_cleanup` fixture will be turned into a module-scoped
>>> fixture
>>> <https://docs.pytest.org/en/latest/fixture.html#scope-sharing-a-fixture-instance-across-tests-in-a-class-module-or-session>.
>>> This allows users to run any number of tests in the module, and the cleanup
>>> fixture will be invoked once after all of them are finished. If I find
>>> corresponding setup functions I will make them into fixtures too, but I
>>> haven't seen those so far in my digging.
>>>
>>> Other, more intrusive changes are possible but this is what I'm
>>> considering targeting as a first step. It is fairly straightforward to
>>> automate this change; I have a script in progress already to do that.
>>>
>>> I intend to make a compatibility shim for the old-style tests that can't
>>> easily be automatically updated. The shim would be a decorator, perhaps
>>> called `@old_test`, which would take 'skip' or 'fail' return values from
>>> tests and turn them into the appropriate `assert False` or `pytest.skip()`
>>> calls.
>>>
>>> I don't intend to make any changes to the C++ tests.
>>>
>>> A couple of complicating factors to be aware of:
>>>
>>>    - pytest only supports python 2.7+. The tests already only run in
>>>    2.7+ in Travis, so no big deal there. I think it's perfectly reasonable to
>>>    only support 2.7+ these days, given that 2.6 has been unsupported since
>>>    2013, but I thought I'd mention it.
>>>    - pytest has no builtin way of *ordering* tests. It is assumed that
>>>    tests are independent of each other. In practice, all tests are run in
>>>    alphabetical name order within a given module. It appears to me that some
>>>    of GDAL's tests are *not* independent of each other within the same
>>>    module. However, most are already in alphabetical name order, so I don't
>>>    believe this is a problem; we are not worse off with pytest than previously.
>>>
>>>
>>> Please let me know your thoughts.
>>>
>>> --
>>> Regards,
>>> Craig de Stigter
>>>
>>> Developer
>>> Koordinates
>>>
>>> +64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates
>>> <https://twitter.com/koordinates>
>>> _______________________________________________
>>> gdal-dev mailing list
>>> gdal-dev at lists.osgeo.org
>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>>
>>
>> --
>> --
>> http://schwehr.org
>>
>
>
> --
> Regards,
> Craig
>
> Developer
> Koordinates
>
> +64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates
> <https://twitter.com/koordinates>
>


-- 
--
http://schwehr.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20181005/8984b7ba/attachment.html>