[gdal-dev] Refactoring the test suite with pytest

Thu Oct 25 18:55:07 PDT 2018

I posted the link earlier in the thread, but to clarify:
https://github.com/OSGeo/gdal/pull/963

On Fri, 26 Oct 2018 at 14:54 Craig de Stigter <
craig.destigter at koordinates.com> wrote:

> My work on this branch is starting to wind down. There's still a few test
> failures to contend with, and I need to work through some inconsistencies
> in the different CI environments.
>
> Could I request some review of the PR, especially around scope and overall
> approach? I believe the PR description is reasonably useful to describe
> what I've been up to, but please ask questions if you have them, or
> comment/review on the PR itself.
>
> Thanks
> Craig de Stigter
>
> On Fri, 5 Oct 2018 at 22:50 Kurt Schwehr <schwehr at gmail.com> wrote:
>
>> The python tests in autotest2 are pretty much a subset of GDAL's
>> autotest.  The C++ and Java tests are mostly outside of what autotest has.
>>
>> On Tue, Oct 2, 2018 at 2:05 AM Craig de Stigter <
>> craig.destigter at koordinates.com> wrote:
>>
>>> Thanks. Are those tests entirely based on the osgeo git tests, or are
>>> there some extra ones? It might be worth getting the extra ones in at least.
>>>
>>>
>>> On Tue, 2 Oct 2018 at 18:06, Kurt Schwehr <schwehr at gmail.com> wrote:
>>>
>>>> I am definitely interested in improvements to the test infrastructure,
>>>> but always fear slowing down Even or getting bogged down in a massive
>>>> effort.
>>>>
>>>> In addition to working in the normal osgeo github world...
>>>>
>>>> My current state: I started "autotest2
>>>> <https://github.com/schwehr/gdal-autotest2/tree/master/python>" back
>>>> in 2013-14 after being unable to make autotest work for me and released
>>>> much of it as opensource.  It's unittest based for python and
>>>> GoogleTest/gmock for c++.  The Java isn't very far along, but it's Truth +
>>>> JUnit4.  It's in use inside google typically running every 10 to 60 minutes
>>>> depending on what's going on and a couple times a day in ASAN, MSAN, and
>>>> TSAN modes.  On the python side, it doesn't cover that much compared to the
>>>> full python test suite in autotest.  But in doing rewrites, I did have to
>>>> make each test independent.  I also have to allow control of where tests
>>>> write temporary data as I work in a system where the source tree is read
>>>> only.  I am probably close to making the tests python 3.7+ only (but likely
>>>> to work on 3.5+).  Whenever I start using a driver for the first time, I
>>>> try to rewrite the python test and create at least basic C++ tests, but I
>>>> never have enough time to do as complete of a job as I would like (err...
>>>> like the partially done pull request I have in to Proj).
>>>>
>>>> You are welcome to use anything in autotest2 for the core gdal tree if
>>>> it fits.  I (meaning Google) are happy to contribute any or all of the code
>>>> to the GDAL tree under the GDAL license.
>>>>
>>>> On Mon, Oct 1, 2018 at 3:32 PM Craig de Stigter <
>>>> craig.destigter at koordinates.com> wrote:
>>>>
>>>>> Hi folks
>>>>>
>>>>> I've been looking at a potential large change to the GDAL test suite,
>>>>> replacing some of the worst parts. I thought I'd bring it up for discussion
>>>>> here, in case anyone has any wisdom or strong feelings about it.
>>>>>
>>>>> I've made a ticket at https://github.com/OSGeo/gdal/issues/949 .
>>>>>
>>>>> Essentially my plan is to use pytest
>>>>> <https://docs.pytest.org/en/latest/> as both a test runner and a
>>>>> helper library. We would remove or replace large parts of the `gdaltest`
>>>>> utilities with pytest equivalents.
>>>>>
>>>>> To give an idea of the main changes, this code:
>>>>>
>>>>> if ds.GetLayer(0).GetSpatialRef() is not None:
>>>>>>     gdaltest.post_reason('did not get expected spatial ref')
>>>>>>     return 'fail'
>>>>>>
>>>>> would change to:
>>>>>
>>>>>> assert ds.GetLayer(0).GetSpatialRef() is None, 'did not get expected
>>>>>> spatial ref'
>>>>>>
>>>>>
>>>>> The pytest runner gives contextual tracebacks from assert statements,
>>>>> as well as relevant variables and line numbers.
>>>>>
>>>>> Other changes:
>>>>>  * The `gdaltest_list` and the `__main__` entry point at the bottom of
>>>>> each file will be removed. Some tests will be renamed if they don't already
>>>>> start with `test_`.
>>>>>  * The `<name>_cleanup` fixture will be turned into a module-scoped
>>>>> fixture
>>>>> <https://docs.pytest.org/en/latest/fixture.html#scope-sharing-a-fixture-instance-across-tests-in-a-class-module-or-session>.
>>>>> This allows users to run any number of tests in the module, and the cleanup
>>>>> fixture will be invoked once after all of them are finished. If I find
>>>>> corresponding setup functions I will make them into fixtures too, but I
>>>>> haven't seen those so far in my digging.
>>>>>
>>>>> Other, more intrusive changes are possible but this is what I'm
>>>>> considering targeting as a first step. It is fairly straightforward to
>>>>> automate this change; I have a script in progress already to do that.
>>>>>
>>>>> I intend to make a compatibility shim for the old-style tests that
>>>>> can't easily be automatically updated. The shim would be a decorator,
>>>>> perhaps called `@old_test`, which would take 'skip' or 'fail' return values
>>>>> from tests and turn them into the appropriate `assert False` or
>>>>> `pytest.skip()` calls.
>>>>>
>>>>> I don't intend to make any changes to the C++ tests.
>>>>>
>>>>> A couple of complicating factors to be aware of:
>>>>>
>>>>>    - pytest only supports python 2.7+. The tests already only run in
>>>>>    2.7+ in Travis, so no big deal there. I think it's perfectly reasonable to
>>>>>    only support 2.7+ these days, given that 2.6 has been unsupported since
>>>>>    2013, but I thought I'd mention it.
>>>>>    - pytest has no builtin way of *ordering* tests. It is assumed
>>>>>    that tests are independent of each other. In practice, all tests are run in
>>>>>    alphabetical name order within a given module. It appears to me that some
>>>>>    of GDAL's tests are *not* independent of each other within the
>>>>>    same module. However, most are already in alphabetical name order, so I
>>>>>    don't believe this is a problem; we are not worse off with pytest than
>>>>>    previously.
>>>>>
>>>>>
>>>>> Please let me know your thoughts.
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Craig de Stigter
>>>>>
>>>>> Developer
>>>>> Koordinates
>>>>>
>>>>> +64 21 256 9488 <+64%2021%20256%209488> / koordinates.com /
>>>>> @koordinates <https://twitter.com/koordinates>
>>>>> _______________________________________________
>>>>> gdal-dev mailing list
>>>>> gdal-dev at lists.osgeo.org
>>>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>>>
>>>>
>>>>
>>>> --
>>>> --
>>>> http://schwehr.org
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Craig
>>>
>>> Developer
>>> Koordinates
>>>
>>> +64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates
>>> <https://twitter.com/koordinates>
>>>
>>
>>
>> --
>> --
>> http://schwehr.org
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20181026/1d1b010a/attachment-0001.html>