[gdal-dev] Refactoring the test suite with pytest

Thu Oct 25 18:54:27 PDT 2018

My work on this branch is starting to wind down. There's still a few test
failures to contend with, and I need to work through some inconsistencies
in the different CI environments.

Could I request some review of the PR, especially around scope and overall
approach? I believe the PR description is reasonably useful to describe
what I've been up to, but please ask questions if you have them, or
comment/review on the PR itself.

Thanks
Craig de Stigter

On Fri, 5 Oct 2018 at 22:50 Kurt Schwehr <schwehr at gmail.com> wrote:

> The python tests in autotest2 are pretty much a subset of GDAL's
> autotest.  The C++ and Java tests are mostly outside of what autotest has.
>
> On Tue, Oct 2, 2018 at 2:05 AM Craig de Stigter <
> craig.destigter at koordinates.com> wrote:
>
>> Thanks. Are those tests entirely based on the osgeo git tests, or are
>> there some extra ones? It might be worth getting the extra ones in at least.
>>
>>
>> On Tue, 2 Oct 2018 at 18:06, Kurt Schwehr <schwehr at gmail.com> wrote:
>>
>>> I am definitely interested in improvements to the test infrastructure,
>>> but always fear slowing down Even or getting bogged down in a massive
>>> effort.
>>>
>>> In addition to working in the normal osgeo github world...
>>>
>>> My current state: I started "autotest2
>>> <https://github.com/schwehr/gdal-autotest2/tree/master/python>" back in
>>> 2013-14 after being unable to make autotest work for me and released much
>>> of it as opensource.  It's unittest based for python and GoogleTest/gmock
>>> for c++.  The Java isn't very far along, but it's Truth + JUnit4.  It's in
>>> use inside google typically running every 10 to 60 minutes depending on
>>> what's going on and a couple times a day in ASAN, MSAN, and TSAN modes.  On
>>> the python side, it doesn't cover that much compared to the full python
>>> test suite in autotest.  But in doing rewrites, I did have to make each
>>> test independent.  I also have to allow control of where tests write
>>> temporary data as I work in a system where the source tree is read only.  I
>>> am probably close to making the tests python 3.7+ only (but likely to work
>>> on 3.5+).  Whenever I start using a driver for the first time, I try to
>>> rewrite the python test and create at least basic C++ tests, but I never
>>> have enough time to do as complete of a job as I would like (err... like
>>> the partially done pull request I have in to Proj).
>>>
>>> You are welcome to use anything in autotest2 for the core gdal tree if
>>> it fits.  I (meaning Google) are happy to contribute any or all of the code
>>> to the GDAL tree under the GDAL license.
>>>
>>> On Mon, Oct 1, 2018 at 3:32 PM Craig de Stigter <
>>> craig.destigter at koordinates.com> wrote:
>>>
>>>> Hi folks
>>>>
>>>> I've been looking at a potential large change to the GDAL test suite,
>>>> replacing some of the worst parts. I thought I'd bring it up for discussion
>>>> here, in case anyone has any wisdom or strong feelings about it.
>>>>
>>>> I've made a ticket at https://github.com/OSGeo/gdal/issues/949 .
>>>>
>>>> Essentially my plan is to use pytest
>>>> <https://docs.pytest.org/en/latest/> as both a test runner and a
>>>> helper library. We would remove or replace large parts of the `gdaltest`
>>>> utilities with pytest equivalents.
>>>>
>>>> To give an idea of the main changes, this code:
>>>>
>>>> if ds.GetLayer(0).GetSpatialRef() is not None:
>>>>>     gdaltest.post_reason('did not get expected spatial ref')
>>>>>     return 'fail'
>>>>>
>>>> would change to:
>>>>
>>>>> assert ds.GetLayer(0).GetSpatialRef() is None, 'did not get expected
>>>>> spatial ref'
>>>>>
>>>>
>>>> The pytest runner gives contextual tracebacks from assert statements,
>>>> as well as relevant variables and line numbers.
>>>>
>>>> Other changes:
>>>>  * The `gdaltest_list` and the `__main__` entry point at the bottom of
>>>> each file will be removed. Some tests will be renamed if they don't already
>>>> start with `test_`.
>>>>  * The `<name>_cleanup` fixture will be turned into a module-scoped
>>>> fixture
>>>> <https://docs.pytest.org/en/latest/fixture.html#scope-sharing-a-fixture-instance-across-tests-in-a-class-module-or-session>.
>>>> This allows users to run any number of tests in the module, and the cleanup
>>>> fixture will be invoked once after all of them are finished. If I find
>>>> corresponding setup functions I will make them into fixtures too, but I
>>>> haven't seen those so far in my digging.
>>>>
>>>> Other, more intrusive changes are possible but this is what I'm
>>>> considering targeting as a first step. It is fairly straightforward to
>>>> automate this change; I have a script in progress already to do that.
>>>>
>>>> I intend to make a compatibility shim for the old-style tests that
>>>> can't easily be automatically updated. The shim would be a decorator,
>>>> perhaps called `@old_test`, which would take 'skip' or 'fail' return values
>>>> from tests and turn them into the appropriate `assert False` or
>>>> `pytest.skip()` calls.
>>>>
>>>> I don't intend to make any changes to the C++ tests.
>>>>
>>>> A couple of complicating factors to be aware of:
>>>>
>>>>    - pytest only supports python 2.7+. The tests already only run in
>>>>    2.7+ in Travis, so no big deal there. I think it's perfectly reasonable to
>>>>    only support 2.7+ these days, given that 2.6 has been unsupported since
>>>>    2013, but I thought I'd mention it.
>>>>    - pytest has no builtin way of *ordering* tests. It is assumed that
>>>>    tests are independent of each other. In practice, all tests are run in
>>>>    alphabetical name order within a given module. It appears to me that some
>>>>    of GDAL's tests are *not* independent of each other within the same
>>>>    module. However, most are already in alphabetical name order, so I don't
>>>>    believe this is a problem; we are not worse off with pytest than previously.
>>>>
>>>>
>>>> Please let me know your thoughts.
>>>>
>>>> --
>>>> Regards,
>>>> Craig de Stigter
>>>>
>>>> Developer
>>>> Koordinates
>>>>
>>>> +64 21 256 9488 <+64%2021%20256%209488> / koordinates.com /
>>>> @koordinates <https://twitter.com/koordinates>
>>>> _______________________________________________
>>>> gdal-dev mailing list
>>>> gdal-dev at lists.osgeo.org
>>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>>
>>>
>>>
>>> --
>>> --
>>> http://schwehr.org
>>>
>>
>>
>> --
>> Regards,
>> Craig
>>
>> Developer
>> Koordinates
>>
>> +64 21 256 9488 <+64%2021%20256%209488> / koordinates.com / @koordinates
>> <https://twitter.com/koordinates>
>>
>
>
> --
> --
> http://schwehr.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20181026/bd2ba56e/attachment.html>