[gdal-dev] Refactoring the test suite with pytest

Kurt Schwehr schwehr at gmail.com
Fri Oct 26 13:37:40 PDT 2018 

OMG awesome!  Sent a couple random thoughts your way on the PR.  I will try
to take a better look through the pr soon.  Looks like Even has some good
failure info for you to work on already.

On Thu, Oct 25, 2018 at 6:55 PM Craig de Stigter <
craig.destigter at koordinates.com> wrote:

> I posted the link earlier in the thread, but to clarify:
> https://github.com/OSGeo/gdal/pull/963
>
> On Fri, 26 Oct 2018 at 14:54 Craig de Stigter <
> craig.destigter at koordinates.com> wrote:
>
>> My work on this branch is starting to wind down. There's still a few test
>> failures to contend with, and I need to work through some inconsistencies
>> in the different CI environments.
>>
>> Could I request some review of the PR, especially around scope and
>> overall approach? I believe the PR description is reasonably useful to
>> describe what I've been up to, but please ask questions if you have them,
>> or comment/review on the PR itself.
>>
>> Thanks
>> Craig de Stigter
>>
>> On Fri, 5 Oct 2018 at 22:50 Kurt Schwehr <schwehr at gmail.com> wrote:
>>
>>> The python tests in autotest2 are pretty much a subset of GDAL's
>>> autotest.  The C++ and Java tests are mostly outside of what autotest has.
>>>
>>> On Tue, Oct 2, 2018 at 2:05 AM Craig de Stigter <
>>> craig.destigter at koordinates.com> wrote:
>>>
>>>> Thanks. Are those tests entirely based on the osgeo git tests, or are
>>>> there some extra ones? It might be worth getting the extra ones in at least.
>>>>
>>>>
>>>> On Tue, 2 Oct 2018 at 18:06, Kurt Schwehr <schwehr at gmail.com> wrote:
>>>>
>>>>> I am definitely interested in improvements to the test infrastructure,
>>>>> but always fear slowing down Even or getting bogged down in a massive
>>>>> effort.
>>>>>
>>>>> In addition to working in the normal osgeo github world...
>>>>>
>>>>> My current state: I started "autotest2
>>>>> <https://github.com/schwehr/gdal-autotest2/tree/master/python>" back
>>>>> in 2013-14 after being unable to make autotest work for me and released
>>>>> much of it as opensource.  It's unittest based for python and
>>>>> GoogleTest/gmock for c++.  The Java isn't very far along, but it's Truth +
>>>>> JUnit4.  It's in use inside google typically running every 10 to 60 minutes
>>>>> depending on what's going on and a couple times a day in ASAN, MSAN, and
>>>>> TSAN modes.  On the python side, it doesn't cover that much compared to the
>>>>> full python test suite in autotest.  But in doing rewrites, I did have to
>>>>> make each test independent.  I also have to allow control of where tests
>>>>> write temporary data as I work in a system where the source tree is read
>>>>> only.  I am probably close to making the tests python 3.7+ only (but likely
>>>>> to work on 3.5+).  Whenever I start using a driver for the first time, I
>>>>> try to rewrite the python test and create at least basic C++ tests, but I
>>>>> never have enough time to do as complete of a job as I would like (err...
>>>>> like the partially done pull request I have in to Proj).
>>>>>
>>>>> You are welcome to use anything in autotest2 for the core gdal tree if
>>>>> it fits.  I (meaning Google) are happy to contribute any or all of the code
>>>>> to the GDAL tree under the GDAL license.
>>>>>
>>>>> On Mon, Oct 1, 2018 at 3:32 PM Craig de Stigter <
>>>>> craig.destigter at koordinates.com> wrote:
>>>>>
>>>>>> Hi folks
>>>>>>
>>>>>> I've been looking at a potential large change to the GDAL test suite,
>>>>>> replacing some of the worst parts. I thought I'd bring it up for discussion
>>>>>> here, in case anyone has any wisdom or strong feelings about it.
>>>>>>
>>>>>> I've made a ticket at https://github.com/OSGeo/gdal/issues/949 .
>>>>>>
>>>>>> Essentially my plan is to use pytest
>>>>>> <https://docs.pytest.org/en/latest/> as both a test runner and a
>>>>>> helper library. We would remove or replace large parts of the `gdaltest`
>>>>>> utilities with pytest equivalents.
>>>>>>
>>>>>> To give an idea of the main changes, this code:
>>>>>>
>>>>>> if ds.GetLayer(0).GetSpatialRef() is not None:
>>>>>>>     gdaltest.post_reason('did not get expected spatial ref')
>>>>>>>     return 'fail'
>>>>>>>
>>>>>> would change to:
>>>>>>
>>>>>>> assert ds.GetLayer(0).GetSpatialRef() is None, 'did not get expected
>>>>>>> spatial ref'
>>>>>>>
>>>>>>
>>>>>> The pytest runner gives contextual tracebacks from assert statements,
>>>>>> as well as relevant variables and line numbers.
>>>>>>
>>>>>> Other changes:
>>>>>>  * The `gdaltest_list` and the `__main__` entry point at the bottom
>>>>>> of each file will be removed. Some tests will be renamed if they don't
>>>>>> already start with `test_`.
>>>>>>  * The `<name>_cleanup` fixture will be turned into a module-scoped
>>>>>> fixture
>>>>>> <https://docs.pytest.org/en/latest/fixture.html#scope-sharing-a-fixture-instance-across-tests-in-a-class-module-or-session>.
>>>>>> This allows users to run any number of tests in the module, and the cleanup
>>>>>> fixture will be invoked once after all of them are finished. If I find
>>>>>> corresponding setup functions I will make them into fixtures too, but I
>>>>>> haven't seen those so far in my digging.
>>>>>>
>>>>>> Other, more intrusive changes are possible but this is what I'm
>>>>>> considering targeting as a first step. It is fairly straightforward to
>>>>>> automate this change; I have a script in progress already to do that.
>>>>>>
>>>>>> I intend to make a compatibility shim for the old-style tests that
>>>>>> can't easily be automatically updated. The shim would be a decorator,
>>>>>> perhaps called `@old_test`, which would take 'skip' or 'fail' return values
>>>>>> from tests and turn them into the appropriate `assert False` or
>>>>>> `pytest.skip()` calls.
>>>>>>
>>>>>> I don't intend to make any changes to the C++ tests.
>>>>>>
>>>>>> A couple of complicating factors to be aware of:
>>>>>>
>>>>>>    - pytest only supports python 2.7+. The tests already only run in
>>>>>>    2.7+ in Travis, so no big deal there. I think it's perfectly reasonable to
>>>>>>    only support 2.7+ these days, given that 2.6 has been unsupported since
>>>>>>    2013, but I thought I'd mention it.
>>>>>>    - pytest has no builtin way of *ordering* tests. It is assumed
>>>>>>    that tests are independent of each other. In practice, all tests are run in
>>>>>>    alphabetical name order within a given module. It appears to me that some
>>>>>>    of GDAL's tests are *not* independent of each other within the
>>>>>>    same module. However, most are already in alphabetical name order, so I
>>>>>>    don't believe this is a problem; we are not worse off with pytest than
>>>>>>    previously.
>>>>>>
>>>>>>
>>>>>> Please let me know your thoughts.
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Craig de Stigter
>>>>>>
>>>>>> Developer
>>>>>> Koordinates
>>>>>>
>>>>>> +64 21 256 9488 <+64%2021%20256%209488> / koordinates.com /
>>>>>> @koordinates <https://twitter.com/koordinates>
>>>>>> _______________________________________________
>>>>>> gdal-dev mailing list
>>>>>> gdal-dev at lists.osgeo.org
>>>>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --
>>>>> http://schwehr.org
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Craig
>>>>
>>>> Developer
>>>> Koordinates
>>>>
>>>> +64 21 256 9488 <+64%2021%20256%209488> / koordinates.com /
>>>> @koordinates <https://twitter.com/koordinates>
>>>>
>>>
>>>
>>> --
>>> --
>>> http://schwehr.org
>>>
>>

-- 
--
http://schwehr.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20181026/71c89f85/attachment.html>

More information about the gdal-dev mailing list