[gdal-dev] More consistent use of pytest's parameterization in autotests

[Kurt Schwehr]

Agreed about test interdependency being rough. Internally at work, we have
a test runner that intentionally does not run tests in order.  All the
autotest2 stuff I did should all be order independent.  Sadly, my old tests
are using pythons normal test setup, not pytest.

On Mon, Feb 20, 2023, 7:57 AM Daniel Evans <daniel.fred.evans at gmail.com>
wrote:

> > Additionally, these loops fix the order that the checks are made, and
> bugs can hide in test order dependency.
>
> As we're on the topic of pytest, I'll mention fixtures as a handy
> feature for test setup: https://docs.pytest.org/en/6.2.x/fixture.html
>
> I discovered a while back that some parts of the GDAL test suite break
> if test order changes, because Test B relies implicitly some state
> being left over from Test A. This prevents some possible improvements
> like parallelising the test suite for speed (at least, without careful
> manual organisation of what can be done in parallel). It was also a
> little annoying trying to debug tests that started segfaulting when
> the order changed!
>
> Regards,
> Daniel
>
> On Mon, 20 Feb 2023 at 15:43, Sean Gillies <sean.gillies at gmail.com> wrote:
> >
> > Hi all,
> >
> > I just saw a maintainer recommend to a contributor that the contributor
> loop over test cases from within a test function and it prompted me to
> speak up about a better practice: using pytest parameterization
> https://docs.pytest.org/en/6.2.x/parametrize.html#pytest-mark-parametrize-parametrizing-test-functions
> .
> >
> > Using a loop, like at
> https://github.com/OSGeo/gdal/blob/master/autotest/gcore/gdal_stats.py#L660,
> can hide bugs and add friction to development. In that particular case,
> there are 5 different fill values to be checked. If there are different
> code paths for each of these values (a worst case scenario) and each has a
> small bug, you will have to run the whole test suite 5 times to expose the
> 5 bugs one at a time. Additionally, these loops fix the order that the
> checks are made, and bugs can hide in test order dependency. Ideally,
> GDAL's tests run in random order.
> >
> > In a nutshell, you hoist the loop and list out into a test function
> decorator and then pytest discovers and runs all the separate cases. Pytest
> doesn't stop on the first failure in the list unless you explicitly tell it
> to do so.
> >
> > # Original GDAL test code
> > def test_fill_value():
> >     for fill_val in [0, 1, 32767, 32768, 65535]:
> >         func(fill_val)
> >         assert something
> >
> > # With pytest parameterization
> > @pytest.mark.parametrize("fill_val", [0, 1, 32767, 32768, 65535])
> > def test_fill_value(fill_val):
> >         func(fill_val)
> >         assert something
> >
> > I'm not proposing rewrites of old tests, but I think it is worth
> adopting a better practice for tests now and in the future.
> >
> > --
> > Sean Gillies
> > _______________________________________________
> > gdal-dev mailing list
> > gdal-dev at lists.osgeo.org
> > https://lists.osgeo.org/mailman/listinfo/gdal-dev
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20230220/d257dfeb/attachment.htm>