[gdal-dev] autotest questions/issues

[Robert Coup]

Hi Greg,

On Wed, 1 Sept 2021 at 01:33, Greg Troxel <gdt at lexort.com> wrote:

> A few comments from the perspective of a gdal test newbie, which I hope
> are helpful:
>
> * autotest is not in the distribution tarball
>

Yeah, it's never been in the release tarballs. And the release tarballs
start one folder level down (gdal/) as well.

TBH I think the release tarballs should just match/be an archive of the git
tag. But that's above my pay grade :-)

@Debian and other package maintainers — are you just getting the Git tags
these days or do you actually use the release tarballs?


>
>   + Does it test the gdal that is installed in one's PATH (and
>   PYTHONPATH), or is is testing the gdal that is in the source tree?
>

CONTRIBUTING.md has a guide to run `scripts/setdevenv.sh` which sets up
in-tree execution for utilities/libraries and the tests.


>
>   + Is one supposed to check out the corresponding branch (or really
>   tag), or is it ok to test 3.3.2rc2 with master?
>

The tests should match the code, so 3.3.2rc2 tests run against 3.3.2rc2
code.



> * no clear path to test a not-yet-installed gdal
>
> When building a new gdal, it seems a good idea to run tests before
> installing it.  autoconf has done this for year with make check.  I see
> no obvious way to do this.
>

setdevenv.sh? Adding some sort of `make test` might be a cleaner approach
though.


>
> * puzzling test dependencies
>
>   It's not really clear if a missing dependency (vs requirements.txt)
>   will cause problems (assuming pytest and lxml are there), in terms of
>   just skipping some things, bad output, or perhaps some missing pretty
>   printing.
>

a mixture. Is there some particular reason installing everything in
requirements.txt from PyPi is a problem?


>   I think pytest-env is
>     https://pypi.org/project/pytest-env/#history
>   but that appears unmaintained, with a last release in 2017.   Perhaps
>   there are no outstanding bugs and no changes and it is actually
>   maintained but there is no reason to change -- but that is highly
>   implausible.   It seems boutique, based on not being in pkgsrc, which
>   means no one else has needed it, even though we have 20K packages.
>

It's about 45 lines of code. Pytest plugins are often fairly minimal, and
just do One Thing.


>   [Now I see that osr tests fail with a warning about env if this is
>   missing, but really the test should be skipped or hard fail and not
>   run without the env.]
>

Yes, I think it should hard-fail


> * I'm guessing the expectation is that all the tests pass.  With
>   autotest from master and 3.3.2rc2 installed (base lib and python
>   bindings), I get:
>

In general all the tests pass. There's skips/xfails for particular known
issues, but maybe you've found a bug? Maybe BSD-specific, BSD isn't part of
CI.


>
> * skipping tests after a failure.


> It would be nice to have hints in README.md about skipping tests, but it
> seems easy enough to run osr and then the following ones.
>

you mean, stopping on the first failure? `pytest -x ...`
autotest/README.md has some docs on running specific tests / modules / etc

GDAL has some odd test layouts with particular inter-test dependencies from
when the test suite was bulk-ported via automation to work under pytest —
this made it a lot saner, but some of the "test 18 depends on test 17
passing" issues remain.

Rob :)

>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20210901/5a3b2a4c/attachment.html>