[gdal-dev] autotest questions/issues

[Greg Troxel]

After a very long time of not running the tests, because I thought it
was harder than it is, I have started down that path.  (Thanks to Even
for an offlist hint.)

I'm running tests on NetBSD 9 amd64, and realize that I may be the first
to do so.


A few comments from the perspective of a gdal test newbie, which I hope
are helpful:

* autotest is not in the distribution tarball

I realize it's 194M, but I don't see how pacakges are supposed to run
tests.  In pkgsrc, a package can declare a test invocation (e.g. "make
check" for autotools packages).  I'm a big fan of testing as installed.
So perhaps I should package autotest - but it's not clear there are
releases.

I am not so much asking for this to change as for it to be explained in
README.md.

* lack of instructions about branches and which code is tested

autotest/README.md doesn't explain

  + Does it test the gdal that is installed in one's PATH (and
  PYTHONPATH), or is is testing the gdal that is in the source tree?

  + Is one supposed to check out the corresponding branch (or really
  tag), or is it ok to test 3.3.2rc2 with master?

Now it's clear that the installed version is tested, and one should
match the branch, from first principles and having seen a test failure
about no CoordinateEpoch in 3.3.2.

* no clear path to test a not-yet-installed gdal

When building a new gdal, it seems a good idea to run tests before
installing it.  autoconf has done this for year with make check.  I see
no obvious way to do this.

* puzzling test dependencies

  It's not really clear if a missing dependency (vs requirements.txt)
  will cause problems (assuming pytest and lxml are there), in terms of
  just skipping some things, bad output, or perhaps some missing pretty
  printing.

  I think pytest-env is
    https://pypi.org/project/pytest-env/#history
  but that appears unmaintained, with a last release in 2017.   Perhaps
  there are no outstanding bugs and no changes and it is actually
  maintained but there is no reason to change -- but that is highly
  implausible.   It seems boutique, based on not being in pkgsrc, which
  means no one else has needed it, even though we have 20K packages.

  [Now I see that osr tests fail with a warning about env if this is
  missing, but really the test should be skipped or hard fail and not
  run without the env.]

* I'm guessing the expectation is that all the tests pass.  With
  autotest from master and 3.3.2rc2 installed (base lib and python
  bindings), I get:

    ogr/ogr_mitab.py .FFFatal Python error: Segmentation fault

    Current thread 0x00007d63dbe3b800 (most recent call first):
      File "/usr/pkg/lib/python3.8/site-packages/osgeo/ogr.py", line 1143 in SetAttributeFilter
      File "/home/n0/gdt/SOFTWARE/GEO/GDAL/gdal/autotest/ogr/ogr_mitab.py", line 177 in test_ogr_mitab_5
      File "/usr/pkg/lib/python3.8/site-packages/_pytest/python.py", line 183 in pytest_pyfunc_call
      File "/usr/pkg/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
      File "/usr/pkg/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
      File "/usr/pkg/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
      File "/usr/pkg/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
      File "/usr/pkg/lib/python3.8/site-packages/_pytest/python.py", line 1641 in runtest
      File "/usr/pkg/lib/python3.8/site-packages/_pytest/runner.py", line 162 in pytest_runtest_call
      File "/usr/pkg/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
      File "/usr/pkg/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>

Looking at gdb briefly, this looks like perhaps some kind fo threads
issue.  I remember from a different context around 10 years ago that
python called functions in signal handlers that were not defined as
async-signal-safe by POSIX, and I wonder if that problem still exists
and is related, or if gdb is just having a bad time.

I also got pyscripts failures:

         - pyscripts/test_gdal_polygonize.py:45 test_gdal_polygonize_1
         - pyscripts/test_gdal_polygonize.py:103 test_gdal_polygonize_2
         - pyscripts/test_ogr2ogr_py.py:663 test_ogr2ogr_py_21
         - pyscripts/test_ogr2ogr_py.py:1110 test_ogr2ogr_py_37
         - pyscripts/test_ogr2ogr_py.py:1173 test_ogr2ogr_py_39
         - pyscripts/test_ogrmerge.py:42 test_ogrmerge_1
         - pyscripts/test_ogrmerge.py:61 test_ogrmerge_2

and utilities

     600 passed
      10 failed
         - utilities/test_gdalwarp.py:324 test_gdalwarp_15
         - utilities/test_gnmutils.py:80 test_gnmmanage_3
         - utilities/test_gnmutils.py:95 test_gnmmanage_4
         - utilities/test_gnmutils.py:107 test_gnmanalyse_1
         - utilities/test_gnmutils.py:121 test_gnmanalyse_2
         - utilities/test_ogr2ogr.py:645 test_ogr2ogr_21
         - utilities/test_ogr2ogr.py:1223 test_ogr2ogr_37
         - utilities/test_ogr2ogr.py:1284 test_ogr2ogr_39
         - utilities/test_ogr2ogr.py:2299 test_ogr2ogr_64
         - utilities/test_ogrinfo.py:557 test_ogrinfo_fielddomains

* skipping tests after a failure.

It would be nice to have hints in README.md about skipping tests, but it
seems easy enough to run osr and then the following ones.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20210831/4d07a360/attachment.sig>