[gdal-dev] autotest questions/issues

[Even Rouault]

Greg,
> I see that now.  I expected to be able to read the README in autotest
> and understand the big view of testing.  It feels like autotest is sort
> of a separable part of gdal not being in the tarball.
People who build GDAL from sources in scripts (Docker images etc) just 
need the sources. So a tarball with tests included is much larger than 
needed for their purposes
>>
>> The tests should match the code, so 3.3.2rc2 tests run against 3.3.2rc2
>> code.
> So is it helpful to file a doc bug, or are you inclined (since you
> actually understand the test plan) to adjust the README.md?
Seemed pretty obvious to me that you should use the gdalautotest-X 
version that matches the gdal-X one :-) Feel free to submit pull 
request(s) with documentation improvement that makes sense.
>> setdevenv.sh? Adding some sort of `make test` might be a cleaner approach
>> though.
> Yes, in general I take the view that it's good if a project meets the
> standard interfaces so that people don't have to understand anything
> about the project's test setup to run it.
When we'll switch to cmake, hopefully people will just have to run ctest 
. I don't think we want to invest more currently in improving the 
current GNUmakefile based infrastructure with a make check target
>
> I was really coming at this from the point of view that normally a
> package lists required dependencies and optional dependencies, and I had
> no idea what would happen if one were missing.
As its name suggests, whatever is in autotest/requirements.txt is 
required. If some components are missing, anything can happen at runtime.
>
>>>    [Now I see that osr tests fail with a warning about env if this is
>>>    missing, but really the test should be skipped or hard fail and not
>>>    run without the env.]
>> Yes, I think it should hard-fail
> Should I file a bug?  What I got was
>
> /home/n0/gdt/SOFTWARE/GEO/GDAL/gdal/autotest/osr/osr_pci.py:200: AssertionError
> =============================== warnings summary ===============================
> ../../../../../../../../usr/pkg/lib/python3.8/site-packages/_pytest/config/__init__.py:1233
>    /usr/pkg/lib/python3.8/site-packages/_pytest/config/__init__.py:1233: PytestConfigWarning: Unknown config option: env
>    
>      self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")
>
> -- Docs: https://docs.pytest.org/en/stable/warnings.html
>
> which I think led to testing without the env var.
It's just that you didn't install pytest-env.
>
> It does, but I didn't understand how to solve my issue, which is a
> python segfault (likely in gdal binary code loaded in a module, I
> speculate without basis) that ended the entire test run.   I worked
> around that my removing ogr/ogr_mitab.py and rerunning.
Once you've installed all requirements, would be interesting to retry 
again. In some cases, Python failures can become segfaults due to 
unexpected bad use of C API, particular for tests that are 
interdependent as Robert mentioned. What can happen sometimes is that 
when unexpected test failures happen, some temporary files in the 
autotest/xxxx/tmp files remain, and cause subsequent test runs to fail. 
Some manual cleaning might be needed.
>
> I also saw things like
>
>    ERROR 1: Driver GTM is considered for removal in GDAL 3.5. You are
>    invited to convert any dataset in that format to another more common one
>    .If you need this driver in fut ure GDAL versions, create a ticket at
>    https://github.com/OSGeo/gdal (look first for an existing one first) to
>    explain how critical it is for you (but the GDAL project may still
>    remove it), and to enable it now, set the
>    GDAL_ENABLE_DEPRECATED_DRIVER_GTM configuration option / environment
>    variable to YES ERROR 1: GPSTrackMaker driver failed to create
>    tmp/testogr2ogr21.gtm
>
> and I wonder if the tests should be setting
> GDAL_ENABLE_DEPRECATED_DRIVER_FOO.
> Alternatively those tests could be in a separate driver file and not
> normally invoked.

Will not happen once pytest-env is installed, since autotest/pytest.ini 
does set those environment variables

Even

-- 
http://www.spatialys.com
My software is free, but my time generally not.