[gdal-dev] GDAL testing
Dmitry Baryshnikov
bishop.dev at gmail.com
Tue Sep 15 07:46:19 PDT 2015
Hi Karl,
I think that some ready tests of GDAL are available. I'm not familiar
with autotest2, so some links to docs and sources are welcome. Also I
think that in gist you provided only some examples, and somewhere else
the repo exists. If not - I'll try to play with this code.
Best regards,
Dmitry
15.09.2015 00:42, Kurt Schwehr пишет:
> Hi Dmitry,
>
> I didn't totally understand your question. I don't have autotest2
> really out yet. You are welcome to use the sample test files that I
> posted in any way that the Apache 2.0 license allows.
>
> -kurt
>
> On Sun, Sep 6, 2015 at 8:27 AM, Dmitry Baryshnikov
> <bishop.dev at gmail.com <mailto:bishop.dev at gmail.com>> wrote:
>
> Hi Kurt,
>
> During code sprint in Korea (FOSS4G 2015) I plan to play with new
> approach of CMake fro GDAL. The one of experiments will be try to
> use CTest. As I plan restructure the sources tree, I can try to
> integrate you work on autotest2 and CTest. Also we can try to
> create new test directory structure more compatible for test and
> sources tree (this is about you wrote: Probably should move python
> code to also match the C++ tree. e.g. tiff_read_test.py ->
> autotest2/py/frmts/gtiff/tiff_read_test.py).
> How can get you tests? What do I need to use autotest2?
>
> Best regards,
> Dmitry
>
> 05.09.2015 20:37, Kurt Schwehr пишет:
>> (Subject change to focus on testing)
>>
>> Hi all,
>>
>> First off... what GDAL has with autotest, travis-ci and coverity
>> is awesome!
>>
>> Thoughts / discussion more than welcome!
>>
>> For my production work, I'm not able to use the autotest python
>> code because of its non-unittest architecture. So... I started
>> creating python unittest and C++ gunit based tests. I use
>> autotest2 in Google's internal continuous integration system in
>> our main code base. I'm using Google's build system... I've got
>> nothing started for running the C++ tests outside of Google.
>>
>> Apologies for not even getting out at least samples of autotest2
>> for folks to inspect and comment on. My intention is to put what
>> I have in a git repo and the to start discussions as to what (if
>> anything) GDAL community wants to do with autotest2. I was
>> hoping to get a lot more coverage and get GDAL 2.x.x support, but
>> that will have to come later. It's only 14K lines at this point
>> (optimistically 2-3% done), but it has been a huge help for me
>> especially with in upgrading versions of gdal and catching bugs
>> in support libs & development toolchains.
>>
>> The tests are more focused on test isolation than autotest. This
>> allows for a lot more parallelism in testing. e.g. It's fair
>> game to run all tests at the same time on the same machine.
>>
>> find . -name \*.py | xargs wc -l | tail -1
>> 10684 total
>>
>> find . -name \*.cc -o -name \*.h | xargs wc -l | tail -1
>> 3734 total
>>
>> Where GDAL's autotest is 204K lines:
>>
>> find . -name \*.py | xargs wc -l | tail -1
>> 193994 total
>> find . -name \*.c\* -o -name \*.h | xargs wc -l | tail -1
>> 10471 total
>>
>> Here are some samples:
>>
>> C++ tests use C++11, gunit, google logging, gflags: (Hoping for
>> C++14 soon.. e.g. make_unique)
>> - autotest2/cpp/port/cpl_conv_test.cc
>> <https://gist.github.com/schwehr/13137d826763763fb031> (Yes, this
>> is massively boring code)
>> - autotest2/cpp/ogr/ogrpoint_test.cc
>> <https://gist.github.com/schwehr/c8ee86a6f6a1c1cc043b>
>> - autotest2/cpp/ogr/ogrsf_frmts/geojson/geojson_test.cc
>> <https://gist.github.com/schwehr/bc44b91a37cd621212c4>
>>
>> Python pretty much follows the autotest layout, but with util
>> files in the same directory. Assumes python 2.7 or >= 3.4 (have
>> not tried py3 yet)
>> - autotest2/gcore/gcore_util.py
>> <https://gist.github.com/schwehr/c143927ca25d03a10265>
>> - autotest2/gdrivers/gdrivers_util.py
>> <https://gist.github.com/schwehr/dd75f73cedf8f7b5357e>
>> - autotest2/gdrivers/tiff_read_test.py
>> <https://gist.github.com/schwehr/a35b2bc8a7956ef1f620> (I'm
>> leading towards moving driver tests in gcore to gdrivers)
>> - autotest2/ogr/geojson_test.py
>> <https://gist.github.com/schwehr/6cbdc3482055d2237ad2>
>>
>> Probably should move python code to also match the C++ tree. e.g.
>>
>> tiff_read_test.py -> autotest2/py/frmts/gtiff/tiff_read_test.py
>>
>> I'm (mostly) following Google's style guides. Public versions
>> here: C++
>> <https://google-styleguide.googlecode.com/svn/trunk/cppguide.html> Python
>> <https://google-styleguide.googlecode.com/svn/trunk/pyguide.html>
>>
>> All C++ should be formatted with "clang-format --style=Google"
>>
>> What does autotest2 not do?
>>
>> Would like to eventually do (unsorted):
>> - Test error handling on a range of corrupt data sources
>> - Fuzz testing, ASAN/MSAN/TSAN/Valgrind/Heap checks (I've done
>> some MSAN & heap checkers by hand)
>> - Performance testing - time and memory usage
>> - Test the C API at the C level
>> - Test platforms other than Linux (MS Win*, Mac OSX, Android,
>> iOS, other embedded oses, BSD*, Solaris, HPUX, etc)
>> - Have more detailed language binding tests for java, ruby, perl, php
>> - Coverage checking
>> - Test parallel processing and multithreading
>> - Test networking (I need to think through isolation)
>> - Test multiple configurations (e.g. all drivers and features
>> enables vrs minimal build).
>> - Check which system calls are used by each driver for read and
>> for write
>> - Check i18n support.
>> - Check distribution packaging
>> - Validating that the given build options result in the expect
>> available features
>>
>> Probably out of scope:
>> - Test for support from older platforms & C++ older than C++11
>> - Actual sandbox checks
>> - Test other bindings to GDAL's C or C++ API such as Fiona & Shapely
>> - Integration tests (e.g. GRASS, QGIS, mapserv, GeoDjango, etc).
>> - ABI compatibility checks
>> - Older versions of dependent libs e.g. netcdf/hdf4/5, kakadu,
>> openjpeg, etc.
>>
>> -kurt
>> Engineer at Google
>>
>>
>> On Sat, Sep 5, 2015 at 7:48 AM, Dmitry Baryshnikov
>> <bishop.dev at gmail.com <mailto:bishop.dev at gmail.com>> wrote:
>>
>> Hi Even,
>>
>> 05.09.2015 17:10, Even Rouault пишет:
>>
>> Dmitry,
>>
>> During the code sprint in FOSS4G 2015 (Korea, Seoul)
>> I plan to start
>> refactoring Cmake for GDAL (everybody are welcome
>> http://2015.foss4g.org/programme/code-sprint/). This
>> is good starting
>> point to try release an idea to reformat source tree
>> (combine drivers on
>> some principles - raster/vector/raster+vector). I
>> digging the mailing
>> list, but didn't found discussion started by Even
>> about this.
>>
>> Regarding unified drivers, it was a bit mentionned in
>> https://trac.osgeo.org/gdal/wiki/rfc46_gdal_ogr_unification
>> . Basically the
>> PCIDSK drivers have been merged in frmts/pcidsk, the PDF
>> ones in frmts/pdf.
>> And the raster side of GPKG has been added to the existing
>> ogr/ogrsf_frmts/geopackage
>> Potential changes on the tree structure were left out in
>> the "Potential
>> changes that are *NOT* included in this RFC" paragraph.
>>
>> I plan to experiment with this and if I get good results, RFC
>> will be written.
>>
>> Also we have
>> new type of drivers - network. So, how it'll be best
>> to organise sources?
>> This can be not only drivers, but the whole source
>> tree. How should the
>> ideal GDAL source tree looks like?
>>
>> Also I plan:
>> 1. Move all internal libraries (zlib, libtiff,
>> libjpeg, etc.) to
>> separate github repos to use CMake ExternalProject
>> feature.
>>
>> Just to give some context: the point for the internal
>> libraries was to have a
>> no-brainer way of building GDAL without any prerequisite.
>> - internal zlib is identical to its upstream v1.2.3 AFAIK
>> - internal libtiff: most files are identical to libtiff
>> CVS, but a few ones
>> (tiffconf.h, tif_config.h) have been modified for
>> integration with GDAL CPL, and
>> tif_vsi.c is GDAL specific (I/O implementation) + a build
>> time hack for TIFF
>> JPEG 12 bit support
>> - internal libjpeg is mostly upstream libjpeg v6b + one
>> patch. There's also
>> the build hack for libjpeg12
>>
>> I only plan to move this internal libraries in separate
>> repos, not to link official ones. So this is only give more
>> structured sources tree.
>>
>>
>> 2. Remove any other building systems
>>
>> That sounds ambitious. Given the complexity and maturity
>> of our current build
>> systems, I guess this would take some time to have CMake
>> catch up.
>>
>> Yes, certainly. But anyhow current CMake branch not fully
>> consistent for current build system. So this have to be done.
>>
>>
>> 3. Try CTest for testing
>>
>> What do you think it will bring w.r.t our current testing
>> system ? Do we want
>> to be dependant of a particular build system for our tests ?
>> Regarding testing, I've somehow understood Kurt had
>> mentionned plans for a
>> "gdalautotest2"
>>
>> This is only subject of experiments. Let's try CTest and see
>> if it fits.
>>
>>
>> Regarding all the above, I assume you mean in a fork of
>> yours ?
>>
>> Yes. All experiments will be on forked GDAL in separate branch.
>>
>>
>> As for me the ideal structure should looks like:
>> + apps
>> + autotests
>> + bindings
>> + core
>> + port
>> + ogr
>> + gcore
>>
>> gnm core would go here too ?
>>
>> Yes
>>
>>
>> + cmake
>> + data
>> + docs
>> + doxygen
>> + readme
>> + drivers
>> + raster
>> + vector
>> + network
>> + combined
>> + CMakeLists.txt
>> + LICENSE
>>
>> So, at the root of sources tree we will have only 8
>> folders and 2 files.
>>
>> Is the churn in the tree structure worth the effort ? Be
>> aware that there are a
>> number of interdependencies between drivers, so this
>> might require fixing a
>> number of source files. What advantages do you see in a
>> new structure ?
>>
>> 1. More ease to understand sources tree for novice.
>> 2. More useful for CMake macro. In current release there are
>> a lot of hardcoded things. Macro give more flexibility.
>> 3. More ease to add some new check needed by separate drivers.
>> 4. More configurable (ease selected depended libraries
>> installed in OS, or should be loaded via ExternalProject),
>> more dependence checks.
>> 5. May be CPack using in future to create distros.
>>
>>
>> I'm afraid that if you want to change multiple things at
>> a time (build system,
>> testing mechanisms, tree structure), you will never
>> manage to get a working
>> result. Incremental approaches when feasible are less
>> risky (but admitedly
>> involve potentially a larger cumulated effort).
>>
>> Yes, you may be right. But it seems to me that current Cmake
>> version is too complicated than it can be. If Ican improve it
>> it'll solve lot of problems, if not - ok this will be only an
>> unsuccessful experiment.
>>
>>
>> Even
>>
>> I do not insist, maybe it's a crazy idea. But as was the
>> discussion of unification, it seemed to me that this worth
>> trying during improvements Cmake build system.
>>
>> Best regards,
>> Dmitry
>>
>>
>
>
>
>
> --
> --
> http://schwehr.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20150915/be50087c/attachment-0001.html>
More information about the gdal-dev
mailing list