[gdal-dev] GDAL testing
Dmitry Baryshnikov
bishop.dev at gmail.com
Sun Sep 6 08:27:09 PDT 2015
Hi Kurt,
During code sprint in Korea (FOSS4G 2015) I plan to play with new
approach of CMake fro GDAL. The one of experiments will be try to use
CTest. As I plan restructure the sources tree, I can try to integrate
you work on autotest2 and CTest. Also we can try to create new test
directory structure more compatible for test and sources tree (this is
about you wrote: Probably should move python code to also match the C++
tree. e.g. tiff_read_test.py ->
autotest2/py/frmts/gtiff/tiff_read_test.py).
How can get you tests? What do I need to use autotest2?
Best regards,
Dmitry
05.09.2015 20:37, Kurt Schwehr пишет:
> (Subject change to focus on testing)
>
> Hi all,
>
> First off... what GDAL has with autotest, travis-ci and coverity is
> awesome!
>
> Thoughts / discussion more than welcome!
>
> For my production work, I'm not able to use the autotest python code
> because of its non-unittest architecture. So... I started creating
> python unittest and C++ gunit based tests. I use autotest2 in
> Google's internal continuous integration system in our main code
> base. I'm using Google's build system... I've got nothing started for
> running the C++ tests outside of Google.
>
> Apologies for not even getting out at least samples of autotest2 for
> folks to inspect and comment on. My intention is to put what I have
> in a git repo and the to start discussions as to what (if anything)
> GDAL community wants to do with autotest2. I was hoping to get a
> lot more coverage and get GDAL 2.x.x support, but that will have to
> come later. It's only 14K lines at this point (optimistically 2-3%
> done), but it has been a huge help for me especially with in upgrading
> versions of gdal and catching bugs in support libs & development
> toolchains.
>
> The tests are more focused on test isolation than autotest. This
> allows for a lot more parallelism in testing. e.g. It's fair game to
> run all tests at the same time on the same machine.
>
> find . -name \*.py | xargs wc -l | tail -1
> 10684 total
>
> find . -name \*.cc -o -name \*.h | xargs wc -l | tail -1
> 3734 total
>
> Where GDAL's autotest is 204K lines:
>
> find . -name \*.py | xargs wc -l | tail -1
> 193994 total
> find . -name \*.c\* -o -name \*.h | xargs wc -l | tail -1
> 10471 total
>
> Here are some samples:
>
> C++ tests use C++11, gunit, google logging, gflags: (Hoping for
> C++14 soon.. e.g. make_unique)
> - autotest2/cpp/port/cpl_conv_test.cc
> <https://gist.github.com/schwehr/13137d826763763fb031> (Yes, this is
> massively boring code)
> - autotest2/cpp/ogr/ogrpoint_test.cc
> <https://gist.github.com/schwehr/c8ee86a6f6a1c1cc043b>
> - autotest2/cpp/ogr/ogrsf_frmts/geojson/geojson_test.cc
> <https://gist.github.com/schwehr/bc44b91a37cd621212c4>
>
> Python pretty much follows the autotest layout, but with util files in
> the same directory. Assumes python 2.7 or >= 3.4 (have not tried py3 yet)
> - autotest2/gcore/gcore_util.py
> <https://gist.github.com/schwehr/c143927ca25d03a10265>
> - autotest2/gdrivers/gdrivers_util.py
> <https://gist.github.com/schwehr/dd75f73cedf8f7b5357e>
> - autotest2/gdrivers/tiff_read_test.py
> <https://gist.github.com/schwehr/a35b2bc8a7956ef1f620> (I'm leading
> towards moving driver tests in gcore to gdrivers)
> - autotest2/ogr/geojson_test.py
> <https://gist.github.com/schwehr/6cbdc3482055d2237ad2>
>
> Probably should move python code to also match the C++ tree. e.g.
>
> tiff_read_test.py -> autotest2/py/frmts/gtiff/tiff_read_test.py
>
> I'm (mostly) following Google's style guides. Public versions here:
> C++ <https://google-styleguide.googlecode.com/svn/trunk/cppguide.html>
> Python <https://google-styleguide.googlecode.com/svn/trunk/pyguide.html>
>
> All C++ should be formatted with "clang-format --style=Google"
>
> What does autotest2 not do?
>
> Would like to eventually do (unsorted):
> - Test error handling on a range of corrupt data sources
> - Fuzz testing, ASAN/MSAN/TSAN/Valgrind/Heap checks (I've done some
> MSAN & heap checkers by hand)
> - Performance testing - time and memory usage
> - Test the C API at the C level
> - Test platforms other than Linux (MS Win*, Mac OSX, Android, iOS,
> other embedded oses, BSD*, Solaris, HPUX, etc)
> - Have more detailed language binding tests for java, ruby, perl, php
> - Coverage checking
> - Test parallel processing and multithreading
> - Test networking (I need to think through isolation)
> - Test multiple configurations (e.g. all drivers and features enables
> vrs minimal build).
> - Check which system calls are used by each driver for read and for write
> - Check i18n support.
> - Check distribution packaging
> - Validating that the given build options result in the expect
> available features
>
> Probably out of scope:
> - Test for support from older platforms & C++ older than C++11
> - Actual sandbox checks
> - Test other bindings to GDAL's C or C++ API such as Fiona & Shapely
> - Integration tests (e.g. GRASS, QGIS, mapserv, GeoDjango, etc).
> - ABI compatibility checks
> - Older versions of dependent libs e.g. netcdf/hdf4/5, kakadu,
> openjpeg, etc.
>
> -kurt
> Engineer at Google
>
>
> On Sat, Sep 5, 2015 at 7:48 AM, Dmitry Baryshnikov
> <bishop.dev at gmail.com <mailto:bishop.dev at gmail.com>> wrote:
>
> Hi Even,
>
> 05.09.2015 17:10, Even Rouault пишет:
>
> Dmitry,
>
> During the code sprint in FOSS4G 2015 (Korea, Seoul) I
> plan to start
> refactoring Cmake for GDAL (everybody are welcome
> http://2015.foss4g.org/programme/code-sprint/). This is
> good starting
> point to try release an idea to reformat source tree
> (combine drivers on
> some principles - raster/vector/raster+vector). I digging
> the mailing
> list, but didn't found discussion started by Even about this.
>
> Regarding unified drivers, it was a bit mentionned in
> https://trac.osgeo.org/gdal/wiki/rfc46_gdal_ogr_unification .
> Basically the
> PCIDSK drivers have been merged in frmts/pcidsk, the PDF ones
> in frmts/pdf.
> And the raster side of GPKG has been added to the existing
> ogr/ogrsf_frmts/geopackage
> Potential changes on the tree structure were left out in the
> "Potential
> changes that are *NOT* included in this RFC" paragraph.
>
> I plan to experiment with this and if I get good results, RFC will
> be written.
>
> Also we have
> new type of drivers - network. So, how it'll be best to
> organise sources?
> This can be not only drivers, but the whole source tree.
> How should the
> ideal GDAL source tree looks like?
>
> Also I plan:
> 1. Move all internal libraries (zlib, libtiff, libjpeg,
> etc.) to
> separate github repos to use CMake ExternalProject feature.
>
> Just to give some context: the point for the internal
> libraries was to have a
> no-brainer way of building GDAL without any prerequisite.
> - internal zlib is identical to its upstream v1.2.3 AFAIK
> - internal libtiff: most files are identical to libtiff CVS,
> but a few ones
> (tiffconf.h, tif_config.h) have been modified for integration
> with GDAL CPL, and
> tif_vsi.c is GDAL specific (I/O implementation) + a build time
> hack for TIFF
> JPEG 12 bit support
> - internal libjpeg is mostly upstream libjpeg v6b + one patch.
> There's also
> the build hack for libjpeg12
>
> I only plan to move this internal libraries in separate repos, not
> to link official ones. So this is only give more structured
> sources tree.
>
>
> 2. Remove any other building systems
>
> That sounds ambitious. Given the complexity and maturity of
> our current build
> systems, I guess this would take some time to have CMake catch up.
>
> Yes, certainly. But anyhow current CMake branch not fully
> consistent for current build system. So this have to be done.
>
>
> 3. Try CTest for testing
>
> What do you think it will bring w.r.t our current testing
> system ? Do we want
> to be dependant of a particular build system for our tests ?
> Regarding testing, I've somehow understood Kurt had mentionned
> plans for a
> "gdalautotest2"
>
> This is only subject of experiments. Let's try CTest and see if it
> fits.
>
>
> Regarding all the above, I assume you mean in a fork of yours ?
>
> Yes. All experiments will be on forked GDAL in separate branch.
>
>
> As for me the ideal structure should looks like:
> + apps
> + autotests
> + bindings
> + core
> + port
> + ogr
> + gcore
>
> gnm core would go here too ?
>
> Yes
>
>
> + cmake
> + data
> + docs
> + doxygen
> + readme
> + drivers
> + raster
> + vector
> + network
> + combined
> + CMakeLists.txt
> + LICENSE
>
> So, at the root of sources tree we will have only 8
> folders and 2 files.
>
> Is the churn in the tree structure worth the effort ? Be aware
> that there are a
> number of interdependencies between drivers, so this might
> require fixing a
> number of source files. What advantages do you see in a new
> structure ?
>
> 1. More ease to understand sources tree for novice.
> 2. More useful for CMake macro. In current release there are a lot
> of hardcoded things. Macro give more flexibility.
> 3. More ease to add some new check needed by separate drivers.
> 4. More configurable (ease selected depended libraries installed
> in OS, or should be loaded via ExternalProject), more dependence
> checks.
> 5. May be CPack using in future to create distros.
>
>
> I'm afraid that if you want to change multiple things at a
> time (build system,
> testing mechanisms, tree structure), you will never manage to
> get a working
> result. Incremental approaches when feasible are less risky
> (but admitedly
> involve potentially a larger cumulated effort).
>
> Yes, you may be right. But it seems to me that current Cmake
> version is too complicated than it can be. If Ican improve it
> it'll solve lot of problems, if not - ok this will be only an
> unsuccessful experiment.
>
>
> Even
>
> I do not insist, maybe it's a crazy idea. But as was the
> discussion of unification, it seemed to me that this worth trying
> during improvements Cmake build system.
>
> Best regards,
> Dmitry
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20150906/bd5f7df4/attachment-0001.html>
More information about the gdal-dev
mailing list