<div dir="ltr"><div>(Subject change to focus on testing)<br></div><div><br></div>Hi all,<div><br></div><div>First off... what GDAL has with autotest, travis-ci and coverity is awesome!</div><div><br></div><div><div>Thoughts / discussion more than welcome!</div></div><div><br></div><div>For my production work, I'm not able to use the autotest python code because of its non-unittest architecture. So... I started creating python unittest and C++ gunit based tests. I use autotest2 in Google's internal continuous integration system in our main code base. I'm using Google's build system... I've got nothing started for running the C++ tests outside of Google.<br></div><div><br></div><div>Apologies for not even getting out at least samples of autotest2 for folks to inspect and comment on. My intention is to put what I have in a git repo and the to start discussions as to what (if anything) GDAL community wants to do with autotest2. I was hoping to get a lot more coverage and get GDAL 2.x.x support, but that will have to come later. It's only 14K lines at this point (optimistically 2-3% done), but it has been a huge help for me especially with in upgrading versions of gdal and catching bugs in support libs & development toolchains. </div><div><br></div><div>The tests are more focused on test isolation than autotest. This allows for a lot more parallelism in testing. e.g. It's fair game to run all tests at the same time on the same machine.</div><div><br></div><div><div>find . -name \*.py | xargs wc -l | tail -1</div><div> 10684 total</div><div><br></div><div>find . -name \*.cc -o -name \*.h | xargs wc -l | tail -1<br></div><div> 3734 total</div></div><div><br></div><div>Where GDAL's autotest is 204K lines:</div><div><br></div><div><div>find . -name \*.py | xargs wc -l | tail -1</div><div> 193994 total</div></div><div><div>find . -name \*.c\* -o -name \*.h | xargs wc -l | tail -1</div><div> 10471 total</div></div><div><br></div><div>Here are some samples:</div><div><br></div><div>C++ tests use C++11, gunit, google logging, gflags: (Hoping for C++14 soon.. e.g. make_unique)</div><div>- <a href="https://gist.github.com/schwehr/13137d826763763fb031">autotest2/cpp/port/cpl_conv_test.cc</a> (Yes, this is massively boring code)<br></div><div>- <a href="https://gist.github.com/schwehr/c8ee86a6f6a1c1cc043b">autotest2/cpp/ogr/ogrpoint_test.cc</a></div><div>- <a href="https://gist.github.com/schwehr/bc44b91a37cd621212c4">autotest2/cpp/ogr/ogrsf_frmts/geojson/geojson_test.cc</a></div><div><br></div><div>Python pretty much follows the autotest layout, but with util files in the same directory. Assumes python 2.7 or >= 3.4 (have not tried py3 yet)</div><div>- <a href="https://gist.github.com/schwehr/c143927ca25d03a10265">autotest2/gcore/gcore_util.py</a></div><div>- <a href="https://gist.github.com/schwehr/dd75f73cedf8f7b5357e">autotest2/gdrivers/gdrivers_util.py</a></div><div>- <a href="https://gist.github.com/schwehr/a35b2bc8a7956ef1f620">autotest2/gdrivers/tiff_read_test.py</a> (I'm leading towards moving driver tests in gcore to gdrivers)</div><div>- <a href="https://gist.github.com/schwehr/6cbdc3482055d2237ad2">autotest2/ogr/geojson_test.py</a></div><div><br></div><div>Probably should move python code to also match the C++ tree. e.g.</div><div><br></div><div> tiff_read_test.py -> autotest2/py/frmts/gtiff/tiff_read_test.py<br></div><div><br></div><div>I'm (mostly) following Google's style guides. Public versions here: <a href="https://google-styleguide.googlecode.com/svn/trunk/cppguide.html">C++</a> <a href="https://google-styleguide.googlecode.com/svn/trunk/pyguide.html">Python</a></div><div><br></div><div>All C++ should be formatted with "clang-format --style=Google"</div>
<div><br></div><div>What does autotest2 not do?</div><div><br></div><div>Would like to eventually do (unsorted):</div><div>- Test error handling on a range of corrupt data sources</div><div>- Fuzz testing, ASAN/MSAN/TSAN/Valgrind/Heap checks (I've done some MSAN & heap checkers by hand)</div><div>- Performance testing - time and memory usage</div><div>- Test the C API at the C level</div><div>- Test platforms other than Linux (MS Win*, Mac OSX, Android, iOS, other embedded oses, BSD*, Solaris, HPUX, etc)</div><div>- Have more detailed language binding tests for java, ruby, perl, php</div><div><div>- Coverage checking<br></div></div><div>- Test parallel processing and multithreading</div><div>- Test networking (I need to think through isolation)</div><div>- Test multiple configurations (e.g. all drivers and features enables vrs minimal build).</div><div>- Check which system calls are used by each driver for read and for write</div><div>- Check i18n support.</div><div>- Check distribution packaging</div><div>- Validating that the given build options result in the expect available features</div><div><br></div><div>Probably out of scope: </div><div>- Test for support from older platforms & C++ older than C++11<br></div><div>- Actual sandbox checks</div><div>- Test other bindings to GDAL's C or C++ API such as Fiona & Shapely <br></div><div>- Integration tests (e.g. GRASS, QGIS, mapserv, GeoDjango, etc).</div><div>- ABI compatibility checks</div><div>- Older versions of dependent libs e.g. netcdf/hdf4/5, kakadu, openjpeg, etc.</div><div><br></div><div>-kurt</div><div>Engineer at Google</div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Sep 5, 2015 at 7:48 AM, Dmitry Baryshnikov <span dir="ltr"><<a href="mailto:bishop.dev@gmail.com" target="_blank">bishop.dev@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi Even,<br>
<br>
05.09.2015 17:10, Even Rouault пишет:<span class=""><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Dmitry,<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
During the code sprint in FOSS4G 2015 (Korea, Seoul) I plan to start<br>
refactoring Cmake for GDAL (everybody are welcome<br>
<a href="http://2015.foss4g.org/programme/code-sprint/" rel="noreferrer" target="_blank">http://2015.foss4g.org/programme/code-sprint/</a>). This is good starting<br>
point to try release an idea to reformat source tree (combine drivers on<br>
some principles - raster/vector/raster+vector). I digging the mailing<br>
list, but didn't found discussion started by Even about this.<br>
</blockquote>
Regarding unified drivers, it was a bit mentionned in<br>
<a href="https://trac.osgeo.org/gdal/wiki/rfc46_gdal_ogr_unification" rel="noreferrer" target="_blank">https://trac.osgeo.org/gdal/wiki/rfc46_gdal_ogr_unification</a> . Basically the<br>
PCIDSK drivers have been merged in frmts/pcidsk, the PDF ones in frmts/pdf.<br>
And the raster side of GPKG has been added to the existing<br>
ogr/ogrsf_frmts/geopackage<br>
Potential changes on the tree structure were left out in the "Potential<br>
changes that are *NOT* included in this RFC" paragraph.<br>
</blockquote></span>
I plan to experiment with this and if I get good results, RFC will be written.<span class=""><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Also we have<br>
new type of drivers - network. So, how it'll be best to organise sources?<br>
This can be not only drivers, but the whole source tree. How should the<br>
ideal GDAL source tree looks like?<br>
<br>
Also I plan:<br>
1. Move all internal libraries (zlib, libtiff, libjpeg, etc.) to<br>
separate github repos to use CMake ExternalProject feature.<br>
</blockquote>
Just to give some context: the point for the internal libraries was to have a<br>
no-brainer way of building GDAL without any prerequisite.<br>
- internal zlib is identical to its upstream v1.2.3 AFAIK<br>
- internal libtiff: most files are identical to libtiff CVS, but a few ones<br>
(tiffconf.h, tif_config.h) have been modified for integration with GDAL CPL, and<br>
tif_vsi.c is GDAL specific (I/O implementation) + a build time hack for TIFF<br>
JPEG 12 bit support<br>
- internal libjpeg is mostly upstream libjpeg v6b + one patch. There's also<br>
the build hack for libjpeg12<br>
</blockquote></span>
I only plan to move this internal libraries in separate repos, not to link official ones. So this is only give more structured sources tree.<span class=""><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
2. Remove any other building systems<br>
</blockquote>
That sounds ambitious. Given the complexity and maturity of our current build<br>
systems, I guess this would take some time to have CMake catch up.<br>
</blockquote></span>
Yes, certainly. But anyhow current CMake branch not fully consistent for current build system. So this have to be done.<span class=""><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
3. Try CTest for testing<br>
</blockquote>
What do you think it will bring w.r.t our current testing system ? Do we want<br>
to be dependant of a particular build system for our tests ?<br>
Regarding testing, I've somehow understood Kurt had mentionned plans for a<br>
"gdalautotest2"<br>
</blockquote></span>
This is only subject of experiments. Let's try CTest and see if it fits.<span class=""><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Regarding all the above, I assume you mean in a fork of yours ?<br>
</blockquote></span>
Yes. All experiments will be on forked GDAL in separate branch.<span class=""><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
As for me the ideal structure should looks like:<br>
+ apps<br>
+ autotests<br>
+ bindings<br>
+ core<br>
+ port<br>
+ ogr<br>
+ gcore<br>
</blockquote>
gnm core would go here too ?<br>
</blockquote></span>
Yes<span class=""><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
+ cmake<br>
+ data<br>
+ docs<br>
+ doxygen<br>
+ readme<br>
+ drivers<br>
+ raster<br>
+ vector<br>
+ network<br>
+ combined<br>
+ CMakeLists.txt<br>
+ LICENSE<br>
<br>
So, at the root of sources tree we will have only 8 folders and 2 files.<br>
</blockquote>
Is the churn in the tree structure worth the effort ? Be aware that there are a<br>
number of interdependencies between drivers, so this might require fixing a<br>
number of source files. What advantages do you see in a new structure ?<br>
</blockquote></span>
1. More ease to understand sources tree for novice.<br>
2. More useful for CMake macro. In current release there are a lot of hardcoded things. Macro give more flexibility.<br>
3. More ease to add some new check needed by separate drivers.<br>
4. More configurable (ease selected depended libraries installed in OS, or should be loaded via ExternalProject), more dependence checks.<br>
5. May be CPack using in future to create distros.<span class=""><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
I'm afraid that if you want to change multiple things at a time (build system,<br>
testing mechanisms, tree structure), you will never manage to get a working<br>
result. Incremental approaches when feasible are less risky (but admitedly<br>
involve potentially a larger cumulated effort).<br>
</blockquote></span>
Yes, you may be right. But it seems to me that current Cmake version is too complicated than it can be. If Ican improve it it'll solve lot of problems, if not - ok this will be only an unsuccessful experiment.<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Even<br>
<br>
</blockquote>
I do not insist, maybe it's a crazy idea. But as was the discussion of unification, it seemed to me that this worth trying during improvements Cmake build system.<br>
<br>
Best regards,<br>
Dmitry<div class=""><div class="h5"></div></div></blockquote></div><div class="gmail_signature"><br></div>
</div></div>