<div dir="ltr">Mateusz,<div><br></div><div>I'll try to do my best to answer inline. What we have with autotest is super important. I've been trying to figure out how we take the first 17 years of knowledge built up in GDAL and make keep going forward over the next couple decades. What will help people contribute to GDAL in the future? I started autotest2 with the assumption that the GDAL community might not be interested in including it in the main GDAL tree, may want to restructure it and include it, or might only want some pieces. I'm fine with any of that. It's been really helpful for me so far.</div><div><br></div><div>Not sure these are the best answers, but here is what I've written off the top of my head. I think I'm being a bit redundant.<br><div class="gmail_extra"><br></div><div class="gmail_extra">However, above everything else, we can't give up the coverage that we currently have. It's a lot and it's taken a huge effort to get to this point, but I think it is not nearly enough. I think that means that autotest in it's current form will be with us for the foreseeable future. Nor is it enough to just get the Coverity issue count to 0.</div><div class="gmail_extra"><br></div><div class="gmail_extra">-kurt</div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Oct 16, 2015 at 6:01 AM, Mateusz Loskot <span dir="ltr"><<a href="mailto:mateusz@loskot.net" target="_blank">mateusz@loskot.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi Kurt,<br>
<br>
I'm interested in this topic, so I'd like to pull some more details to<br>
better<br>
understand the idea, if you don't mind.<br>
<br>
<br>
Kurt Schwehr-2 wrote<br>
<span>> For my production work, I'm not able to use the autotest python code<br>
> because of its non-unittest architecture.<br>
<br>
</span>Could you elaborate more on what properties exactly are missing<br>
from the current autotest suite?<br>
IOW, what kind of gap autotest2 fills.<br>
<br></blockquote><div><br></div><div>Things that are missing from autotest python code:</div><div><br></div><div>- By not being python unittest/unittest2/mock based, a lot of tools don't know how to discover the tests. <a href="https://pypi.python.org/pypi/nose/" target="_blank">nose</a>, <a href="http://pytest.org/" target="_blank">pytest</a>, and lots of folks have their own test runners. e.g. Google has the internal version of Blaze: Bazel. I tried writing a thin unittest wrapper and it turned into a mess. I'm sure it can be done, but the env I work in is super constrained. And the test reporting with unittest is a lot stronger than what we have with autotest.</div><div>- It has a syntax that is unfamiliar to the average python coder. The junit style is super common, so there are more folks available who really know unittest based tools.</div><div>- There is a lot of well developed utility in the the unittest/unittest2 infrastructure that is very robust</div><div>- Isolated tests. There are a lot of tests that cascade. That makes running tests in parallel much more difficult</div><div>- The python mocks library is pretty great for making tests of things like databases. We could have standard mocks for database interactions that would help folks test their python code that uses gdal+{mysql,postgis,mssql,oracle,etc}. That could really accelerate development for many people.</div><div>- unittest and other related tools have nice ways to allow running subsets of tests</div><div><br></div><div>Python unittest is often all of the things you describe. It doesn't do well with UI and integration testing of super different libraries, but it is just as capable as what is currently in autotest.<br></div><div><br></div><div>It's worth taking a look at some examples. e.g. <a href="https://github.com/Toblerity/Fiona/tree/master/tests" target="_blank">Fiona's tests</a></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Yes, autotest is not a unit testing suite.<br>
To me, it combines functional/integration/blackbox/system/regression<br>
testing.<br>
<br>
<br>
Kurt Schwehr-2 wrote<br>
<span>> So... I started creating python<br>
> unittest and C++ gunit based tests.<br>
<br>
</span>Since you mention unittest, then gmock too?<br>
<br></blockquote><div><br></div><div>Yes, gunit and gmock are usually used together.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Kurt Schwehr-2 wrote<br>
<span>> The tests are more focused on test isolation than autotest. This allows<br>
> for a lot more parallelism in testing.<br>
<br>
</span>Isolation at what level?<br>
I suppose, you mean test case isolation, but not isolation of code/unit<br>
under test.<br></blockquote><div><br></div><div>Both types of isolation would be nice. GDAL isn't designed for isolated component testing of the drivers. I've mostly been able to do isolated testing on ports/cpl_*. The biggest isolation issue is between each test function/method. Using the same filenames or using global variables and having one test create a file and then follow on tests use that file. Some of these show up like this:</div><div><br></div><div>cd autotest</div><div><div>egrep 'gdaltest.[a-z]+_ds =' */*.py | grep -v None | head</div><div>gdrivers/fast.py: gdaltest.fast_ds = gdal.Open( 'data/L71118038_03820020111_HPN.FST' )</div><div>gdrivers/georaster.py: gdaltest.oci_ds = ogr.Open( os.environ.get('OCI_DSNAME') )</div><div>gdrivers/mem.py: gdaltest.mem_ds = drv.Create( 'mem_1.mem', 50, 3 )</div><div>gdrivers/pcidsk.py: gdaltest.pcidsk_ds = driver.Create( 'tmp/pcidsk_5.pix', 400, 600, 1,</div><div>gdrivers/pcidsk.py: gdaltest.pcidsk_ds = gdal.Open( 'tmp/pcidsk_5.pix', gdal.GA_Update )</div><div>gdrivers/pcidsk.py: gdaltest.pcidsk_ds = gdal.Open( 'tmp/pcidsk_5.pix', gdal.GA_Update )</div><div>gdrivers/pcidsk.py: gdaltest.pcidsk_ds = gdal.Open( 'tmp/pcidsk_5.pix', gdal.GA_Update )</div><div>gdrivers/vrtrawlink.py: gdaltest.rawlink_ds = gdal.Open( 'tmp/rawlink.vrt', gdal.GA_Update )</div><div>gdrivers/vrtrawlink.py: gdaltest.rawlink_ds = gdal.Open( 'tmp/rawlink.vrt', gdal.GA_Update )</div><div>gdrivers/vrtwarp.py: gdaltest.vrtwarp_ds = gdal.AutoCreateWarpedVRT( gcp_ds )</div></div><div><br></div><div><div>egrep 'gdaltest.[a-z]+_ds =' */*.py | grep -v None | tail</div><div>ogr/ogr_sqlite.py: gdaltest.sl_ds = ogr.Open( 'tmp/sqlite_test.db', update = 1 )</div><div>ogr/ogr_sqlite.py: gdaltest.sl_ds = ogr.Open( 'tmp/sqlite_test.db' )</div><div>ogr/ogr_sqlite.py: gdaltest.sl_ds = ogr.Open( 'tmp/sqlite_test.db', update = 1 )</div><div>ogr/ogr_sqlite.py: gdaltest.sl_ds = ogr.Open( 'tmp/sqlite_test.db', update = 1 )</div><div>ogr/ogr_svg.py: gdaltest.svg_ds = ogr.Open( 'data/test.svg' )</div><div>ogr/ogr_sxf.py: gdaltest.sxf_ds = ogr.Open( 'data/100_test.sxf' )</div><div>ogr/ogr_vfk.py: gdaltest.vfk_ds = ogr.Open('data/bylany.vfk')</div><div>ogr/ogr_vfk.py: gdaltest.vfk_ds = ogr.Open('data/bylany.vfk')</div><div>ogr/ogr_vrt.py: gdaltest.vrt_ds = ogr.Open( 'data/vrt_test.vrt' )</div><div>ogr/ogr_wasp.py: gdaltest.wasp_ds = wasp_drv.CreateDataSource( 'tmp.map' )</div></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
So, the former means each test case runs in isolated environment,<br>
with exclusive access to files/databases/memory/resources required by<br>
particular test.<br>
For example, each run of ogr_mysql.py, also in parallel, exclusively targets<br>
a 'unique' database.<br>
<br>
Correct?<br>
<br></blockquote><div><br></div><div>Correct. Or that it is possible to enable isolation.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Kurt Schwehr-2 wrote<br>
<span>> Here are some samples:<br>
><br>
> C++ tests use C++11, gunit, google logging, gflags: (Hoping for C++14<br>
> soon.. e.g. make_unique)<br>
> - autotest2/cpp/port/cpl_conv_test.cc<br>
</span>> <<a href="https://gist.github.com/schwehr/13137d826763763fb031>" rel="noreferrer" target="_blank">https://gist.github.com/schwehr/13137d826763763fb031></a>; (Yes, this is<br>
> massively boring code)<br>
<br>
AFAIS, architecture of tests won't change much, typically, test cases will<br>
target functional modules (drivers).<br>
Or, do you plan for proper unit tests targeting individual<br>
interfaces/classes<br>
in internal implementation of drivers, etc.?<br></blockquote><div><br></div><div>Looking at drivers, I think the level and type of testing that is appropriate varies quite a bit. Some drivers are super simple and some are very complex. In general, extensive testing of internals isn't super important. What we do need a ton more of is testing the failure paths. e.g. I took AFL and ran it against ogr2ogr and a custom little loader program to read geojson. GDAL code should never crash or hang, but it most definitely does. We need to turn those input files into test cases. </div><div><br></div><div>We also can use AFL to generate test cases that cover more of the failure modes. I did that with minixml: <a href="https://trac.osgeo.org/gdal/changeset/30854/" target="_blank">r30854</a></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
<br>
Kurt Schwehr-2 wrote<br>
<span>> I'm (mostly) following Google's style guides.<br>
</span><span>> All C++ should be formatted with "clang-format --style=Google"<br>
<br>
</span>Unless it is not planned to include autotest2 into GDAL source code,<br>
shouldn't it follow GDAL style?<br><br></blockquote><div><br></div><div>Not necessarily. I'm working for Google and starting from scratch with Google engineers doing the reviews, so I'm using the Google Style Guides. The style for GDAL isn't what users of the library are likely to do. A part of what I'm trying to do with autotest2 is also give examples of what I think are good ways to use GDAL. People are likely going to use C++11/14 and Hungarian-ish notation is not something I've seen in any other project I've worked on. The Google style guide is pretty well flushed out, where as RFC 8 for GDAL is very minimalistic. I prefer to work in a space where I can just follow.</div><div> </div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Kurt Schwehr-2 wrote<br>
<span>> Would like to eventually do (unsorted):<br>
</span><span>> - Fuzz testing, ASAN/MSAN/TSAN/Valgrind/Heap checks (I've done some MSAN<br>
> &<br>
> heap checkers by hand)<br>
> - Performance testing - time and memory usage<br>
<br>
</span>IMHO, those belong to separate suites.<br>
<span><br></span></blockquote><div><br></div><div>Why separate the memory and thread tests? If you write really solid unittests in C++, you can flip on or off those modes. As long as your unittests properly cleanup, then they are great code to use for ASAN/MSAN/etc. For fuzzing, it is definitely helpful to isolate things like the geojson driver from all the mechanics of which driver to use. We'd like to be able to pass anything to the driver directly to trigger code paths. For performance, it really is a challenge and likely going to be challenging to figure out what is worth doing.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span>
<br>
- Test the C API at the C level<br>
</span></qoute><br></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
If plain unittests are planned, mocking might be possible.<br></blockquote><div><br></div><div>What do you mean by this?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Best regards,<br>
Mateusz<br>
<br>
<br>
<br>
--<br>
View this message in context: <a href="http://osgeo-org.1560.x6.nabble.com/Re-GDAL-testing-tp5222783p5230596.html" rel="noreferrer" target="_blank">http://osgeo-org.1560.x6.nabble.com/Re-GDAL-testing-tp5222783p5230596.html</a><br>
Sent from the GDAL - Dev mailing list archive at Nabble.com.<br><br></blockquote></div><div><br></div>
</div></div></div>