<div dir="ltr">I posted the link earlier in the thread, but to clarify: <a href="https://github.com/OSGeo/gdal/pull/963">https://github.com/OSGeo/gdal/pull/963</a><br></div><br><div class="gmail_quote"><div dir="ltr">On Fri, 26 Oct 2018 at 14:54 Craig de Stigter <<a href="mailto:craig.destigter@koordinates.com">craig.destigter@koordinates.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>My work on this branch is starting to wind down. There's still a few test failures to contend with, and I need to work through some inconsistencies in the different CI environments.</div><div><br></div><div>Could I request some review of the PR, especially around scope and overall approach? I believe the PR description is reasonably useful to describe what I've been up to, but please ask questions if you have them, or comment/review on the PR itself.</div><div><br></div><div>Thanks</div></div><div dir="ltr"><div>Craig de Stigter<br></div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, 5 Oct 2018 at 22:50 Kurt Schwehr <<a href="mailto:schwehr@gmail.com" target="_blank">schwehr@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">The python tests in autotest2 are pretty much a subset of GDAL's autotest. The C++ and Java tests are mostly outside of what autotest has.</div><br><div class="gmail_quote"><div dir="ltr">On Tue, Oct 2, 2018 at 2:05 AM Craig de Stigter <<a href="mailto:craig.destigter@koordinates.com" target="_blank">craig.destigter@koordinates.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Thanks. Are those tests entirely based on the osgeo git tests, or are there some extra ones? It might be worth getting the extra ones in at least.</div><br></div><br><div class="gmail_quote"><div dir="ltr">On Tue, 2 Oct 2018 at 18:06, Kurt Schwehr <<a href="mailto:schwehr@gmail.com" target="_blank">schwehr@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I am definitely interested in improvements to the test infrastructure, but always fear slowing down Even or getting bogged down in a massive effort.<div><br></div><div>In addition to working in the normal osgeo github world...</div><div><br></div><div>My current state: I started "<a href="https://github.com/schwehr/gdal-autotest2/tree/master/python" target="_blank">autotest2</a>" back in 2013-14 after being unable to make autotest work for me and released much of it as opensource. It's unittest based for python and GoogleTest/gmock for c++. The Java isn't very far along, but it's Truth + JUnit4. It's in use inside google typically running every 10 to 60 minutes depending on what's going on and a couple times a day in ASAN, MSAN, and TSAN modes. On the python side, it doesn't cover that much compared to the full python test suite in autotest. But in doing rewrites, I did have to make each test independent. I also have to allow control of where tests write temporary data as I work in a system where the source tree is read only. I am probably close to making the tests python 3.7+ only (but likely to work on 3.5+). Whenever I start using a driver for the first time, I try to rewrite the python test and create at least basic C++ tests, but I never have enough time to do as complete of a job as I would like (err... like the partially done pull request I have in to Proj).</div><div><br></div><div>You are welcome to use anything in autotest2 for the core gdal tree if it fits. I (meaning Google) are happy to contribute any or all of the code to the GDAL tree under the GDAL license.</div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Oct 1, 2018 at 3:32 PM Craig de Stigter <<a href="mailto:craig.destigter@koordinates.com" target="_blank">craig.destigter@koordinates.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi folks</div><div><br></div><div>I've been looking at a potential large change to the GDAL test suite, replacing some of the worst parts. I thought I'd bring it up for discussion here, in case anyone has any wisdom or strong feelings about it.</div><div><br></div><div>I've made a ticket at <a href="https://github.com/OSGeo/gdal/issues/949" target="_blank">https://github.com/OSGeo/gdal/issues/949</a> . <br></div><div><br></div><div>Essentially my plan is to use <a href="https://docs.pytest.org/en/latest/" target="_blank">pytest</a> as both a test runner and a helper library. We would remove or replace large parts of the `gdaltest` utilities with pytest equivalents.</div><div><br></div><div>To give an idea of the main changes, this code:</div><div><br></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="font-family:monospace,monospace">if ds.GetLayer(0).GetSpatialRef() is not None:<br> gdaltest.post_reason('did not get expected spatial ref')<br> return 'fail'</span><br></blockquote></div><div>would change to:</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><span style="font-family:monospace,monospace">assert ds.GetLayer(0).GetSpatialRef() is None, 'did not get expected spatial ref'</span></div></blockquote><div><br></div><div>The pytest runner gives contextual tracebacks from assert statements, as well as relevant variables and line numbers.</div><div><br></div><div>Other changes:</div><div> * The `gdaltest_list` and the `__main__` entry point at the bottom of each file will be removed. Some tests will be renamed if they don't already start with `test_`.</div><div> * The `<name>_cleanup` fixture will be turned into a <a href="https://docs.pytest.org/en/latest/fixture.html#scope-sharing-a-fixture-instance-across-tests-in-a-class-module-or-session" target="_blank">module-scoped fixture</a>. This allows users to run any number of tests in the module, and the cleanup fixture will be invoked once after all of them are finished. If I find corresponding setup functions I will make them into fixtures too, but I haven't seen those so far in my digging.<br></div><div><div><br></div><div>Other,
more intrusive changes are possible but this is what I'm considering targeting as a
first step. It is fairly straightforward to automate this change; I have a script in progress already to do that.<br></div><div><br></div><div>I intend to make a compatibility shim for the old-style tests that can't easily be automatically updated. The shim would be a decorator, perhaps called `@old_test`, which would take 'skip' or 'fail' return values from tests and turn them into the appropriate `assert False` or `pytest.skip()` calls.<br></div></div><div><br></div><div>I don't intend to make any changes to the C++ tests.</div><div><br></div><div>A couple of complicating factors to be aware of:</div><div><ul><li>pytest only supports python 2.7+. The tests already only run in 2.7+ in Travis, so no big deal there. I think it's perfectly reasonable to only support 2.7+ these days, given that 2.6 has been unsupported since 2013, but I thought I'd mention it.</li><li>pytest has no builtin way of <i>ordering</i> tests. It is assumed that tests are independent of each other. In practice, all tests are run in alphabetical name order within a given module. It appears to me that some of GDAL's tests are <i>not</i> independent of each other within the same module. However, most are already in alphabetical name order, so I don't believe this is a problem; we are not worse off with pytest than previously.<br></li></ul></div></div><div dir="ltr"><br></div><div>Please let me know your thoughts.<br></div><div dir="ltr"><div><br>-- <br><div dir="ltr" class="m_7947248856314831719m_-1835412017800768411m_6373695736397912728m_-2186695064264496504m_-6758934610287997101gmail_signature"><div dir="ltr"><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px">Regards,</div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px">Craig de Stigter<br></div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px"><br></div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px">Developer</div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px">Koordinates</div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px"><br></div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px"><a href="tel:+64%2021%20256%209488" style="color:rgb(17,85,204)" target="_blank">+64 21 256 9488</a> / <a href="http://koordinates.com/" style="color:rgb(17,85,204)" target="_blank">koordinates.com</a> / <a href="https://twitter.com/koordinates" style="color:rgb(17,85,204)" target="_blank">@koordinates</a></div></div></div></div></div></div></div></div></div>
_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_7947248856314831719m_-1835412017800768411m_6373695736397912728m_-2186695064264496504gmail_signature" data-smartmail="gmail_signature">--<div><a href="http://schwehr.org" target="_blank">http://schwehr.org</a></div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="m_7947248856314831719m_-1835412017800768411m_6373695736397912728gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px">Regards,</div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px">Craig</div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px"><br></div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px">Developer</div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px">Koordinates</div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px"><br></div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px"><a href="tel:+64%2021%20256%209488" style="color:rgb(17,85,204)" target="_blank">+64 21 256 9488</a> / <a href="http://koordinates.com/" style="color:rgb(17,85,204)" target="_blank">koordinates.com</a> / <a href="https://twitter.com/koordinates" style="color:rgb(17,85,204)" target="_blank">@koordinates</a></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_7947248856314831719m_-1835412017800768411gmail_signature" data-smartmail="gmail_signature">--<div><a href="http://schwehr.org" target="_blank">http://schwehr.org</a></div></div>
</blockquote></div>
</blockquote></div>