<div dir="ltr"><div>On Fri, Aug 8, 2014 at 8:29 AM, Sören Gebbert <<a href="mailto:soerengebbert@googlemail.com">soerengebbert@googlemail.com</a>> wrote:<br>><br>><br>> * Please consider to integrate your benchmark suite into the GRASS<br>
> testing framework GSoC project [1] which already includes runtime<br>> analysis.<br>><br>Hi Jachym,<br><br>that's great that you are doing this and that you want to standardize it. I'm not sure what exactly Soeren means by "already includes runtime analysis" because testing framework does not contain anything like that yet (not counting timing included in PyGRASS [2]) but there is certainly an overlap between benchmarking and testing.<br>
<br>First, we plan to include timing into the testing framework, so that when you call a module you get also time as a part of (test) report. Some other things are also interesting, namely profiling, memory check (valgrind), coverage and perhaps some other things from clang/LLVM world [3]. I think that profiling and timing are surely the overlap between testing framework and benchmarking framework/suite. The idea, for testing, currently is that the measurement of module execution would be done in `assertModule()` function (which executes the module) [4] and the results will be collected for each test.<br>
<br>Second, and I was not much thinking about this yet, benchmark can has a very similar structure to test: preparation, benchmark/test and cleanup. I don't think that they should be mixed because benchmarks would slow down tests significantly but maybe the system can be the same. On the other hand, there is certainly a large areas in benchmarking API, implementation and report generation which are very different from testing, for example a graph of cell count on X axis and time on Y axis and everything what is needed to get this graph. Also note that some projects actually monitor time of test execution and compare current state to the previous one to warn if there was some slow down (this is applicable for both testing and benchmarking).<br>
<br>Anyway, once the ideas and code are stabilized and relation to testing framework is solved, this is something I would like to see in GRASS core.<br><br>2014-08-08 13:00 GMT+02:00 Jachym Cepicky <<a href="mailto:jachym.cepicky@gmail.com">jachym.cepicky@gmail.com</a>>:<br>
> I started with the dataset from Martin, but I thing, I'll switch<br>> over to spearfish, one resolution for now.<br><br></div>Please, switch to NC SPM, this is the dataset/location which is (should be) used in manual pages now and which is partially used also by tests.<br>
<div><br></div><div>Best,<br></div><div>Vaclav<br></div><div><br>[1] <a href="http://trac.osgeo.org/grass/wiki/GSoC/2014/TestingFrameworkForGRASS">http://trac.osgeo.org/grass/wiki/GSoC/2014/TestingFrameworkForGRASS</a><br>
[2] <a href="http://trac.osgeo.org/grass/browser/grass/trunk/lib/python/pygrass/modules/interface/module.py#L537">http://trac.osgeo.org/grass/browser/grass/trunk/lib/python/pygrass/modules/interface/module.py#L537</a><br>
[3] <a href="http://trac.osgeo.org/grass/wiki/GSoC/2014/TestingFrameworkForGRASS#AnalyzingmodulerunusingValgrindorothers">http://trac.osgeo.org/grass/wiki/GSoC/2014/TestingFrameworkForGRASS#AnalyzingmodulerunusingValgrindorothers</a><br>
[4] <a href="http://trac.osgeo.org/grass/browser/grass/trunk/lib/python/gunittest/case.py?rev=61459#L983">http://trac.osgeo.org/grass/browser/grass/trunk/lib/python/gunittest/case.py?rev=61459#L983</a><br><br>><br>><br>
> 2014-08-08 13:00 GMT+02:00 Jachym Cepicky <<a href="mailto:jachym.cepicky@gmail.com">jachym.cepicky@gmail.com</a>>:<br>> > Hi all,<br>> ><br>> > just some intellectual thoughts for the weekend (I'm off for a<br>
> > weekend, so no need to rush with the answer). I started to write some<br>> > script [1], which tries to estimate CPU time, needed for each GRASS<br>> > module (currently, I have about 4-5 raster modules). Target is to<br>
> > create suit (rather then simple script), which could estimate how each<br>> > process scales (based on number of features or raster cells, which<br>> > need to be processed) and put GRASS modules into row. Also estimate<br>
> > rawly CPU time, which is consumed by each module.<br>> ><br>> > Any thoughts to this?<br>> ><br>> > I started with the dataset from Martin [2], but I thing, I'll switch<br>> > over to spearfish, one resolution for now. I did not figure out yet,<br>
> > how to implement equivalent of unix time [3] program in Python<br>> ><br>> > Thanks<br>> ><br>> > Jachym<br>> ><br>> > [1] <a href="https://github.com/jachym/grass-performace">https://github.com/jachym/grass-performace</a><br>
> > [2] <a href="http://geo.fsv.cvut.cz/data/grasswikicz/freegeodatacz/aktualni/">http://geo.fsv.cvut.cz/data/grasswikicz/freegeodatacz/aktualni/</a><br>> > [3] <a href="https://github.com/jachym/grass-performace">https://github.com/jachym/grass-performace</a><br>
> ><br>> > --<br>> > Jachym Cepicky<br>> > e-mail: jachym.cepicky gmail com<br>> > URL: <a href="http://les-ejk.cz">http://les-ejk.cz</a><br>> > GPG: <a href="http://les-ejk.cz/pgp/JachymCepicky.pgp">http://les-ejk.cz/pgp/JachymCepicky.pgp</a><br>
> ><br>> > Give your code freedom with PyWPS - <a href="http://pywps.wald.intevation.org">http://pywps.wald.intevation.org</a><br>> > _______________________________________________<br>> > grass-dev mailing list<br>
> > <a href="mailto:grass-dev@lists.osgeo.org">grass-dev@lists.osgeo.org</a><br>> > <a href="http://lists.osgeo.org/mailman/listinfo/grass-dev">http://lists.osgeo.org/mailman/listinfo/grass-dev</a><br>> _______________________________________________<br>
> grass-dev mailing list<br>> <a href="mailto:grass-dev@lists.osgeo.org">grass-dev@lists.osgeo.org</a><br>> <a href="http://lists.osgeo.org/mailman/listinfo/grass-dev">http://lists.osgeo.org/mailman/listinfo/grass-dev</a><br>
<br></div></div>