[GRASS-dev] measuring cpu time for each module and estimating modules scalability

[Vaclav Petras]

On Fri, Aug 8, 2014 at 8:29 AM, Sören Gebbert <soerengebbert at googlemail.com>
wrote:
>
>
> * Please consider to integrate your benchmark suite into the GRASS
> testing framework GSoC project [1] which already includes runtime
> analysis.
>
Hi Jachym,

that's great that you are doing this and that you want to standardize it.
I'm not sure what exactly Soeren means by "already includes runtime
analysis" because testing framework does not contain anything like that yet
(not counting timing included in PyGRASS [2]) but there is certainly an
overlap between benchmarking and testing.

First, we plan to include timing into the testing framework, so that when
you call a module you get also time as a part of (test) report. Some other
things are also interesting, namely profiling, memory check (valgrind),
coverage and perhaps some other things from clang/LLVM world [3]. I think
that profiling and timing are surely the overlap between testing framework
and benchmarking framework/suite. The idea, for testing, currently is that
the measurement of module execution would be done in `assertModule()`
function (which executes the module) [4] and the results will be collected
for each test.

Second, and I was not much thinking about this yet, benchmark can has a
very similar structure to test: preparation, benchmark/test and cleanup. I
don't think that they should be mixed because benchmarks would slow down
tests significantly but maybe the system can be the same. On the other
hand, there is certainly a large areas in benchmarking API, implementation
and report generation which are very different from testing, for example a
graph of cell count on X axis and time on Y axis and everything what is
needed to get this graph. Also note that some projects actually monitor
time of test execution and compare current state to the previous one to
warn if there was some slow down (this is applicable for both testing and
benchmarking).

Anyway, once the ideas and code are stabilized and relation to testing
framework is solved, this is something I would like to see in GRASS core.

2014-08-08 13:00 GMT+02:00 Jachym Cepicky <jachym.cepicky at gmail.com>:
> I started with the dataset from Martin, but I thing, I'll switch
> over to spearfish, one resolution for now.

Please, switch to NC SPM, this is the dataset/location which is (should be)
used in manual pages now and which is partially used also by tests.

Best,
Vaclav

[1] http://trac.osgeo.org/grass/wiki/GSoC/2014/TestingFrameworkForGRASS
[2]
http://trac.osgeo.org/grass/browser/grass/trunk/lib/python/pygrass/modules/interface/module.py#L537
[3]
http://trac.osgeo.org/grass/wiki/GSoC/2014/TestingFrameworkForGRASS#AnalyzingmodulerunusingValgrindorothers
[4]
http://trac.osgeo.org/grass/browser/grass/trunk/lib/python/gunittest/case.py?rev=61459#L983

>
>
> 2014-08-08 13:00 GMT+02:00 Jachym Cepicky <jachym.cepicky at gmail.com>:
> > Hi all,
> >
> > just some intellectual thoughts for the weekend (I'm off for a
> > weekend, so no need to rush with the answer). I started to write some
> > script  [1], which tries to estimate CPU time, needed for each GRASS
> > module (currently, I have about 4-5 raster modules). Target is to
> > create suit (rather then simple script), which could estimate how each
> > process scales (based on number of features or raster cells, which
> > need to be processed) and put GRASS modules into row. Also estimate
> > rawly CPU time, which is consumed by each module.
> >
> > Any thoughts to this?
> >
> > I started with the dataset from Martin [2], but I thing, I'll switch
> > over to spearfish, one resolution for now. I did not figure out yet,
> > how to implement equivalent of unix time [3] program in Python
> >
> > Thanks
> >
> > Jachym
> >
> > [1] https://github.com/jachym/grass-performace
> > [2] http://geo.fsv.cvut.cz/data/grasswikicz/freegeodatacz/aktualni/
> > [3] https://github.com/jachym/grass-performace
> >
> > --
> > Jachym Cepicky
> > e-mail: jachym.cepicky gmail com
> > URL: http://les-ejk.cz
> > GPG: http://les-ejk.cz/pgp/JachymCepicky.pgp
> >
> > Give your code freedom with PyWPS - http://pywps.wald.intevation.org
> > _______________________________________________
> > grass-dev mailing list
> > grass-dev at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/grass-dev
> _______________________________________________
> grass-dev mailing list
> grass-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20140808/50261528/attachment-0001.html>