[GRASS-dev] measuring cpu time for each module and estimating modules scalability

[Sören Gebbert]

Hello,

2014-08-08 16:04 GMT+02:00 Vaclav Petras <wenzeslaus at gmail.com>:
> On Fri, Aug 8, 2014 at 8:29 AM, Sören Gebbert <soerengebbert at googlemail.com>
> wrote:
>>
>>
>> * Please consider to integrate your benchmark suite into the GRASS
>> testing framework GSoC project [1] which already includes runtime
>> analysis.
>>
> Hi Jachym,
>
> that's great that you are doing this and that you want to standardize it.
> I'm not sure what exactly Soeren means by "already includes runtime
> analysis" because testing framework does not contain anything like that yet
> (not counting timing included in PyGRASS [2]) but there is certainly an
> overlap between benchmarking and testing.

[2] Is what i was referencing with "runtime" not the profiling and
memory checking that we would like to support.

> First, we plan to include timing into the testing framework, so that when
> you call a module you get also time as a part of (test) report. Some other
> things are also interesting, namely profiling, memory check (valgrind),
> coverage and perhaps some other things from clang/LLVM world [3]. I think
> that profiling and timing are surely the overlap between testing framework
> and benchmarking framework/suite. The idea, for testing, currently is that
> the measurement of module execution would be done in `assertModule()`
> function (which executes the module) [4] and the results will be collected
> for each test.
>
> Second, and I was not much thinking about this yet, benchmark can has a very
> similar structure to test: preparation, benchmark/test and cleanup. I don't
> think that they should be mixed because benchmarks would slow down tests
> significantly but maybe the system can be the same. On the other hand, there
> is certainly a large areas in benchmarking API, implementation and report
> generation which are very different from testing, for example a graph of
> cell count on X axis and time on Y axis and everything what is needed to get
> this graph. Also note that some projects actually monitor time of test
> execution and compare current state to the previous one to warn if there was
> some slow down (this is applicable for both testing and benchmarking).
>
> Anyway, once the ideas and code are stabilized and relation to testing
> framework is solved, this is something I would like to see in GRASS core.
>
>
> 2014-08-08 13:00 GMT+02:00 Jachym Cepicky <jachym.cepicky at gmail.com>:
>> I started with the dataset from Martin, but I thing, I'll switch
>
>> over to spearfish, one resolution for now.
>
> Please, switch to NC SPM, this is the dataset/location which is (should be)
> used in manual pages now and which is partially used also by tests.
>
> Best,
> Vaclav
>
> [1] http://trac.osgeo.org/grass/wiki/GSoC/2014/TestingFrameworkForGRASS
> [2]
> http://trac.osgeo.org/grass/browser/grass/trunk/lib/python/pygrass/modules/interface/module.py#L537
> [3]
> http://trac.osgeo.org/grass/wiki/GSoC/2014/TestingFrameworkForGRASS#AnalyzingmodulerunusingValgrindorothers
> [4]
> http://trac.osgeo.org/grass/browser/grass/trunk/lib/python/gunittest/case.py?rev=61459#L983
>
>
>>
>>
>> 2014-08-08 13:00 GMT+02:00 Jachym Cepicky <jachym.cepicky at gmail.com>:
>> > Hi all,
>> >
>> > just some intellectual thoughts for the weekend (I'm off for a
>> > weekend, so no need to rush with the answer). I started to write some
>> > script  [1], which tries to estimate CPU time, needed for each GRASS
>> > module (currently, I have about 4-5 raster modules). Target is to
>> > create suit (rather then simple script), which could estimate how each
>> > process scales (based on number of features or raster cells, which
>> > need to be processed) and put GRASS modules into row. Also estimate
>> > rawly CPU time, which is consumed by each module.
>> >
>> > Any thoughts to this?
>> >
>> > I started with the dataset from Martin [2], but I thing, I'll switch
>> > over to spearfish, one resolution for now. I did not figure out yet,
>> > how to implement equivalent of unix time [3] program in Python
>> >
>> > Thanks
>> >
>> > Jachym
>> >
>> > [1] https://github.com/jachym/grass-performace
>> > [2] http://geo.fsv.cvut.cz/data/grasswikicz/freegeodatacz/aktualni/
>> > [3] https://github.com/jachym/grass-performace
>> >
>> > --
>> > Jachym Cepicky
>> > e-mail: jachym.cepicky gmail com
>> > URL: http://les-ejk.cz
>> > GPG: http://les-ejk.cz/pgp/JachymCepicky.pgp
>> >
>> > Give your code freedom with PyWPS - http://pywps.wald.intevation.org
>> > _______________________________________________
>> > grass-dev mailing list
>> > grass-dev at lists.osgeo.org
>> > http://lists.osgeo.org/mailman/listinfo/grass-dev
>> _______________________________________________
>> grass-dev mailing list
>> grass-dev at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/grass-dev
>