[Benchmarking] Rebooting the Benchmarking effort

[Andrea Aime]

On Wed, Sep 4, 2013 at 12:28 PM, Arnulf Christl <
arnulf.christl at metaspatial.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> All,
> thanks for the suggestions so far. I am currently rearranging the
> content of the presentation and joining efforts with the WPS
> functionality comparison group. We will present the WPS part and bits
> and pieces around various other benchmarking efforts and report from
> other conferences (FOSS4G NA, FOSSGIS, others?)
>
> At the end of the presentation I would like to share a proposal how to
> do benchmarking in the longer term. It would involve that OSGeo makes
> the benchmark an organizational effort and specifically request for
> benchmark sponsoring. Sponsors can then put a focus on either usability,
> performance, cartographic output quality, or whatever else they come up
> with (and they will). OSGeo should then invest the new funds to support
> code sprints where we dedicate one day to benchmarking. Or maybe even
> have a dedicated benchmark sprint before FOSS4G (just like it was in the
> good old days anyway).
>

Raising the bar a bit, why not try to make this a "service" so that teams
get
something back in return "continuosly"?
The idea I had in mind is, setup some server used for benchmarks, automate
them, and allow teams to upgrade their software, and add new types of
benchmarks, and have the latest results published somewhere.

A presentation can be derived from that data, maybe giving it a historical
perspective that would maybe also add fun ("until May server X was the
fastest in this test, but then server Y got better and in August server W
blew all of them out").

Requires quite a bit more effort, but would also allow people to keep
on seeing how they are doing over time in a particular benchmark
(since they are always there)

More effort, I know. Just an idea :-p

Cheers
Andrea

-- 
==
Our support, Your Success! Visit http://opensdi.geo-solutions.it for more
information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/benchmarking/attachments/20130904/5807b347/attachment-0001.html>