[Benchmarking] Ideas for next year

[Vincent Heurteaux]

Ok, please excuse Johann's e-mail rudeness, after talking with him, you can do "sed 's/upset/desapointed/ johann's_email" and so on ...

His frustration on this exercise is entirely my fault. Due to lack of time to spend on this game (this is a game IMHO), he had to work night and day in a really short time period to keep Constellation in the competition.
Then I engage Myself to give more time to Martin and Johann next year to prepare the use cases, and play the game in the right conditions.

Cheers,

Vincent

Le 17 sept. 2011 à 09:39, johann.sorel at geomatys.com a écrit :

> Hi,
> 
> Just my thinking : Last year event had much more competitors, this year it was a Mapnik vs Mapserver mainly.
> So I understand you want to compare both projects.
> 
> This year was my first participation as a developer for the Constellation server and I have been really upset most of the time.
> I was hoping to work on our engine improvments but at the end I spend 70% of my time on a parser to convert
> Mapfile to SLD. without this effort both Constellation and Geoserver would have been out of the bench.
> 
> So Definitly next year if we intend to have more competitors (and not even less) there is a need to describe the objective in a neutral way for all teams, both styling and datas. I'm not saying it must be OGC SLD/SE, a text describing the expected result is enough, each team can then implement it with it's own style model.
> 
> Talking about datas, only about 3 or 4 weeks before the bench was decided to use BIL files for pseudo-hillshading. since both mapserver and mapnik rely on gdal/ogr they had no problems but that's not the case for everyone. so I also hope last minutes change linked to data format will not happen in the futur.
> 
> I also noticed those tests did not involve vector reprojections. after all we are providing Mapping servers not Painting servers. so reprojection should take more place in the tests. I think running queries in ten or more different projections would be nice.
> 
> johann
> 
> 
> On 17/09/2011 07:23, Iván Sánchez Ortega wrote:
>> Hi all,
>> 
>> During some beers at the Wynkoop, I had an idea that I think is worth sharing.
>> 
>> Until now, the results focus on throughtput per number of concurrent requests.
>> This is fine, but other metrics are possible.
>> 
>> Then, I heard that Mapnik excels at requests with few vector features, while
>> Mapserver does a very good job when there are many vector features to be
>> rendered.
>> 
>> You can guess where this goes. I will propose that, for next year (years?),
>> requests should be classified into groups depending on the number of features
>> contained in that extent. e.g. requests with<10 feats, 10-50, 50-100,
>> 100-500,>500. Measure latency/throughput for every group, put the results in
>> a graph.
>> 
>> 
>> 
>> I don't know if this is feasible. Anyway, will see you tomorrow at the Code
>> Sprint,
> 
> _______________________________________________
> Benchmarking mailing list
> Benchmarking at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/benchmarking