[Benchmarking] Ideas for next year

[Martin Desruisseaux]

Just for balancing a little bit the propsed idea, we mut admit that we (the 
Constellation team) started to work very late on the Benchmark - which is our 
own fault. So for next year, I understand the first thing to do on our side is 
to find some way to participate sooner (which is our reponsability) so we can 
hopefully bring some constructive help and tools.

On my side, one of my which is to propose some script for transfering the JMeter 
results in a SQL database so we can perform more easily statistical analysis. I 
was not there today to explain my idea about confidence interval (I think that 
my proposal is still misunderstood according what I have heard), but will try to 
explain better during the following year.

Best regards,

     Martin



Le 17/09/11 17:39, johann.sorel at geomatys.com a écrit :
> Hi,
>
> Just my thinking : Last year event had much more competitors, this year it was 
> a Mapnik vs Mapserver mainly.
> So I understand you want to compare both projects.
>
> This year was my first participation as a developer for the Constellation 
> server and I have been really upset most of the time.
> I was hoping to work on our engine improvments but at the end I spend 70% of 
> my time on a parser to convert
> Mapfile to SLD. without this effort both Constellation and Geoserver would 
> have been out of the bench.
>
> So Definitly next year if we intend to have more competitors (and not even 
> less) there is a need to describe the objective in a neutral way for all 
> teams, both styling and datas. I'm not saying it must be OGC SLD/SE, a text 
> describing the expected result is enough, each team can then implement it with 
> it's own style model.
>
> Talking about datas, only about 3 or 4 weeks before the bench was decided to 
> use BIL files for pseudo-hillshading. since both mapserver and mapnik rely on 
> gdal/ogr they had no problems but that's not the case for everyone. so I also 
> hope last minutes change linked to data format will not happen in the futur.
>
> I also noticed those tests did not involve vector reprojections. after all we 
> are providing Mapping servers not Painting servers. so reprojection should 
> take more place in the tests. I think running queries in ten or more different 
> projections would be nice.
>
> johann