[Benchmarking] data block caching technique

[Luc Donea]

Hi All,

Looking at MapServer and Geoserver results, it seems clear to me that these numbers mainly result from server-side data block caching. 

Many discussions have already been done on this subject and I think that many agree that caching data blocks and using the exact same requests in the 3 runs cannot produce realistic results. Indeed, in a realistic use-case, the exact same request with the exact same bounding box will never occur twice.

I see that this has also been noted by Constellation-SDI during their own testing (http://wiki.osgeo.org/wiki/Benchmarking_2010/Constellation-SDI) "Note that this is merely an academic exercise since for any non-trivial dataset, the size of data on disk will be so much larger than the size of available memory that these numbers will never be achieved". Moreover, 

We think that this combination of unrealistic test conception and data block caching technique is unfair to other participants and will make their results looks bad, while they might perform as good or even better in a real world use-case.

I think that every one should publish all 3 run results and guarantee that these have been measured just after server restarting. We would also like that the ones using such technique rerun their test after disabling it.

Regards,

Luc

ERDAS APOLLO Product Manager
ERDAS
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/benchmarking/attachments/20100906/ad8ca880/attachment.html