[Benchmarking] Test matrix (aka putting tests on a diet)

[Andrea Aime]

Shanmugam Ganeshkumar ha scritto:
> Paul +1
> 
> A typical WMS request is not tile based. The real world experience 
> varies anything between 600x400 (512x512, if you want to consider tile) 
> to 1024x768 (4x3 tiles). As Paul puts it, tiles based measurement is 
> good only for tile based service (GWC, TileCache, ArcGIS TMS). If the 
> benchmarking is on the GS, MS and ArcGIS and WMS/WFS as protocols then 
> you should consider one agreed tile size and do more fine grain tests 
> rather than diluting by testing few tile sizes. Make sure that content 
> is NOT cached  to measure the real throughput of the server.

We try hard to avoid caching by using a random generator for the 
requests, JMeter will issue 1500 requests during a benchmark run,
each of them has a randomized width, height and bbox.

That said, the vector data sets are smaller than the memory available
so a server system could try and cache the data set fully in memory.
This is something we should prevent, but if a server is designed to
take advantage of big memory systems we can only hope there is
a way to turn the cache off (and maybe show a comparison between
cache on and off).

MapServer and GeoSErver perform no caching of vector data, not
sure about ArgGis.

Cheers
Andrea

-- 
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.