[Benchmarking] Chat about the load testing setup

[Andrea Aime]

Hi,
me and Frank had a quick chat about how to setup the
test loading scripts against the ECW layers (but, more
in general, against any layer we're going to test).

We thought it could be of interest to others as well,
so here is a transcript:

------------------------------------------------------------------

aaime: So here is how I was picturing the test
aaime: we set up one or more ECW files
aaime: create a random set of bboxes for typical tile requests
aaime: 256x256, different zoom levels
aaime: so that during the test the same area is never requested twice
aaime: and having the request jump on the zoom level just randomly
FrankW: yes, that sounds fine
aaime: parallel threads going from 1 to 40
aaime: with a minimum of 100 request no matter what the thread count is
aaime: The test load generator would be JMeter
aaime: it requires a JDK, but it's the only one I know that makes it 
easy to have a load that scales up
aaime: from few to many threads
FrankW: sounds good.  I presume we play the same set of BBOX requests 
against both servers?
aaime: right, someone will make a script that generates a csv file with 
the bboxes
aaime: and the jmeter script is parametrized with it
aaime: so the same exact sequence of calls is made
aaime: on every run
aaime: (actually JMeter requires only a JRE, not the full JDK)
aaime: I want to use basically the same setup for all test
FrankW: jmeter will run on the 1 cpu server I assume?
aaime: yes
aaime: the script should be parametrized by area of interest, zoom 
levels and number of requests so that we can make it generate random 
sequenecs for all the different layers we use in the tests
aaime: I thought to make it follow a typical tiling grid just for the 
sake of simplicity
aaime: wondering if it's a good idea
aaime: or if we should make it go fully random instead
FrankW: I would be inclined to let it be irregular but I doubt it makes 
much difference.
aaime: irregular... it may be even easier to create the script
aaime: we just need to ensure we don't generate too big request areas I 
guess
aaime: (at least for the tests where the style is not scale dependent)
aaime: what about the form factor then? always following the queried area
FrankW: We should be aiming for typical web mapping sized requests so 
certainly a modest upper bound on size.
aaime: or do we want deformed requests as well?
FrankW: I don't mind one way or the other.
aaime: so image size less than 1024x768
aaime: and actual area not too big, so that it does not load 1M features 
in a single request
FrankW: Right, that would be a reasonable upper bound.
aaime: (getting the latter right may not be easy...)
FrankW: Arguably we ought to include "overview requests" as it is not an 
uncommon case.
aaime: right.. yet usually overview requests are made against some 
reduced dataset
FrankW: But I can see it might be inappropriate for some feature based 
maps that don't have special cases for zoomed out.
aaime: like, the road map is 3M features...
FrankW: For raster data I'd like a wide variety of zoom levels if possible.
aaime: not sure anyone wants to paint it in a overview map as is?
FrankW: Perhaps "zoom constraints" could be a parameter to the bbox 
generator.
aaime: right, assuming we have overviews it's more natural
aaime: and yes, min/max scale denominator should be a parameter
aaime: do you have enough spare time to make a python script that does 
the above? :-)
aaime: I can write the JMeter scripts in turn :)
FrankW: I could do so.
FrankW: I'll make a first crack at it this afternoon and then we can see 
if we want to refine the rules a bit.
aaime: perfect
aaime: last time I wrote a similar one (which silly me I forgot to backup)
aaime: it generated a csv file containing
aaime: width,height,x1,y1,x2,y2
FrankW: ok, I'll produce the same.
aaime: I can easily hook that up to JMeter so that each row will be used 
for a different request

------------------------------------------------------------------

Cheers
Andrea

-- 
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.