[Benchmarking] Load testing tool: JMeter this year too?

[Andrea Aime]

Hi,
one of the pain points for last year exercise was the
lack of familiarity with the load testing tool, JMeter.

JMeter is using some quite verbose xml configuration files
and it's meant to be used as a GUI, though scripting it
from the command line is possible (and that's what we
did last year).

The normal workflow would be to setup the test graphically
in the GUI and then use them as-is from the command line
using a few scripts.
Each script runs a number of different thread groups in sequence,
each one using a different number of threads, and each driven
by a csv file containing a randomized set of wms request parameters.
The setup was made so that a full run did not ever use the same
request twice. In particular:
- the 1st thread group used 1 thread and did 100 requests
- the 2nd thread group used 10 thread and did 200 requests
- the 3nd thread group used 20 thread and did 400 requests
- the 4nd thread group used 40 thread and did 800 requests
The csv file driving the run contained 1500 randomized requests
params.

Each team took the JMeter template test files and adapted to
their server (changing layer names, ip address and ports and so on).

In order to run the tests in an automated fashion we had a bit
of scripts:
* http://svn.osgeo.org/osgeo/foss4g/benchmarking/scripts/wms_request.py
   The script we used to generate a csv file containing a set
   of randomized width, height and bboxes used to drive the
   test run
* http://svn.osgeo.org/osgeo/foss4g/benchmarking/scripts/benchmark
   The script that actually runs JMeter from the command line
* http://svn.osgeo.org/osgeo/foss4g/benchmarking/scripts/summarizer.py
   The script that grabs the output of the benchmark run and generates
   a summary table with avg/min/max per thread group contained
   in the run
* Each team created a further set of scripts to fully automated the
   full set of tests (running all of them took one/two hours last year):
   http://svn.osgeo.org/osgeo/foss4g/benchmarking/scripts/mapservconf

It is my understanding that this year we're going to do something 
similar, but using 1,2,4,8,16,32,64 threads instead (maybe we should
just stop at 32, not much point going high if the hardware we'll
get has one dual core cpu).

Anyways, the thing is, we're not bound to specifically use JMeter,
the main advantage of it is that some of us have some familiarity
with it from the past experiences and we have the machinery to
run multithreaded tests with randomized requests already.

If anybody wants to push for a different load testing tool it would
be a good idea to do so right now, otherwise I'd suggest to get
familiar with it.

You can download it here:
http://jakarta.apache.org/jmeter/
and there are some simple tutorials about it on the net:
http://www.devx.com/webdev/Article/17950/1954?pf=true
(there are more, just google for them)
besides of course the JMeter own manual:
http://jakarta.apache.org/jmeter/usermanual/index.html

Cheers
Andrea

-- 
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.