[Benchmarking] First pair of load scripts and some considerations

Mon Aug 24 13:22:47 EDT 2009

Hi,
I've been playing a bit with sample JMeter load tests using
using Frank's python script to generate random bboxes to be
painted.

First off, I've fixed a bug in the script (it used to generate
bboxes whose upper corner was outside the requested bounds)
and modified the output a little so that it looks
width;height;x1,y1,x2,y2
This makes it easier to extract the bbox as a single param.
Modified script attached.

I've then used the python script and some existing JMeter
scripts to put togheter a couple of sample load scripts.

The first, bluemarble_ecw.jmx can be used to load a server
exposing the bluemarble raster dataset, the second,
tiger_texas.jmx, can be used to test the tiger roads
layer.

The .csv files driving the requests have been obtained running
the fixed script with params:
wms_request.py -count 2000 -region -180 -90 180 90 -minres 0.0042 
-maxres 0.5 > bluemarble.csv
wms_request.py -count 2000 -region -102 30 -94 34 -minres 0.00005 
-maxres 0.001 > texas.csv

(the script is generating random output, so if you run them again
  you won't get the same output).

To run the scripts you need to download JMeter (get the latest
version, 2.3.4), which in turn requires
a Java runtime. I'll be installing it on the test box soon
(that does not require to have its OS reinstalled, right?)
but in the meantime you can also run them on your machine.

JMeter runs as a GUI, and in that case you can use
Run -> Start to start the tests, Run -> clear to cleanup
the results, and look into "Summary Report" in the tree
to see it working.

JMeter can also be run headless, but things become more
complicated here.
To run it headless, one has to execute:
jmeter -p jmeter.properties -n -t bluemarble_ecw.jmx -l 
bluemarble_samples.csv

and that unfortunately results just in the log of the requests
done, with timing and response.
That file can then be loaded into the JMeter GUI to get a nice
looking summary. To do that:
- generate the .csv file with the above command
- scp to your host if needed
- open SummaryLoader.jmx via the GUI, go into the
   "Summary Report" item, browse for the file, done

Or, we can make up a script that does the summary
for us and outputs it as a text table directly on the
server, without any GUI.

To get more information about the csv file contents, see here:
http://jakarta.apache.org/jmeter/usermanual/listeners.html
Some idea on how to make a summarizer script are here:
http://wiki.apache.org/jakarta-jmeter/LogAnalysis

I hope the above is not too confusing. Anyways, anyone
working on running the scripts should get some familiarity with
JMeter (unless we decide to use something else as the runner,
which is doable, provided you find a runner that can be
driven by a csv file so that we never make the same request
twice in the run to avoid possible in memory caching in the
servers tested).

Cheers
Andrea
-------------- next part --------------
A non-text attachment was scrubbed...
Name: wms_request.py
Type: text/x-python
Size: 4592 bytes
Desc: not available
Url : http://lists.osgeo.org/pipermail/benchmarking/attachments/20090824/5c420eb2/wms_request-0001.py
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bluemarble.csv
Type: text/csv
Size: 99255 bytes
Desc: not available
Url : http://lists.osgeo.org/pipermail/benchmarking/attachments/20090824/5c420eb2/bluemarble-0001.bin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bluemarble_ecw.jmx
Type: text/xml
Size: 28986 bytes
Desc: not available
Url : http://lists.osgeo.org/pipermail/benchmarking/attachments/20090824/5c420eb2/bluemarble_ecw-0001.xml
-------------- next part --------------
#---------------------------------------------------------------------------
# Results file configuration
#---------------------------------------------------------------------------

# This section helps determine how result data will be saved.
# The commented out values are the defaults.

# legitimate values: xml, csv, db.  Only xml and csv are currently supported.
jmeter.save.saveservice.output_format=csv

# true when field should be saved; false otherwise

# assertion_results_failure_message only affects CSV output
#jmeter.save.saveservice.assertion_results_failure_message=false
#
#jmeter.save.saveservice.data_type=true
#jmeter.save.saveservice.label=true
#jmeter.save.saveservice.response_code=true
# response_data is not currently supported for CSV output
#jmeter.save.saveservice.response_data=false
# Save ResponseData for failed samples
#jmeter.save.saveservice.response_data.on_error=false
#jmeter.save.saveservice.response_message=true
#jmeter.save.saveservice.successful=true
#jmeter.save.saveservice.thread_name=true
#jmeter.save.saveservice.time=true
#jmeter.save.saveservice.subresults=true
#jmeter.save.saveservice.assertions=true
#jmeter.save.saveservice.latency=true
#jmeter.save.saveservice.samplerData=false
#jmeter.save.saveservice.responseHeaders=false
#jmeter.save.saveservice.requestHeaders=false
#jmeter.save.saveservice.encoding=false
#jmeter.save.saveservice.bytes=true
jmeter.save.saveservice.url=true
#jmeter.save.saveservice.filename=false
#jmeter.save.saveservice.hostname=false
#jmeter.save.saveservice.thread_counts=false
#jmeter.save.saveservice.sample_count=false

# Timestamp format
# legitimate values: none, ms, or a format suitable for SimpleDateFormat
#jmeter.save.saveservice.timestamp_format=ms
#jmeter.save.saveservice.timestamp_format=MM/dd/yy HH:mm:ss

# Put the start time stamp in logs instead of the end
sampleresult.timestamp.start=true

# legitimate values: none, first, all
#jmeter.save.saveservice.assertion_results=none

# For use with Comma-separated value (CSV) files or other formats
# where the fields' values are separated by specified delimiters.
# Default:
#jmeter.save.saveservice.default_delimiter=,
# For TAB, since JMeter 2.3 one can use:
#jmeter.save.saveservice.default_delimiter=\t

#jmeter.save.saveservice.print_field_names=false

# Optional list of JMeter variable names whose values are to be saved in the result data files.
# Use commas to separate the names. For example:
#sample_variables=SESSION_ID,REFERENCE
# N.B. The current implementation saves the values in XML as attributes,
# so the names must be valid XML names.
# Versions of JMeter after 2.3.2 send the variable to all servers
# to ensure that the correct data is available at the client.

# Optional xml processing instruction for line 2 of the file:
#jmeter.save.saveservice.xml_pi=<?xml-stylesheet type="text/xsl" href="sample.xsl"?>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: texas.csv
Type: text/csv
Size: 99122 bytes
Desc: not available
Url : http://lists.osgeo.org/pipermail/benchmarking/attachments/20090824/5c420eb2/texas-0001.bin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tiger_texas.jmx
Type: text/xml
Size: 29294 bytes
Desc: not available
Url : http://lists.osgeo.org/pipermail/benchmarking/attachments/20090824/5c420eb2/tiger_texas-0001.xml
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SummaryLoader.jmx
Type: text/xml
Size: 2174 bytes
Desc: not available
Url : http://lists.osgeo.org/pipermail/benchmarking/attachments/20090824/5c420eb2/SummaryLoader-0001.xml