[Benchmarking] Context for Benchmarks

[Schlagel, Joel D IWR]

All - here is a rough idea of how we might put the OGC benchmarking in
context, and maybe simplify the presentation

The storyboard is interesting to me, and may help reduce the number of
variables that need to be tested, because in the end the main benchmark is
just on best effort.

Fine grain benchmarks that will be of interest to developers in improving
software will be generated along the way, but I think workflow and best
effort will be an easier story to tell, and may even allow an undisputed
champion to reign for a year.

This is just an idea, so no problem if the consensus is to keep benchmarking
as a purely technical exercise.



OSGEO WMS Shoot out Scenario Š putting the benchmarks in context
2011 Scenario ­ Colorado County Government Response to a Disaster
 
Part 1   - Routine Operations
 
 Colorado County operates a web based GIS.   The server caches tiles from
level 1 to N for the base map.  Their imagery is stored on file system and
updated on schedule X while their vector layers are stored in a relational
database and updated on schedule Y.   They provide OGC WMS interface to the
server including SLD, etc.

This Benchmark tests the capability & efficiency of software to generate map
tiles for routine web mapping operations.

Generate image tile cache from large satellite image and N tiles of aerial
imagery

Generate vector overlay tile cache

-      roads
-      rivers
-      parcels
-      parks
-      utilities

The benchmark is time to complete workflow (ie generate all tiles) using
³best effort.²  teams choose the OS, backend database (must be RDBMS), and
mapping server. Parallel requests vs sequential is not a concern - total
time to complete the workflow is the benchmark.   Will use complex symbology
and SLD.
 

Sub-benchmarks can test client vs backend database combinations (e.g
mapserver on oracle vs postgis) and connection strategies.  The overall
measure is  ³best effort² time to complete the work flow.

            
Part 2  Response

A large Fire in Colorado County Forest.  New aerial imagery and incident
data such as current fire area, weather, forecast direction, response points
arrive in form of shape file.  These need to be served right away to
interested public. 

In this benchmark, file system shape files and imagery are served under
load.  A random set of requests is created for various scales and areas
beforehand and same random requests are used for each software.  This leaves
out all database interaction and just deals with efficient random file
access.  Benchmark is wms requests per second, throughput under max load.