[Benchmarking] Comparison: avg time or throughput?

[Andrea Aime]

Frank Warmerdam ha scritto:
> Jeff McKenna wrote:
>> Andrea Aime wrote:
>>> Hi,
>>> are we going to use average time or throughput for the
>>> comparison?
>>>
>>> I'm asking since I can configure GeoServer to optimize
>>> one or the other, but not both, so to get the final
>>> numbers I'd like to know what configuration I should use.
>>>
>>
>> I had assumed that we would point out differences in results for both 
>> average time and throughput, and let the users decide which is more 
>> important.  I don't think we have time to run any more tests and 
>> digest the results.
> 
> Andrea,
> 
> I would have no objection to your rerunning geoserver tests to focus on
> throughput if you think that would make a noticable difference.  I'd be
> interested to hear how significantly different geoserver strategies affect
> the different performance numbers.

This morning I rerun the vector tests I could with different 
configurations. Basically with a setup like the MapServer one (only 8 
workers) I get better throughput and better max time, but worse average
time. With a configuration that allows up to 40 workers the opposite,
better average time but higher max time and lower throughput.

If the focus in on throughput I should reran the SDE tests and the
raster tests with the configuration optimized for throughput, I could
not do either of them this morning. I'll have time Saturday morning
Sydney time when I arrive, now it's not possible as I'm about to
get onto a plane.

As I said before on IRC the setup using only 8 workers is imho not
realistic, as slow clients and bigger requests might keep
the workers busy for a long time, basically resulting in a denial
of service (but I'm not sure how FCGI works, if Apache gathers
the output to disk and streams it out later it might work for the
WMS case if you make sure no one can make request taking a lot
of time).
But if the target is throughput in a local network enviroment
the configuration has to be tweaked accordingly (and I miss numbers
for SDE and raster data in that case).

Cheers
Andrea