[Benchmarking] potential caching issue ?

[Andrea Aime]

Anne-Sophie Collignon ha scritto:

> Hi All,
> 
>  
> 
> We would like to raise a potential issue in the way the Jmeter tests are 
> designed..
> 
> We had the confirmation today that the GetMap BBOX are randomly chosen 
> (which is good) but once chosen, these are used in the 3 runs.
> 
> This means that one single csv is used for the 3 runs.
> 
>  
> 
> We have concern that some servers can simply cache responses for 
> subsequent runs, which would produce unrealistic throughput results.

If the servers are actually caching the response or even just caching
the data in memory they are violating the bechmarking rules.

> We understand the fact that we have 3 runs, indeed the server needs to 
> be initialized, but server initialization and caching are from our point 
> of view different topics.
> 
>  
> 
> We think that caching the responses is not a realistic use-case. It 
> would mean that we measure the network bandwidth (caching results = 
> sending back bytes on the wire) rather than the real getmap performances 
> of the servers.
>  
> 
> At this level, we understand it may be too late to change the test (E.g: 
> to have the BBOX slightly different in all 3 runs, that way, the server 
> will be initialized but will be unable to cache the data blocks and the 
> results responses).
>  
> 
> What do you think can be done to avoid this scenario?

Hum... it's difficult to achieve what you say, but not impossible.
In theory we could have a single jmeter tests having 3 times as much
thread groups, e.g., 
1a,2a,4a,8a,16a,32a,64a,1b,2b,4b,8b,16b,32b,64b,1c,2c,4c,8c,16c,32c,64c
and have the csv file contain 3 times as much rows.

 From what I can see none of the servers I've managed to look at
actually caches the results, but some manage to read small enough
data chunks from disk that the run eventually stops being disk bound.
Those that manage quickly get into the tens of results per seconds
and move to being cpu bound instead of disk bound, those that do not
are stuck to something like 2-6 results per second

If this is realistic or not, it's hard to say. A real server in 
production will be unlikely to be cpu bound, but also unlikely
to be fully disk bound, as the requests coming from users are normally
not completely random, there is correlation between the requests from
a single user and, in public sites, correlation due to events
of interest (like the news reporting a fire in a certain area
you know nothing about).

The formula was chosen as is because it makes tests cheaply repeatable
(a fully cold benchmark would require the server to be restarted).

As for changing it, yeah, it's late. On the other side, it's probably
a few hours of work to change the csv files and the jmeter templates.
So I'm open to both.

Cheers
Andrea


-- 
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.