[Benchmarking] potential caching issue ?

[Adrian Custer]

On Wed, 2010-09-01 at 17:08 +0200, Andrea Aime wrote:
> Adrian Custer ha scritto:
> > On Wed, 2010-09-01 at 15:34 +0200, Andrea Aime wrote:
> >> If the servers are actually caching the response or even just caching
> >> the data in memory they are violating the bechmarking rules.
> > 
> > Except that that is, of course, what is happening when servers stop
> > being 'disk bound'.
> 
> Not really. You still have to parse the bytes the OS has in memory
> into geometries and attributes.

which are no longer 'data'

> That's nothing compared to actually reading from disk, but can make
> a significant difference in the non disk bound results.

Indeed, this would be a worthwhile comparison between servers, as would,
at the opposite end, examining the read efficiency of servers in a
totally 'disk bound' situation. 

So the benchmarking could have been setup to ensure that all servers are
'disk bound' for a particular set of tests, and that all are 'cpu' bound
for another. However, the discussion of how to design a well structured
set of tests never managed to get off the ground and it is too late to
embark on that design work now. It sounds like there will be a mix of
'disk bound' and 'cpu bound' servers in the same test which may or may
not be of use and interest to users.

--adrian


> 
> Cheers
> Andrea
> 
>