[Benchmarking] split contour files are different

[Andrea Aime]

Jason Birch wrote:
> In the real world, decent integrators would ETL the data into the
> optimal format for the rendering server, generalize the data for
> display at different levels, etc, to achieve the highest-possible
> performance.
> 
> This test isn't about real world, it's about taking a set of arbitrary
> data and benchmarking against it. Some servers will be hindered,
> others will benefit. The only real conclusion that can be drawn from
> non-best-effort tests is that the server was good or poor at serving
> some particular data configuration.
> Nothing wrong with this as long as the parameters are understood, but
> it's not a great metric for potential implementors to be using when
> deciding what system to use.

I disagree. From my experience the baseline test is much closer to
the real world than the best effort one.

I see lots of newcomers on the GeoServer users list that just try
to use whatever they have and they are very happy to see anything
working at all. Those users would go and setup N layers, they don't
even know what an ETL is.

The best effort test instead allows to use all knowledge developers
have to setup a high performance serving environment, and time allowing,
to perform specific optimizations in the code. This might
leverage some deep knowledge of how the internals work, which is
something not even a skilled user has.

The "decent integrator" case falls in the middle of the two, how
close to each one depends a lot on the skills of the integrator
(what makes one decent?) and the specific knowledge one needs
to make a certain set of layers fly (which might change depending
on the type of styling and other variables).

Cheers
Andrea