RE : [Benchmarking] data block caching technique

[Liujian (LJ) Qian]

  Hi,
This is just my opinion on how to improve future benchmarks. I think in 
order for the tests to be more real-world like, we need to make sure the 
total amount of hot data  that went through the rendering pipe is much 
bigger than the amount of available physical memory.  So for instance on 
a 8GB box we should be rendering 16GB of raw data  for a full run. My 
guestimate for this year's vector working set (from those 2152 query 
windows) is around 2GB, which as some team discovered can be coerced 
into memory cache (at the OS level) with some conscious efforts from its 
map server.

Maybe in addition to this large working set, we can devise a smaller one 
so that every team is encouraged to cache it in memory, thereby 
eliminating any disk boundness.  This will be a good indication of the 
raw 'map rendering' performance.  But it probably does not say much 
about scalability in a real world situation.

thanks
LJ


On 9/6/2010 11:47 AM, Luc Donea wrote:
>
> Hi Adrian and all,
>
> Thanks for your quick answer Adrian. I like your idea to generate 
> newer, shorter scripts.
>
> I would not limit to only two threads, I think it stay important to 
> show the server scalability at 64 users. We could do 1, 4 and 64 users 
> to stay short. The important thing to me is to to create a different 
> csv for the 2nd and 3rd runs. This would ensure a more realistic 
> result and still allow servers to warm up. I also think we should 
> rerun the imagery but also the vector tests. Do you think you would be 
> able to create these tests? I must admit that I am unable to do 
> that... :-)
>
> It would be indeed really nice to run the benchmarks all together so 
> everyone can follow what is happening, but I guess this can take quite 
> awhile to run 16 tests and starting 8 servers. I guess it can easily 
> turn to a 3 hours session.
>
> I'd like to know what the others are thinking about this and what 
> would be everyone's availability on Tuesday?
>
> Thanks,
>
> Luc
>
> -------- Message d'origine--------
> De: benchmarking-bounces at lists.osgeo.org de la part de Adrian Custer
> Date: lun. 06/09/2010 04:58
> À: benchmarking at lists.osgeo.org
> Objet : Re: [Benchmarking] data block caching technique
>
> Hey all,
>
>
> On Mon, 2010-09-06 at 09:38 +0200, Luc Donea wrote:
> >
> > We think that this combination of unrealistic test conception and data
> > block caching technique is unfair to other participants and will make
> > their results looks bad, while they might perform as good or even
> > better in a real world use-case.
> >
> We tried to raise this issue early on by saying that all those in the
> benchmarking effort really needed to agree on what kind of a setup we
> were trying to mimic in the benchmark so that we could then build tests
> which reasonably represented that setup.
>
> Because we did not do that work, it seems we have stumbled into an edge
> case for which some servers are able to work only from main memory. When
> we agreed to use the current tiny raster data set (compared to the 1.3Tb
> full .ecw dataset for all of Spain), we realized that we would not be
> benchmarking a real, industrial dataset. However, we did not know that
> it would be just small enough that, coupled with repeated request sets,
> some servers would be working from main memory.
>
>
> > I think that every one should publish all 3 run results and guarantee
> > that these have been measured just after server restarting. We would
> > also like that the ones using such technique rerun their test after
> > disabling it.
>
> The question of how to resolve this situation is more difficult.
>
>
> We had a vote on which scripts to use, and the vote result was in favour
> of switching. Seeing the results of the vote, our team started all our
> runs with the newer scripts.
>
> However, the vote seems to have been totally ignored. I personally do
> not like working through this voting process but would rather work
> through the slower but more friendly and productive process of getting
> everyone to agree on a consensus position. Nonetheless up until the
> script vote, everything in this benchmarking process was done through
> voting. I am puzzled as to why, on this issue, the vote was ignored.
>
>
>
> The proposal you make, Luc, would be difficult to follow. I imagine few
> of the teams bothered to flush the memory cache before making their
> runs. I have observed Constellation-SDI both after filling the caches
> and after emptying them---the results are, unsurprisingly, totally
> different. So your proposal boils down to every team re-running a set of
> benchmarks.
>
> I am open to any productive suggestion of how to resolve the issue. We
> could easily generate newer, shorter scripts, say with only one or two
> thread combinations to test servers as if they were serving the whole
> imagery dataset for Spain yet be able to complete runs for all the teams
> in the remaining time. We might even be able to make the runs for all
> servers during our meeting time on the 7th. It would seem generally a
> good strategy anyhow to run the benchmarks all together so everyone can
> follow what is happening.
>
>
> --adrian custer
>
>
>
>
> _______________________________________________
> Benchmarking mailing list
> Benchmarking at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/benchmarking
>
>
> _______________________________________________
> Benchmarking mailing list
> Benchmarking at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/benchmarking

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/benchmarking/attachments/20100906/05216a79/attachment-0001.html