Benchmarking Effort - Well done all. Was Re: RE : [Benchmarking] data block caching technique

[Michael Smith]

All,

I think the most important thing from this effort is what we all learn about
our respective software each year. All of these exercise are a little
contrived to actually make it possible.

We could have a 1Tb or 10tb dataset but then we would never be able to
distribute that data set back out for others to run. That's one point I
think we forget in trying to have better numbers in this or that test. We
want these tests and benchmarks to be available for others to run outside of
our environment. 

We are also running on servers that have been provided by the US Army Corps
for this effort. If people want to have better servers to test with, then
please, contribute your hardware, your bandwidth, your funds.

This is starting to sound a little harsh and that is not my intention. I
simply wish to say I think that has been an outstanding effort by all
involved. Getting this many people to work together on this effort has been
amazing and rewarding to me and, to the software I use. I hope it was to you
all.

See you tomorrow all and thank you all for your effort, suggestions,
patience and time,

Mike


-- 
Michael Smith
US Army Corps of Engineers
Remote Sensing/GIS Center
Hanover, NH 

 


On 9/6/10 11:09 PM, "Liujian (LJ) Qian" <LJ.Qian at oracle.com> wrote:

> 
>   Well actually I did run a re-org on the contour table in Oracle
> spatial based on a tessellation key column we computed (you will notice
> an additional linear_key column in our contour table).  I did not
> however notice a big difference, and now that you mentioned how it's
> created, that may explain it since my re-org probably even messed up the
> existing clustering. Of course it could also due to the fact that the DB
> box has only 4GB memory to work with. In our internal testing however
> for some large tables after this type of re-org we did get better
> throughput.
> 
> thanks
> LJ
> 
> On 9/6/2010 4:23 PM, Andrea Aime wrote:
>> Right right, that's actually some really nice improvement that we can
>> try next year: clustering the shapefiles
>> along the spatial index. It's a common technique in databases that
>> nobody tried out in this benchmark
>> (and which would have been a valid best effort approach).
>> 
>> That said I think at least the contour shapefiles do present some
>> actual spatial clustering given the
>> way they were created (afaik Ivan merged them from smaller files that
>> he created, and each of
>> those files is clearly visible if you preview the shapefile at the
>> whole Spain level).
>> 
>> Cheers
>> Andrea
>> 
>> On Mon, Sep 6, 2010 at 10:16 PM, Liujian (LJ) Qian<LJ.Qian at oracle.com>
>> wrote:
>>>   Or that.  Note however most blocks (as cached by the FS) will probably
>>> contain data that does not belong to the working set?   In other words, I
>>> assume the shapefiles do not always store geographically adjacent records in
>>> the same or neighboring blocks; but I don't really know about the
>>> clustered-ness within the .shp and .dbf files.
>> _______________________________________________
>> Benchmarking mailing list
>> Benchmarking at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/benchmarking
> 
> _______________________________________________
> Benchmarking mailing list
> Benchmarking at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/benchmarking