GeoTIFF overviews / TILEINDEX / Large dataset performance

[Dan Greve]

To everyone,

Are GeoTiff overviews taken advantage of by Mapserver?  What's the best way 
to handle large datasets (300 GB, 260,000 files in my case) when you want 
the user to be able to view the whole dataset.

Mark,

I was reading one of your OLD posts to Mapserver, and was hoping your 
suggestion might improved my server, but I'm not sure what you meant. Here's 
the original post...

<snip>
Ed,

A very good point. Let me interject one idea that I've used that really 
speeds up performance.
I use a tile index and check performance by zooming in to about 1:20K. Then 
I zoom out until performance drops.
I create a new tile grid shapefile using that map extent as the size of one 
tile. I tile the entire map area (in my case the world).
I create a new aggregate image layer using calls to the map server to 
generate an image for each new tile.
I repeat this process until I have good performance at any scale. I then 
tune the scales in the map file to match the layers.
This "pyramid" scheme is used in several map server products.
Granted 528 images are not that many - but displaying all at once takes a 
bit of time.
I've created a world mosaic from 360,000 images that displays at any scale 
in no more than 4 seconds.

Mark
</snip>

I have a lot of data, probably about 300GB spread among 260,000 tiles.  
Let's just say the region is... Texas.  I want the user to be able to see 
the data set at ANY zoom factor.  He'd start out looking at the entire state 
of texas, and be able to zoom progressively into a city block, and back out 
again.  If the data format could handle 300 GB in a single file (I'm using 
GeoTIFF), theoretically the performance would be better than if I created a 
TILEINDEX (shapefile) of the 260,000 tiles.  I've seen this in smaller 
datasets when requesting the entire scene, even just a 13000x13000 dataset 
with just over 2500 tiles.

When you say

"I create a new tile grid shapefile using that map extent as the size of one 
tile. I tile the entire map area (in my case the world). "

Do you mean you just duplicate the entire dataset with larger tiles when a 
TILEINDEX search would take longer?

When you say

"I create a new aggregate image layer using calls to the map server to 
generate an image for each new tile."

I have no idea what you meant by "aggregate image layer"

Are you downsampling the image at all as you increase the tile sizes?  The 
raster howto on the UMN site has a snippet about Frank W. wanting to 
implement using GeoTIFF overviews in the mapserver.  Does mapserver 
currently take advantage of this? Could you elaborate on your pyramid 
scheme?

-- Dan Greve, Software Engineer
-- AGS/BM Integrated Systems
-- Northrop Grumman Corporation
-- P.O. Box 9650, M.S. C04-222
-- Melbourne, Florida 32902-9650
-- (321) 951 6923
-- grevedan at hotmail.com